Operation risk real-time monitoring system based on multi-angle video and AI identification

Abstract

The invention discloses an operation risk real-time monitoring system based on multi-angle video and AI identification, and belongs to the technical field of video monitoring, and the operation risk real-time monitoring system comprises an operation data extraction module, an interaction prediction module, a posture prediction module, a risk decision module and an early warning execution module, the interaction prediction module predicts the interaction risk probability between dynamic entities, the attitude prediction module predicts the motion trail and attitude instability probability of a person, the risk decision module outputs a risk level judgment result, and the early warning execution module adopts a distributed computing architecture and executes a multi-channel early warning mechanism at an edge computing node. And judging the release condition of the risk level in real time. According to the method, the three-dimensional space pose sequence, the motion trail sequence, the three-dimensional human skeleton key point time sequence data and the attribute characteristics are extracted, low-delay closed-loop early warning is achieved through edge calculation, and the problems of perception missing, prediction blank, early warning lag and open loop are solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video monitoring, and particularly relates to a job risk real-time monitoring system based on multi-angle video and AI recognition. BACKGROUND

[0002] In the high-risk environment of industry, energy and construction operations, ensuring the safety of operating personnel and the site environment is the key to project management. In recent years, in order to meet the increasing requirements for site safety management, job risk real-time monitoring systems have been widely used and developed, aiming to use modern information technology to continuously collect and monitor data on the operation environment, personnel status and operation process through the integration of various sensors and monitoring devices arranged on the site. The core function is to efficiently and actively identify potential violations that may lead to safety accidents based on safety standards and risk models. After identifying the risks, alarm signals are generated and sent to achieve early warning, rapid response and effective control of risks, thereby comprehensively improving the safety management efficiency and accident prevention ability of the operation site.

[0003] In order to solve the problem that the existing monitoring system can only identify static violations and cannot predict dynamic interaction risks and posture risks in real time, the existing technology uses a method based on two-dimensional video image analysis to identify the static violation state of a single target. However, there is a lack of three-dimensional spatial structure perception in the construction site, which cannot obtain the accurate spatial position, motion trajectory and attribute characteristics of dynamic entities, cannot analyze the complex space-time interaction between personnel and machinery, personnel and environment, cannot predict the future motion trajectory and posture instability of personnel, and the data analysis and decision-making are heavily dependent on cloud computing, the overall system delay is high, and there is a lack of automatic closed-loop confirmation mechanism after warning, which leads to the problems of missing dynamic interaction risks, false reporting of static information caused by occlusion, and serious lag of warning information, making it difficult to effectively intervene in impending collision and falling accidents. SUMMARY

[0004] The application aims to provide a job risk real-time monitoring system based on multi-angle video and AI recognition, which realizes dynamic entity three-dimensional accurate perception, interaction and posture risk prediction in advance through multi-angle video acquisition and AI modeling, and relies on edge computing to achieve low-delay hierarchical warning and closed-loop confirmation, solving the problems of perception lack, prediction deficiency and warning lag of traditional monitoring systems.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the application is: The job risk real-time monitoring system based on multi-angle video and AI recognition comprises: The job data extraction module is configured to collect and pre-process construction job data, extract a three-dimensional spatial pose sequence of dynamic entities in a construction site, a motion trajectory sequence, three-dimensional human skeleton key point time series data and attribute features of personnel; The interaction prediction module is configured to input the three-dimensional spatial pose sequence, the motion trajectory sequence and the attribute features into a spatio-temporal interaction risk prediction model adopting a spatio-temporal graph neural network structure, and predict an interaction risk probability between the dynamic entities. The posture prediction module is configured to input the three-dimensional human skeleton key point time series data into a personnel posture intention prediction model, and predict a motion trajectory and a posture instability probability of the personnel. The risk decision module is configured to construct a risk comprehensive decision model based on a multi-layer perception machine based on the interaction risk probability and the motion trajectory and the posture instability probability, train and optimize the risk comprehensive decision model using a labeled data set containing historical risk events, set a job risk determination logic, and output a risk level determination result. The early warning execution module is configured to execute a multi-channel early warning mechanism at an edge computing node according to the risk level determination result using a distributed computing architecture, and determine a risk level release in real time.

[0006] Preferably, in the job data extraction module, the process of collecting and pre-processing the construction job data comprises: The construction job data comprises spatio-temporal video data of the construction site, operation parameter data of the machinery and real-time position state data of the personnel. The spatio-temporal video data is collected in real time by deploying multi-angle fixed cameras, personnel wearable cameras and mobile device cameras in the construction site. The operation parameter data of the machinery and the real-time position state data of the personnel are collected by deploying sensors on the construction machinery and position sensors on the personnel wearable devices. The multi-angle fixed cameras, the personnel wearable cameras, the mobile device cameras and the sensors are kept synchronized with the central time server in terms of time stamp, thereby providing a unified time reference for the collected spatio-temporal video data, the operation parameter data and the real-time position state data. The spatial coordinate system registration is performed, the internal parameter matrix and the external parameter matrix of the multi-angle fixed cameras are obtained by camera calibration method, and the mapping relationship between the world coordinate system points and the pixel coordinate system points is established, the spatio-temporal video data is unified under the same world coordinate system of the construction site, the pre-processed spatio-temporal video data is generated, and the real-time position state data of the personnel is converted to the same world coordinate system of the construction site. The sensor data cleaning is performed, and the operation parameter data of the machinery and the real-time position state data of the personnel are denoised and interpolated, thereby obtaining the pre-processed operation parameter data and the real-time position state data.

[0007] Preferably, in the work data extraction module, the process of extracting the three-dimensional spatial pose sequence, motion trajectory sequence, and three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site includes: Dynamic entities include personnel entities, machine entities, and material entities; Based on the preprocessed spatiotemporal video data, a multi-view geometric reconstruction algorithm is used to extract the three-dimensional spatial pose sequence and motion trajectory sequence of dynamic entities in the construction site. In spatiotemporal video data, run an instance segmentation model to obtain dynamic entities in multiview. Figure Two Bounding boxes and segmentation masks in 3D images; By using segmentation masks, internal parameter matrices, and external parameter matrices, a 3D point cloud model of a dynamic entity is generated through a multi-view contour reconstruction method. Perform 3D bounding box fitting and pose estimation on the 3D point cloud model of a dynamic entity to obtain the 3D spatial pose of the dynamic entity. A multi-target tracking algorithm is used to temporally correlate the three-dimensional spatial poses in consecutive frames to generate a three-dimensional spatial pose sequence and a motion trajectory sequence of dynamic entities. For human entities, extract the temporal data of key points of the three-dimensional human skeleton from the preprocessed spatiotemporal video data; In spatiotemporal video data, a two-dimensional human pose estimation algorithm is run to extract human entities in multi-view... Figure Two Key points of the two-dimensional human skeleton in a 3D image; Using the internal parameter matrix and the external parameter matrix, triangulation reconstruction of key points of the two-dimensional human skeleton is performed, and the coordinates of key points of the three-dimensional human skeleton are calculated. A multi-target tracking algorithm is used to temporally correlate the coordinates of key points of the three-dimensional human skeleton in consecutive frames to generate temporal data of key points of the three-dimensional human skeleton of a person. The pre-processed machine operation parameter data and the real-time location status data of personnel are integrated as attribute features of dynamic entities.

[0008] Preferably, in the interaction prediction module, the process of predicting the probability of interaction risk between dynamic entities includes: Construct a time series graph structure, which includes graph nodes, graph edges, node features, and edge features at each time step; Define the extracted dynamic entities as graph nodes; Assign the 3D spatial pose sequence, motion trajectory sequence, and attribute features as node features of the graph nodes; The dynamic spatiotemporal relationships between dynamic entities are defined as edges of a graph; Calculate the relative spatial distance and relative motion velocity between dynamic entities, and assign the calculated relative spatial distance and relative motion velocity as edge features of the graph. input the time sequence diagram structure into the space-time interaction risk prediction model, the space-time interaction risk prediction model adopts a space-time graph neural network structure, learns the time sequence changes of the graph node and edge features, and calculates and outputs the interaction risk probability between dynamic entities in a future preset time window; a spatial graph convolution layer is used to aggregate the node features and edge features at each time step to generate spatial aggregation features; a time sequence convolution layer is used to learn the change pattern of the spatial aggregation features at consecutive time steps; based on the change pattern, the interaction risk probability is calculated and output through a fully connected layer.

[0009] Preferably, in the posture prediction module, the process of predicting the motion trajectory of the person and the posture instability probability includes: at each time step of the three-dimensional human body skeleton key point time sequence data, a preset root node is selected as a reference point, the three-dimensional relative coordinates of each key point in the three-dimensional human body skeleton key points relative to the reference point are calculated, and normalized time sequence data are generated; input the normalized time sequence data into the personnel posture intention prediction model, the personnel posture intention prediction model adopts a long short-term memory network structure, the long short-term memory network structure learns the time sequence dependency of the normalized time sequence data through an internal gating unit, and outputs a posture feature vector; input the posture feature vector into the trajectory prediction layer, the trajectory prediction layer is used to calculate and output the three-dimensional human body skeleton key point coordinates of the person in a future preset time window as the motion trajectory of the person; input the posture feature vector into the instability classification layer, and the instability classification layer calculates and outputs the posture instability probability of the person.

[0010] Preferably, in the risk decision module, the process of constructing a risk comprehensive decision model based on a multilayer perceptron includes: splice and flatten the interaction risk probability, the motion trajectory of the person and the posture instability probability to generate an input feature vector of the risk comprehensive decision model; construct a risk comprehensive decision model adopting a multilayer perceptron structure, the multilayer perceptron structure includes an input layer, a hidden layer and an output layer; the input layer receives the input feature vector; the hidden layer maps the input feature vector to high-dimensional features through a nonlinear activation function; the output layer outputs a linear output value corresponding to a risk level, and the risk level is divided into an emergency level, a warning level, a prompt level and a normal level.

[0011] Preferably, in the risk decision module, the process of training and optimizing using a labeled data set containing historical risk events includes: the historical risk events include violations, quasi-accidents and accidents; The historical interaction risk probability, the historical personnel motion trajectory and the historical posture instability probability in the labeled data set are spliced and flattened to generate a training feature vector; The training feature vector is input into the risk comprehensive decision model to obtain a linear output value; A loss function value between the linear output value and a corresponding real risk level label in the labeled data set is calculated; The network weight parameters in the multi-layer perception structure are updated by using a back propagation algorithm until the loss function value converges, so that the risk comprehensive decision model trained and optimized is obtained.

[0012] Preferably, in the risk decision module, the process of setting operation risk judgment logic and outputting a risk level judgment result comprises: The input feature vector generated in real time is introduced into the risk comprehensive decision model trained and optimized to obtain a linear output value; A multi-class classification function is used to calculate a prediction score of each risk level; The risk level with the highest score in the prediction score is selected as the final risk level judgment result.

[0013] Preferably, in the early warning execution module, according to the risk level judgment result, a distributed computing architecture is used to execute a multi-channel early warning mechanism on an edge computing node, and the process comprises: When the risk level judgment result is an emergency level, target early warning information is sent to a wearable device of a person involved in the risk, an operation room display console of a related machine and a machine automatic control system; When the risk level judgment result is a warning level, target early warning information is sent to the wearable device of the person involved in the risk and a safety management personnel terminal; When the risk level judgment result is a prompt level, target early warning information is sent to the safety management personnel terminal.

[0014] Preferably, in the early warning execution module, the process of judging the release of the risk level in real time comprises: After the target early warning information is sent, the risk decision module continuously outputs a risk level judgment result of a subsequent time step; If the risk level judgment result of the subsequent time step is a normal level, it is judged that the risk level has been released, and the sending of the target early warning information is stopped; If the risk level judgment result of the subsequent time step is not a normal level, it is judged that the risk level has not been released, and the sending of the target early warning information is continued.

[0015] The present application has the following advantages: This invention solves the problems of lack of three-dimensional spatial structure perception and inability to obtain accurate position information caused by existing technologies relying on two-dimensional image analysis by collecting and extracting three-dimensional spatial pose sequences, motion trajectory sequences, three-dimensional human skeleton key point time sequence data and attribute features of dynamic entities, and realizes high-precision three-dimensional reconstruction and continuous tracking of dynamic entities at construction sites.

[0016] This invention utilizes a spatiotemporal graph neural network structure and a temporal prediction network structure, combined with a multilayer perceptron for risk decision-making. It solves the problem that existing technologies can only identify static violations and cannot predict dynamic risks. It achieves accurate prediction of the risk of interaction between dynamic entities, the probability of future movement trajectory and posture instability of personnel, and has the ability to proactively intervene before accidents occur.

[0017] This invention employs a distributed computing architecture to perform real-time judgment and early warning on edge computing nodes, and has the function of real-time judgment of risk level clearance. It solves the problems of serious delay in early warning information and lack of early warning closed-loop confirmation mechanism caused by existing technologies relying on cloud computing, and ensures extremely low latency of system early warning and effectiveness of closed-loop management. Attached Figure Description

[0018] Figure 1 This is a structural block diagram of the system of the present invention; Figure 2 This is a simulation test diagram of the normal monitoring stage during the verification process of this invention; Figure 3 This is a simulation test diagram of the prompt-level early warning stage during the verification process of this invention; Figure 4 This is a simulation test diagram of the warning level early warning stage during the verification process of this invention; Figure 5 This is a simulation test diagram of the emergency warning stage during the verification process of this invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0020] Example 1: As Figure 1 As shown, the real-time operational risk monitoring system based on multi-angle video and AI recognition includes: The operation data extraction module is used to collect and preprocess construction operation data, and extract the three-dimensional spatial pose sequence, motion trajectory sequence, three-dimensional human skeleton key point time sequence data and attribute features of dynamic entities in the construction site.

[0021] Collect and preprocess construction operation data, which includes spatiotemporal video data of the construction site, mechanical operation parameter data, and real-time location status data of personnel.

[0022] Through the multi-angle fixed camera deployed on the construction site, the personnel wearable camera and the mobile device camera, real-time collection of multi-source heterogeneous space-time video data.

[0023] Through the sensor deployed on the construction machinery and the position sensor of the personnel wearable device, collection of the working parameter data of the machinery and the real-time position state data of the personnel.

[0024] Make the multi-angle fixed camera, the personnel wearable camera, the mobile device camera and the sensor keep time stamp synchronization with the center time server, provide a unified time reference for the collected space-time video data, the working parameter data and the real-time position state data.

[0025] Perform spatial coordinate system registration, through the camera calibration method, obtain the internal parameter matrix K and the external parameter matrix of the multi-angle fixed camera, for establishing the mapping relationship between the world coordinate system point and the pixel coordinate system point p, unify the space-time video data to the same world coordinate system of the construction site, generate the preprocessed space-time video data, and convert the real-time position state data of the personnel to the same world coordinate system of the construction site, the calculation formula of the mapping relationship is as follows: ; Wherein, represents a two-dimensional point in the pixel coordinate system, u and v are respectively the column and row coordinates of the pixel in the image, represents a three-dimensional point in the world coordinate system, is the three-dimensional coordinates of the spatial point in the world coordinate system, is the scale factor, K is the 3*3 internal parameter matrix, which contains the focal length and the principal point, R is the 3*3 rotation matrix, and T is the 3*1 translation vector.

[0026] Perform sensor data cleaning, denoising and interpolation processing on the working parameter data of the machinery and the real-time position state data of the personnel, to obtain the preprocessed working parameter data and real-time position state data.

[0027] Dynamic entities include personnel entities, machinery entities and material entities.

[0028] Based on the preprocessed space-time video data, use the multi-view geometry reconstruction algorithm to extract the three-dimensional space pose sequence and motion trajectory sequence of the dynamic entities in the construction site.

[0029] In the space-time video data, run the instance segmentation model to obtain the bounding box and segmentation mask of the dynamic entities in the multi-view image. Figure Two

[0030] ​The three-dimensional point cloud model of the dynamic entity is generated by using the segmentation mask, the internal parameter matrix and the external parameter matrix through a multi-view contour reconstruction method.

[0031] The three-dimensional point cloud model of the dynamic entity is fitted with a three-dimensional bounding box and the pose is estimated to obtain the three-dimensional spatial pose of the dynamic entity.

[0032] The three-dimensional spatial pose sequence and the motion trajectory sequence of the dynamic entity are generated by time-series correlating the three-dimensional spatial poses in consecutive frames through a multi-target tracking algorithm.

[0033] The three-dimensional human body skeleton key point time-series data of the personnel entity is extracted from the pre-processed spatio-temporal video data.

[0034] The two-dimensional human body pose estimation algorithm is run in the spatio-temporal video data to extract the two-dimensional human body skeleton key points of the personnel entity in multi-view images. Figure Two

[0035] The internal parameter matrix and the external parameter matrix are used to triangulate and reconstruct the two-dimensional human body skeleton key points to calculate the three-dimensional human body skeleton key point coordinates.

[0036] The three-dimensional human body skeleton key point time-series data of the personnel is generated by time-series correlating the three-dimensional human body skeleton key point coordinates in consecutive frames through a multi-target tracking algorithm.

[0037] The pre-processed mechanical operation parameter data and the real-time position state data of the personnel are integrated as the attribute features of the dynamic entity.

[0038] The mechanical operation parameter data is associated to the corresponding mechanical entity according to the unique identification of the mechanical.

[0039] The real-time position state data of the personnel is associated to the corresponding personnel entity according to the unique identification of the personnel.

[0040] The associated operation parameter data and real-time position state data are unified as the attribute features of the dynamic entity.

[0041] The interactive prediction module is configured to input the three-dimensional spatial pose sequence, the motion trajectory sequence and the attribute features into a spatio-temporal interactive risk prediction model adopting a spatio-temporal graph neural network structure to predict the interactive risk probability between the dynamic entities.

[0042] A time-series graph structure is constructed, which includes graph nodes, graph edges, node features and edge features at each time step.

[0043] The extracted dynamic entities are defined as graph nodes.

[0044] The three-dimensional spatial pose sequence, the motion trajectory sequence and the attribute features are assigned as the node features of the graph nodes.​

[0045] The dynamic spatio-temporal relationship between dynamic entities is defined as the edge of the graph.

[0046] The relative spatial distance and the relative motion speed between the dynamic entities are calculated, and the calculated relative spatial distance and the relative motion speed are assigned as the edge features of the edges of the graph.

[0047] The time series graph structure is input into the spatio-temporal interaction risk prediction model, and the spatio-temporal interaction risk prediction model adopts a spatio-temporal graph neural network structure, learns the time series changes of the graph node and edge features, calculates and outputs the interaction risk probability between the dynamic entities within a preset time window in the future.

[0048] The spatial graph convolution layer is used to aggregate the node features and edge features at each time step to generate spatial aggregation features.

[0049] The time series convolution layer is used to learn the change pattern of the spatial aggregation features at consecutive time steps.

[0050] Based on the change pattern, the interaction risk probability is calculated and output through the fully connected layer.

[0051] The spatio-temporal interaction risk prediction model takes a spatio-temporal graph neural network (AI deep learning architecture) as the core, first processes the input three-dimensional space pose sequence, motion trajectory sequence and other data through AI recognition technology. For example, use instance segmentation AI model to identify the boundary features of dynamic entities (personnel, machinery, materials), and use multi-target tracking AI algorithm to identify the continuous motion trajectory of the entity; then through the AI logic of the graph neural network, the dynamic entities are defined as graph nodes, and the spatio-temporal relationship between the entities is defined as graph edges, the time series change rule of the nodes and edges is learned, and finally the interaction risk probability between the entities within a preset time window in the future is identified and predicted. It is an AI intelligent identification and risk prediction of the interaction relationship between dynamic entities.

[0052] The posture prediction module is used to input the three-dimensional human skeleton key point time series data into the personnel posture intention prediction model to predict the motion trajectory and posture instability probability of the personnel.

[0053] At each time step of the three-dimensional human skeleton key point time series data, a preset root node is selected as a reference point, the three-dimensional relative coordinates of each key point in the three-dimensional human skeleton key points relative to the reference point are calculated, and normalized time series data is generated.

[0054] The normalized time series data is input into the personnel posture intention prediction model, and the personnel posture intention prediction model adopts a long short-term memory network structure. The long short-term memory network structure learns the time series dependency of the normalized time series data through the internal gating unit, and outputs a posture feature vector.

[0055] The posture feature vector is input into a trajectory prediction layer, and the trajectory prediction layer is configured to calculate and output three-dimensional human skeleton key point coordinates of the person in a preset time window in the future as a motion trajectory of the person.

[0056] The posture feature vector is input into a posture instability classification layer, and the posture instability classification layer is configured to calculate and output a posture instability probability of the person wherein z is a linear output of the posture instability classification layer, and e is a natural constant.

[0057] A risk decision module is configured to construct a risk comprehensive decision model based on a multi-layer perception and based on the interaction risk probability, the motion trajectory, and the posture instability probability, and to train and optimize the risk comprehensive decision model using a labeled data set containing historical risk events, to set a work risk determination logic, and to output a risk level determination result.

[0058] The person posture intention prediction model takes a long short-term memory network (LSTM, an AI time series prediction model) as a core, extracts two-dimensional skeleton key points of the person through a two-dimensional human posture estimation algorithm (an AI vision recognition technology), and obtains three-dimensional skeleton key point time series data through triangulation reconstruction (an AI space geometry calculation logic); then the LSTM model learns time dependence of the time series data through an AI algorithm, identifies a dynamic change rule of the person posture, and further predicts a future motion trajectory (AI recognition of the posture intention) and a posture instability probability (AI recognition of a risk posture feature) of the person, so as to realize AI intelligent conversion from posture data to intention and risk.

[0059] The interaction risk probability, the motion trajectory of the person, and the posture instability probability are spliced and flattened to generate an input feature vector of the risk comprehensive decision model.

[0060] The risk comprehensive decision model adopts a multi-layer perception structure, and the multi-layer perception structure includes an input layer, a hidden layer, and an output layer.

[0061] The input layer receives the input feature vector.

[0062] The hidden layer performs high-dimensional feature mapping on the input feature vector through a nonlinear activation function.

[0063] The output layer outputs a linear output value corresponding to a risk level, and the risk level is divided into an emergency level, a warning level, a prompt level, and a normal level.

[0064] The historical risk events include a violation, a quasi-accident, and an accident.

[0065] The historical interaction risk probability, the historical motion trajectory of the person, and the historical posture instability probability in the labeled data set are spliced and flattened to generate a training feature vector.

[0066] The training feature vector is input into the risk comprehensive decision model to obtain a linear output value.

[0067] A loss function value between the linear output value and a real risk level label corresponding in the labeled data set is calculated.

[0068] The network weight parameters in the multi-layer perceptron structure are updated by using a back propagation algorithm until the loss function value converges, so as to obtain the risk comprehensive decision model with completed training optimization.

[0069] The input feature vector generated in real time is introduced into the risk comprehensive decision model with completed training optimization to obtain a linear output value.

[0070] A multi-class classification function is used to calculate a prediction score of each risk level wherein j is an index of the current risk level, C is a total number of risk level categories, is a linear output value of the output layer for the jth risk level, is a linear output value for the kth risk level.

[0071] The risk level with the highest prediction score is selected as the final risk level determination result.

[0072] The early warning execution module is configured to execute a multi-channel early warning mechanism at an edge computing node according to the risk level determination result by using a distributed computing architecture, and to determine a release condition of the risk level in real time.

[0073] When the risk level determination result is the emergency level, target early warning information is sent to a wearable device of a person involved in the risk, an operation room display console of a related machine, and a machine automatic control system.

[0074] When the risk level determination result is the warning level, target early warning information is sent to the wearable device of the person involved in the risk and a terminal of a safety management personnel.

[0075] When the risk level determination result is the prompt level, target early warning information is sent to the terminal of the safety management personnel.

[0076] After the target early warning information is sent, the risk decision module continuously outputs risk level determination results of subsequent time steps.

[0077] If the risk level determination result of the subsequent time step is the normal level, it is determined that the risk level has been released, and the sending of the target early warning information is stopped.

[0078] If the risk level determination result of the subsequent time step is not the normal level, it is determined that the risk level has not been released, and the sending of the target early warning information is continued.

[0079] The three-dimensional spatial pose sequence, the motion trajectory sequence and the risk level determination result of the extracted dynamic entity are presented in real time in the three-dimensional visualization interface.

[0080] Firstly, deploy hardware in the construction site, including cameras, sensors, central time servers and edge computing nodes, and build a distributed computing architecture. Secondly, use a labeled dataset containing historical risk events to train and optimize the risk comprehensive decision model, the spatio-temporal interaction risk prediction model and the personnel posture intention prediction model, and obtain the trained and optimized model. The construction data extraction module starts to collect and preprocess the construction data in real time, and extracts the three-dimensional spatial pose sequence, the motion trajectory sequence, the three-dimensional human skeleton key point time series data and the attribute features of the dynamic entity. The system performs real-time risk determination on the edge computing node. The interaction prediction module inputs the three-dimensional spatial pose sequence, the motion trajectory sequence and the attribute features to predict the interaction risk probability between dynamic entities. The posture prediction module inputs the three-dimensional human skeleton key point time series data to predict the motion trajectory and posture instability probability of personnel. The risk decision module imports the trained and optimized risk comprehensive decision model based on the interaction risk probability, the motion trajectory and the posture instability probability, and outputs the risk level determination result. The warning execution module receives the risk level determination result on the edge computing node. If the determination result is emergency level, warning level or prompt level, the multi-channel warning mechanism is triggered. If the determination result is normal level, no warning is executed. At the same time, the relevant three-dimensional sequence and risk level are presented in real time in the three-dimensional visualization interface. Finally, the warning execution module performs a warning closed-loop determination. After triggering the warning, the system continuously determines the subsequent risk level. If the subsequent risk level determination result is normal level, it is determined that the risk level has been removed, and the warning is stopped. If the subsequent risk level is not removed, the warning information is continuously sent.

[0081] Through simulation experiments, the effectiveness of the system of the embodiment is verified. In the simulated construction site, the dynamic interaction risk and posture instability risk are predicted in real time, and the ability of multi-level warning is executed under the typical scenario that the worker approaches and enters the dangerous area of mechanical operation.

[0082] A simulation scene is constructed by simulating a construction site in a MATLAB simulation environment, using a worker (simulated 3D skeleton), a central machine (black square), three material entities (triangles) and a core dangerous area marked by a dashed circle. The simulation experiment diagram is shown in Figures 2-5 .

[0083] Figures 2-5 In the simulation experiment, the left side is a simulation scene diagram, and the right side is a data dashboard output by the system of the embodiment in real time. The dashboard dynamically displays five core indicators, including interaction risk probability, predicted trajectory, posture instability probability, comprehensive risk and warning channel.

[0084] Figure 2 As shown in the normal level monitoring stage, the worker is located in the safety area outside the dangerous area, at this time, the interactive risk probability output by the interactive prediction module of the present application is 0.02, the posture instability probability output by the posture prediction module is 0.23, the risk decision module determines that the comprehensive risk is normal level, and no pre-warning is performed.

[0085] Figure 3 As shown in the prompt level pre-warning stage, the worker continues to walk and approaches the dangerous area, at this time, the interactive prediction module predicts that the interactive risk probability rises to 0.14, the posture prediction module predicts that the trajectory of the worker will lead to instability risk, and outputs a posture instability probability of 0.41, the risk decision module comprehensively determines that the comprehensive risk is a prompt level, and the pre-warning execution module starts to send target pre-warning information to the management terminal.

[0086] Figure 4 As shown in the warning level pre-warning stage, the worker has entered the edge of the dangerous area, both the interactive prediction module and the posture prediction module of the system output high probability, the interactive risk probability rises to 0.45, the posture instability probability rises to 0.65, the risk decision module determines that the risk is upgraded to a warning level, the pre-warning execution module increases the pre-warning channel, and sends target pre-warning information to the wearable device and the management terminal at the same time.

[0087] Figure 5 As shown in the emergency level pre-warning stage, the worker has entered the center of the dangerous area, at this time, both the interactive risk probability (0.98) and the posture instability probability (0.91) are close to saturation, the risk decision module determines that the comprehensive risk is an emergency level, the pre-warning execution module triggers the highest level of pre-warning, displays “!!! emergency risk!!!” in the scene, and sends target pre-warning to the wearable device, the mechanical control console and the automatic control, to realize closed-loop intervention before the accident occurs.

[0088] The simulation example verifies that the embodiment effectively solves the problem that the traditional monitoring system can only identify static violations and cannot predict dynamic interactive risk and posture instability by combining the spatio-temporal graph neural network and the long short-term memory network, and through the multi-layer perceptron for multi-source risk fusion and four-level (normal level, prompt level, warning level and emergency level) risk determination, the accuracy and timeliness of the pre-warning are ensured.

[0089] In summary, the system of the embodiment has strong dynamic risk prediction and grading pre-warning capability in the dynamic operation scene where the worker approaches and enters the dangerous source, at the same time, by combining the multi-modal AI prediction model and the risk comprehensive decision model, a closed-loop management system from pre-prediction to real-time pre-warning is constructed.

Claims

1. A real-time operational risk monitoring system based on multi-angle video and AI recognition, characterized in that: include: The operation data extraction module is used to collect and preprocess construction operation data, and extract the three-dimensional spatial pose sequence, motion trajectory sequence, three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site. The interaction prediction module is used to input three-dimensional spatial pose sequences, motion trajectory sequences and attribute features into a spatiotemporal interaction risk prediction model with a spatiotemporal graph neural network structure to predict the probability of interaction risk between dynamic entities. The posture prediction module is used to input the time-series data of the key points of the three-dimensional human skeleton into the human posture intention prediction model to predict the human's motion trajectory and the probability of posture instability. The risk decision module is used to construct a comprehensive risk decision model based on a multilayer perceptron, based on the probability of interaction risk, the probability of motion trajectory and attitude instability, and to train and optimize it using a labeled dataset containing historical risk events, set the operation risk judgment logic, and output the risk level judgment result. The early warning execution module is used to execute a multi-channel early warning mechanism on edge computing nodes based on the risk level determination results, using a distributed computing architecture, and to determine the risk level clearance status in real time.

2. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, The process of collecting and preprocessing construction operation data in the operation data extraction module includes: Construction operation data includes spatiotemporal video data of the construction site, mechanical operation parameter data, and real-time location status data of personnel; Real-time spatiotemporal video data is collected using multi-angle fixed cameras, personnel wearable cameras, and mobile device cameras deployed at the construction site. By using sensors deployed on construction machinery and position sensors on personnel's wearable devices, the system collects operational parameter data of the machinery and real-time location status data of the personnel. This ensures that multi-angle fixed cameras, wearable cameras, mobile device cameras and sensors are synchronized with the central time server, providing a unified time reference for the collected spatiotemporal video data, operational parameter data and real-time location status data; Perform spatial coordinate system registration. Through camera calibration methods, obtain the internal parameter matrix and external parameter matrix of the multi-angle fixed camera. Use this matrix to establish the mapping relationship between world coordinate system points and pixel coordinate system points. Unify the spatiotemporal video data to the same world coordinate system of the construction site, generate preprocessed spatiotemporal video data, and convert the real-time position status data of personnel to the same world coordinate system of the construction site. Perform sensor data cleaning, which involves denoising and interpolating the machine's operating parameter data and the personnel's real-time position status data to obtain pre-processed operating parameter data and real-time position status data.

3. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 2, characterized in that, The process of extracting the three-dimensional spatial pose sequence, motion trajectory sequence, and three-dimensional human skeleton key point time series data and attribute features of dynamic entities in the construction site in the operation data extraction module includes: Dynamic entities include personnel entities, machine entities, and material entities; Based on the preprocessed spatiotemporal video data, a multi-view geometric reconstruction algorithm is used to extract the three-dimensional spatial pose sequence and motion trajectory sequence of dynamic entities in the construction site. In spatiotemporal video data, run an instance segmentation model to obtain the bounding boxes and segmentation masks of dynamic entities in multi-view 2D images; By using segmentation masks, internal parameter matrices, and external parameter matrices, a 3D point cloud model of a dynamic entity is generated through a multi-view contour reconstruction method. Perform 3D bounding box fitting and pose estimation on the 3D point cloud model of a dynamic entity to obtain the 3D spatial pose of the dynamic entity. A multi-target tracking algorithm is used to temporally correlate the three-dimensional spatial poses in consecutive frames to generate a three-dimensional spatial pose sequence and a motion trajectory sequence of dynamic entities. For human entities, extract the temporal data of key points of the three-dimensional human skeleton from the preprocessed spatiotemporal video data; In spatiotemporal video data, a two-dimensional human pose estimation algorithm is run to extract the two-dimensional human skeleton key points of the human entity in multi-view two-dimensional images; Using the internal parameter matrix and the external parameter matrix, triangulation reconstruction of key points of the two-dimensional human skeleton is performed, and the coordinates of key points of the three-dimensional human skeleton are calculated. A multi-target tracking algorithm is used to temporally correlate the coordinates of key points of the three-dimensional human skeleton in consecutive frames to generate temporal data of key points of the three-dimensional human skeleton of a person. The pre-processed machine operation parameter data and the real-time location status data of personnel are integrated as attribute features of dynamic entities.

4. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the interaction prediction module, the process of predicting the probability of interaction risks between dynamic entities includes: Construct a time series graph structure, which includes graph nodes, graph edges, node features, and edge features at each time step; Define the extracted dynamic entities as graph nodes; Assign the 3D spatial pose sequence, motion trajectory sequence, and attribute features as node features of the graph nodes; The dynamic spatiotemporal relationships between dynamic entities are defined as edges of a graph; Calculate the relative spatial distance and relative motion velocity between dynamic entities, and assign the calculated relative spatial distance and relative motion velocity as edge features of the graph. The temporal graph structure is input into the spatiotemporal interaction risk prediction model. The spatiotemporal interaction risk prediction model adopts a spatiotemporal graph neural network structure. By learning the temporal changes of graph node and edge features, it calculates and outputs the probability of interaction risk between dynamic entities within a future preset time window. Spatial graph convolutional layers are used to aggregate node features and edge features at each time step to generate spatial aggregated features; Temporal convolutional layers are used to learn the changing patterns of spatial aggregate features at consecutive time steps; Based on the change pattern, the probability of interaction risk is calculated and output through a fully connected layer.

5. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the attitude prediction module, the process of predicting the trajectory of a person and the probability of attitude instability includes: At each time step of the three-dimensional human skeleton key point time series data, a preset root node is selected as the reference point, and the three-dimensional relative coordinates of each key point in the three-dimensional human skeleton key point relative to the reference point are calculated to generate normalized time series data. Normalized time-series data is input into the human posture and intention prediction model. The human posture and intention prediction model adopts a long short-term memory network structure. The long short-term memory network structure learns the temporal dependence of normalized time-series data through internal gating units and outputs posture feature vectors. The posture feature vector is input into the trajectory prediction layer, which is used to calculate and output the coordinates of the three-dimensional human skeleton key points of the person within a preset time window in the future, as the person's motion trajectory. The posture feature vector is input into the instability classification layer, which calculates and outputs the probability of posture instability of the person.

6. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the risk decision-making module, the process of constructing a comprehensive risk decision-making model based on a multilayer perceptron includes: The probability of interactive risks, the trajectory of personnel and the probability of posture instability are spliced ​​and flattened to generate the input feature vector of the risk comprehensive decision model. A risk comprehensive decision-making model with a multilayer perceptron structure is constructed, which includes an input layer, a hidden layer, and an output layer. The input layer receives the input feature vector; Hidden layers perform high-dimensional feature mapping on the input feature vector using non-linear activation functions; The output layer outputs a linear output value corresponding to the risk level, which is divided into emergency, warning, alert, and normal levels.

7. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 1, characterized in that, In the risk decision-making module, the process of training and optimizing using a labeled dataset containing historical risk events includes: Historical risk events include violations, near-accidents, and accidents; The historical interaction risk probability, historical personnel movement trajectory and historical posture instability probability in the labeled dataset are concatenated and flattened to generate training feature vectors. Input the trained feature vectors into the risk comprehensive decision-making model to obtain linear output values; Calculate the loss function value between the linear output value and the corresponding true risk level label in the labeled dataset; The backpropagation algorithm is used to update the network weight parameters in the multilayer perceptron structure until the loss function value converges, thus obtaining the trained and optimized risk comprehensive decision model.

8. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 6, characterized in that, In the risk decision-making module, the process of setting up the operational risk assessment logic and outputting the risk level assessment result includes: The real-time generated input feature vector is imported into the trained and optimized risk comprehensive decision-making model to obtain a linear output value. A multi-class classification function is used to calculate the predicted score for each risk level. The risk level with the highest predicted score is selected as the final risk level determination.

9. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 2, characterized in that, In the early warning execution module, based on the risk level determination results, the process of executing a multi-channel early warning mechanism on edge computing nodes using a distributed computing architecture includes: When the risk level is determined to be emergency, targeted early warning information is sent to the wearable devices of the personnel involved in the risk, the control console of the control room of the relevant machinery, and the automatic control system of the machinery. When the risk level assessment result is warning level, targeted early warning information is sent to the wearable devices of the personnel involved in the risk and the terminals of the safety management personnel. When the risk level assessment result is at the warning level, a targeted early warning message is sent to the safety management personnel's terminal.

10. The real-time operational risk monitoring system based on multi-angle video and AI recognition according to claim 9, characterized in that, The process of determining the clearance status of risk levels in real time within the early warning execution module includes: After sending targeted early warning information, the risk decision module continuously outputs the risk level determination results for subsequent time steps; If the risk level assessment result of the subsequent time step is normal, then the risk level is determined to be lifted, and the sending of targeted early warning information will be stopped. If the risk level assessment result in subsequent time steps is not at the normal level, the risk level is determined to be not resolved, and targeted early warning information will continue to be sent.

Images