Underground cavern group construction safety monitoring method and device based on digital twin intelligent agent

Abstract

The application discloses a kind of underground cavern group construction safety monitoring methods based on digital twin intelligent agent, effective fragment in output picture is retrieved to field monitoring video using fine tuning VLM, and two-dimensional pose estimation is carried out to target object, three-dimensional reconstruction model is used to convert two-dimensional pose into three-dimensional pose and is projected back to two dimensions according to camera parameter, and projection two-dimensional pose is obtained;The distance error of predicted two-dimensional pose and projection two-dimensional pose and bone length consistency loss are combined into the hybrid loss function of three-dimensional reconstruction model;Target object three-dimensional pose coordinates are converted to its own local coordinate system, and the motion constraint of target object is simplified and motion constraint equation is constructed, the motion trajectory of target object is predicted using Taylor series expansion, to construct the dynamic dangerous adjacent area of target object;Safety monitoring intelligent agent carries out adjacent monitoring to the dynamic dangerous adjacent area of multiple digital twins.The application uses safety monitoring intelligent agent to realize closed-loop construction safety monitoring.

Description

Technical Field

[0001] This invention relates to the field of engineering construction safety monitoring, and in particular to a method and equipment for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents. Background Technology

[0002] Currently, fixed surveillance cameras are commonly deployed at underground cavern construction sites for safety monitoring and progress verification. However, due to the limitations of enclosed spaces, obstructed views, and the intensity of work, on-site safety risks exhibit multi-source, multi-trigger, and strongly coupled characteristics: relative movement of personnel and construction equipment, congestion and intersections in passageways, abnormal equipment operation, accidental entry into prohibited areas, and obstruction of emergency evacuation routes. To address these risks, constructing a digital twin construction safety monitoring system capable of continuously mapping on-site conditions, predicting risks, and triggering early warnings and interventions is of great significance.

[0003] Currently, construction safety monitoring of underground cavern complexes still relies primarily on manual inspections, making it difficult to quantify the spatial relationships and dynamic risks among personnel, equipment, and passageways in a timely and objective manner. In environments with multiple cameras operating simultaneously, narrow passageways, limited field of view, and extremely uneven lighting (overexposure / backlight / darkness), manual identification is labor-intensive, prone to errors, and struggles to establish stable closed-loop intervention. Sensor-based monitoring requires installing multiple sensors on various types of on-site construction machinery and personnel, resulting in high costs, extensive installation and maintenance, and the need for regular calibration to suppress data drift. Solutions based on Real-Time Location Systems (RTLS) or Global Positioning Systems (GNSS) are affected by underground cavern obstructions and resource constraints, leading to unstable or even unusable positioning signals. In contrast, computer vision-based monitoring methods offer advantages such as non-contact operation, zero equipment modification, reusability of existing monitoring networks, convenient deployment, and strong real-time performance, making them well-suited to the complex environment of underground caverns. Computer vision-based monitoring methods can be further categorized into two-stage and one-stage methods based on their detection paradigm. Compared to two-stage methods that first generate candidate regions and then classify and localize them, single-stage methods directly perform feature extraction and target classification / localization, making them suitable for real-time monitoring. Single-stage detection algorithms (such as the YOLO series) are widely used due to their balance between inference speed and accuracy.

[0004] Digital twins map multi-source data of physical objects, processes, and environments into a computable virtual entity in real time within a unified spatiotemporal coordinate system. Through continuous synchronization and historical playback, a state representation of "visualization—computability—traceability" is formed. An intelligent agent typically refers to a software system or automated device capable of proactively perceiving the environment, making autonomous decisions, and taking action. Combining digital twins with intelligent agents allows the digital twin to provide the intelligent agent with a "world model," integrating and unifying multi-source sensory information. The intelligent agent uses this as input to map the spatiotemporal trajectories and action patterns of personnel and equipment into risk indicators and output interventions. Simultaneously, strategy simulation and sandbox verification can be performed within the twin before being deployed to the field, forming a closed loop of "perception—assessment—early warning / control." Furthermore, in the safety monitoring of underground cavern construction, traditional manual inspection methods rely heavily on experience and subjective judgment. The frequency of inspections and the range of vision are limited, making continuous coverage in complex scenarios difficult. Moreover, they cannot objectively quantify the three-dimensional spatiotemporal relationships of multiple targets in a two-dimensional image, often resulting in delayed "post-event alarms" and lacking a traceable and verifiable intervention loop. In contrast, digital twins integrate multi-source data under a unified spatiotemporal coordinate system, providing three-dimensional scenes and historical playback. Based on this, intelligent agents can map personnel-machine trajectories and action patterns into quantifiable dynamic danger proximity areas, enabling closed-loop intervention from advance assessment to early warning / control.

[0005] However, existing research still has some limitations: First, construction work is not continuous in underground cavern complexes. Real-time identification at all times would consume computing power and bandwidth, and the lack of an on-demand triggering retrieval and identification mechanism affects the economics and deployability of field applications. Second, existing methods are mostly designed for open outdoor scenes, making them difficult to adapt to the complex environments of underground caverns with exposure / backlighting / darkness, dust blurring, narrow terrain, and severe obstruction. Third, existing video data-based monitoring methods mostly fail to meet the engineering requirements for construction safety monitoring in three-dimensional space, and the lack of three-dimensional information in monocular image / video data limits the accuracy of related vision-based monitoring methods. Finally, existing research often separates visual recognition monitoring and safety early warning strategies into two stages, generally lacking a closed loop of visual information constructing a digital twin to drive construction safety monitoring, making it difficult to conduct timely risk simulation, strategy verification, and feedback on intervention effects. Therefore, a digital twin intelligent agent construction method is needed for intelligent monitoring of construction safety in complex underground cavern groups. This method can achieve on-demand target retrieval, robust identification of research objects in complex underground cavern group environments, construction of digital twin models, prediction of dynamic hazardous adjacent areas and risk indicators, and implementation of early warning control, thereby reducing collision risks and improving construction safety and management efficiency. Summary of the Invention

[0006] This invention provides a method and equipment for construction safety monitoring of underground cavern groups based on digital twin intelligent agents to solve the technical problems existing in the prior art.

[0007] The technical solution adopted by this invention to solve the technical problems existing in the prior art is as follows: A method for construction safety monitoring of underground cavern groups based on digital twin intelligent agents, the method comprising the following steps: Step 1: Deploy on-site monitoring cameras and obtain relevant parameters of the on-site monitoring cameras and a unified coordinate system on-site through on-site calibration and measurement. Step 2: Construct a digital twin of the target object, a data processing module, and a safety monitoring intelligent agent. The data processing module includes: a video slicing module, a video segment annotation module, a video frame extraction module, an image annotation module, an annotation format conversion module, and a dataset partitioning module. The safety monitoring intelligent agent includes: a large visual model for filtering out valid video segments containing the target object from video segments; a two-dimensional attitude prediction model for predicting the two-dimensional attitude data of the target object from image data containing the target object; a three-dimensional reconstruction model for reconstructing the two-dimensional attitude data of the target object into three-dimensional attitude data of the target object; a regional hazard assessment module for predicting the three-dimensional attitude trajectory of the target object and constructing a dynamic hazard proximity area; and an early warning module for monitoring and warning of construction safety distances. Step 3: The construction video of the underground cavern is collected by the monitoring camera to create training samples; the collected video is segmented into segments, the video segments are labeled, and the pre-trained visual model is fine-tuned using the labeled video segments; the fine-tuned visual model is used to process the video segments and extract the video segments containing the target object. Step 4: Extract frames from the video clip containing the target object, process them into image sequence data, and label them. Use the labeled image sequence data to train the two-dimensional pose prediction model. Step 5: The trained 2D pose prediction model is used to predict the image sequence data to obtain the predicted 2D pose sequence data; the predicted 2D pose sequence data is used to train the 3D reconstruction model, and the 3D pose data output by the 3D reconstruction model is used as the intermediate 3D pose data. Step 6: Based on the camera parameters, use the pinhole-distortion projection model to project the intermediate 3D pose data into 2D pose data; incorporate the distance error between the 2D pose data predicted by the 2D pose prediction model and the 2D pose data projected by the pinhole-distortion projection model into the training loss of the 3D reconstruction model. Step 7: During construction, the monitoring camera collects real-time video of the construction site, segments the video into segments and inputs them into the fine-tuned visual model. The visual model extracts video segments containing the target object, extracts frames from the video segments containing the target object and processes them into image sequence data, and then inputs them into the trained two-dimensional pose prediction model. The predicted two-dimensional pose sequence data output by the two-dimensional pose prediction model is input into the trained three-dimensional reconstruction model, and the three-dimensional reconstruction model predicts the three-dimensional pose data of the target object in real time. Step 8: Real-time predicted 3D attitude data and position trajectory of the target object are synchronized to its digital twin in real time; the real-time 3D attitude coordinates of the target object are transformed from the unified coordinate system on site to the local coordinate system of the target object itself through coordinate transformation; the regional hazard assessment module uses Taylor series analysis to predict the trajectory of the target object and form a dynamic hazard adjacent area in real time. Step 9: The early warning module monitors the dynamic dangerous proximity zone distance of the digital twin in real time and implements early warning control to achieve real-time safety monitoring of the target object.

[0008] Furthermore, step 1 includes the following method steps: Manually calibrating the calibration board at multiple locations within the monitoring camera area, including the four corners, center, and near, middle, and far points, is performed. Simultaneously, measuring tools are used to acquire the coordinates of multiple control points on-site, establishing the origin and a unified coordinate system for the field. ; Using existing tools and procedures, based on the pinhole-distortion projection model, and using multiple sets of collected data, the camera intrinsic parameters were analyzed. Distortion coefficient and camera external parameters Solve for camera extrinsic parameters. For each frame of multiple sets of calibration data, the corresponding data is calculated, and then the average value is taken to determine the result. ; ; ; In the formula: Represents camera intrinsic parameters, which are 3×3 matrices describing imaging geometry and pixel scale; Indicating camera extrinsic parameters involves unifying the coordinate system of the site. The parameters when transforming points in the coordinate system to the camera coordinate system; This represents a 3×3 rotation orthogonal matrix used to describe the orientation from the field unified coordinate system to the camera coordinate system; This represents a 3×1 translation vector used to describe the position of the origin of the unified field coordinate system in the camera coordinate system; The horizontal focal length in pixels of the camera; Represents the camera's vertical focal length in pixels; Indicates the horizontal coordinates of the camera's principal point; Indicates the longitudinal coordinates of the camera's principal point; This represents the camera skew coefficient, used to describe the non-orthogonality between pixel axes; Represents the lateral coordinates of the undistorted normalized image plane; Represents the normalized vertical coordinates of the undistorted image plane; Represents the normalized lateral coordinates of the image plane after distortion; Represents the normalized vertical coordinates of the image plane after distortion; Represents the normalized radius from the undistorted normalized image plane coordinates to the camera coordinates; , , Indicates the camera's radial distortion coefficient; , Indicates the camera's tangential distortion coefficient; Represents a point in a unified coordinate system on site; This represents a point in the camera coordinate system.

[0009] Furthermore, step 2 includes the following method steps: Based on measurement data and structural parameters of construction equipment, a digital twin of the site is constructed; A two-dimensional pose prediction model is constructed based on Yolov11-pose, and a three-dimensional reconstruction model is constructed based on deep neural networks, convolutional neural networks, neural radiation field networks, or holographic generative networks.

[0010] Furthermore, the method for constructing a two-dimensional pose prediction model based on Yolov11-pose includes the following steps; Auxiliary reversible branches are introduced into the backbone network of Yolov11-pose to form a dual backbone architecture. The main branch is responsible for routine feature extraction and prediction, while the auxiliary branch injects complete input information during the loss calculation stage to improve the gradient and alleviate the information bottleneck. By preserving the gradient and local information and reducing the training memory usage, the network's ability to extract image features is enhanced, and the accuracy and stability of the model are improved in scenes with dust, occlusion, and poor lighting conditions. The first layer of the dual-backbone network uses augmented detail convolutional modules, which include horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules. During the training phase, the augmented detail convolutional modules are composed of horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules connected in parallel. During the working inference phase, the parallel horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules are equivalently merged into a standard convolution.

[0011] Furthermore, step 3 includes the following sub-steps: Step 3-1: The video slicing module uses Python to randomly segment the collected on-site monitoring video, with the segmentation time being greater than or equal to a set threshold, forming several video clips; the video clip annotation module annotates the segmented video clips accordingly, while the annotation format conversion module uses Python to convert the file format, thereby creating dataset A for fine-tuning the large-scale visual model; the dataset partitioning module uses Python to partition dataset A into a training set, a validation set, and a test set, with a ratio of 7:2:1 for each subset; when creating dataset A, the annotation content includes the target object in the video clip and the timestamp of the corresponding excavator image, which helps guide the large-scale visual model to identify the target object and whether the target object exists in the video frame; Step 3-2: Input dataset A into the visual big model to train and fine-tune the visual big model. After fine-tuning, the visual big model can identify target objects and output timestamps of valid video segments. Based on the timestamps output by the visual big model, the video slicing module is further called to segment the random monitoring video segments, thereby deleting redundant video segments that do not contain target objects and retaining valid video segments that contain target objects.

[0012] Furthermore, step 4 includes the following method steps: The video frame extraction module extracts effective video segments from the output of the large visual model and processes them into image data. The image annotation module then uses this image data to create dataset B for training the 2D pose prediction model. The annotations include the bounding boxes and pose keypoints of the target objects. The dataset partitioning module divides dataset B into training, validation, and test sets. The 2D pose prediction model is trained using dataset B and outputs the 2D pose data of the target objects. The performance of the optimized model is evaluated using average precision, recall, and / or F1 score metrics to obtain and output robust 2D pose data of the target objects.

[0013] Furthermore, step 6 includes the following sub-steps: Step 6-1: Using the known camera parameters, project the intermediate 3D pose data of the target object through a pinhole-distortion projection model to generate 2D pose data. Construct the reprojection error loss between the predicted 2D pose data and the projected 2D pose data of the target object, and its expression is as follows: ; In the formula: This represents the reprojection error loss, used to describe the pixel deviation between the projected 2D pose and the predicted 2D pose of the target object. Indicates the index of the pose key points of the target object; Indicates the target object's first The weights corresponding to each pose key point; Represents a robust loss function; Represents the two-dimensional attitude coordinates obtained by reprojection; This represents the two-dimensional attitude coordinates of the target object predicted by the two-dimensional attitude prediction model; This represents the L2 norm, used to describe the Euclidean distance between two points; Step 6-2: Introduce a bone length consistency constraint to help the hybrid loss function converge, the expression of which is as follows: ; In the formula: This represents the bone length consistency loss, used to describe the deviation loss of the change in link length between adjacent frames; This indicates the total number of image frames; Indicates the total number of links in the target object; Indicates the sequence number of the image frame; Indicates the serial number of the link in the target object; Indicates the first The target object in the frame image The length of the connecting rod; Indicates the first The target object in the frame image The length of the connecting rod; Step 6-3: Weight the reprojection error loss and bone length consistency loss to form a hybrid loss function, which is used as the training loss for the 3D reconstruction model: Then we have: ; In the formula: The hyperparameter representing the degree of control constraint is used to balance the relative contributions of reprojection error loss and bone length consistency loss. The training loss objective function of the 3D reconstruction model; The hybrid loss function enables the 3D reconstruction model to learn to reduce the time variation of link length, thereby avoiding random jitter of key points of the target object.

[0014] Furthermore, the 3D reconstruction model includes several reconstruction modules connected in sequence. Each reconstruction module consists of four parts: a Conv1D layer, a Batch Normalization layer, a ReLU layer, and a Dropout layer.

[0015] Furthermore, step 8 includes the following sub-steps: Step 8-1: Transform the final 3D attitude coordinates of the target object from the unified coordinate system on site to its own local coordinate system using the coordinate transformation formula, as follows:

[0016] In the formula: Indicates the sequence number of the key points of the target object's pose; This represents the angle between the local coordinate system of the target object and the unified coordinate system of the site along the X-axis. This represents the X-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system. This represents the Y-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system; This represents the Z-axis coordinate of the q-th attitude key point of the target object in the unified coordinate system on site; The X-axis represents the x-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; The Y-axis represents the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; This represents the Z-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; X represents the horizontal axis of the unified coordinate system on site; Y represents the vertical axis of the unified coordinate system on site; Z represents the deep axis of the unified on-site coordinate system; X q Let x be the horizontal axis of the local coordinate system of the target object; Y q The vertical axis of the local coordinate system of the target object; Z q The deep axis of the local coordinate system of the target object; Step 8-2: Simplify the target object's motion posture into a motion posture described by changing joint angles, while considering the target object's reaction time. The trajectory of the target object's expected movement within the reaction time is considered as a dangerous adjacent area; a Taylor series is used to fit a trajectory with a higher probability based on the current motion state. The Taylor series expression is as follows: ; In the formula: Indicates the current time of the trajectory prediction; The index representing the order of the summation is a non-negative integer; This represents the time increment, which is equivalent to the reaction time of the target object. A function representing time; express The function at time The First-order time derivative; Step 8-3: Use Taylor series to predict the motion posture of the target object described by the variable joint angles, and obtain the dynamic danger proximity area of ​​the target object; Step 8-4: Transmit the 3D attitude data and predicted trajectory of the target object to the digital twin in real time. The early warning module monitors the dynamic danger proximity area of ​​each target object in real time. When the dynamic danger proximity areas of any two or more target objects overlap, intersect, or intrude into each other, the early warning module sends an early warning and control command to the corresponding digital twin. The corresponding digital twin sends action instructions to the corresponding physical entity of the target object, and the physical entity executes the corresponding action according to the action instructions.

[0017] The present invention also provides an apparatus for a method of monitoring the construction safety of underground cavern groups based on a digital twin intelligent agent, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the method of monitoring the construction safety of underground cavern groups based on a digital twin intelligent agent as described above.

[0018] The advantages and positive effects of this invention are: (1) The present invention uses fine-tuned VLM to understand and retrieve the construction monitoring of underground cavern groups in order to eliminate redundant information in the monitoring video, significantly reduce the computing power and bandwidth occupation caused by continuous all-time reasoning, and improve the economy and deployability of the system.

[0019] (2) This invention addresses the problems of narrow terrain, severe obstruction, and exposure / backlighting / darkness, dust blurring in monitoring videos in underground cavern groups. It improves upon the Yolov11-pose model by introducing detail enhancement convolution and auxiliary reversible branch, which effectively adapts to the complex underground cavern group environment, thereby improving the accuracy and robustness of two-dimensional pose recognition of target objects.

[0020] (3) This invention introduces a reprojection error loss and combines it with bone length consistency constraints to form a hybrid loss function, thereby achieving robust three-dimensional pose estimation of the target object and effectively overcoming the limitations of existing methods in three-dimensional pose estimation under monocular conditions.

[0021] (4) Based on the three-dimensional attitude estimation of the target object, the present invention simplifies the motion attitude of the target object into a motion attitude described by variable joint angles and uses Taylor series to predict the trajectory to form a dynamic danger proximity area. Based on the digital twin and the development of a safety monitoring intelligent agent, a closed-loop construction safety monitoring is realized, effectively combining visual monitoring and safety early warning, realizing the closed loop of "perception-evaluation-early warning / control", and improving the timeliness and reliability of engineering construction safety monitoring. Attached Figure Description

[0022] Figure 1 This is a flowchart of a construction safety monitoring method for underground cavern groups based on a digital twin intelligent agent, according to the present invention.

[0023] Figure 2 This is a framework diagram of a two-dimensional attitude prediction model based on Yolov11-pose in this invention.

[0024] Figure 3 This is a schematic diagram of the construction method of a digital twin intelligent agent for intelligent monitoring of construction safety in complex underground cavern groups according to the present invention.

[0025] Figure 4 This is a simplified structural diagram of the connection of key points of the excavator's posture, a target object, according to the present invention.

[0026] Figure 5 This is a simplified motion constraint diagram of the connection of key points of the excavator's posture according to the present invention.

[0027] Figure 4 , 5 middle: 1. Right tip of the bucket; 2. Left tip of the bucket; 3. Connection between the stick and the bucket; 4. Connection between the boom and the stick; 5. Connection between the boom and the boom cylinder; 6. Connection between the boom and the vehicle body; 7. Rear of the excavator body.

[0028] X' represents the translation along the X-axis of the unified coordinate system on site; Y' represents the translation along the Y-axis of the unified coordinate system on site; Z' indicates the translation along the Z-axis of the unified coordinate system on site; The arrows on the coordinate axes indicate the positive direction of the coordinates; X q The x-axis of the local coordinate system of the target object; Y q The vertical axis of the local coordinate system of the target object; Z q The deep axis of the local coordinate system of the target object; The link formed by attitude point boom_base and attitude point stick_base and the local coordinate system X q The included angle of the axis; The link formed by attitude point stick_base and attitude point stick_end and the local coordinate system X q The included angle of the axis; The plane formed by the attitude points stick_end, bucket_left, and bucket_right intersects with the local coordinate system X. q The included angle of the axis; Excavator local coordinate system X q The angle between the axis and the X-axis of the unified coordinate system on site. Detailed Implementation

[0029] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0030] In the description of this invention, the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," and "bottom," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and do not require the invention to be constructed and operated in a specific orientation; therefore, they should not be construed as limitations on the invention. The terms "connected" and "linked" used in this invention should be interpreted broadly. For example, they can refer to a fixed connection or a detachable connection; a direct connection or an indirect connection through intermediate components; or an electrical connection or signal transmission. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0031] The Chinese definitions of the following English words, phrases, and abbreviations are as follows: boom_base: The connection point between the boom and the vehicle body.

[0032] boom_ end: The connection point between the boom and the boom cylinder.

[0033] stick_base: The connection point between the boom and the stick.

[0034] stick_end: The connection point between the stick and the bucket.

[0035] bucket_left: The left tip of the bucket.

[0036] bucket_right: The right tip of the bucket.

[0037] Conv1D: One-dimensional convolutional layer.

[0038] Batch Normalization: Batch normalization.

[0039] ReLU: Rectified Linear Unit, representing the activation function. .

[0040] Dropout: Random deactivation, which means setting the output of some neurons in the model to zero according to probability.

[0041] Yolov11-pose: The pose estimation variant of the 11th generation of the You Only Look Once algorithm series.

[0042] Detail-enhanced Convolution: Convolution with enhanced details.

[0043] Labelme: An open-source image annotation tool.

[0044] VLM: Vision-Language Model.

[0045] Python: A general-purpose programming language.

[0046] RTK: Real-Time Kinematic, a measurement tool for real-time dynamic carrier phase differential.

[0047] Please see Figures 1 to 5 A method for construction safety monitoring of underground cavern groups based on digital twin intelligent agents, the method comprising the following steps: Step 1: Deploy on-site monitoring cameras and obtain relevant parameters of the on-site monitoring cameras and a unified coordinate system on-site through on-site calibration and measurement. Step 2: Construct a digital twin of the target object, a data processing module, and a safety monitoring intelligent agent. The data processing module includes: a video slicing module, a video segment annotation module, a video frame extraction module, an image annotation module, an annotation format conversion module, and a dataset partitioning module. The safety monitoring intelligent agent includes: a large visual model for filtering out valid video segments containing the target object from video segments; a two-dimensional attitude prediction model for predicting the two-dimensional attitude data of the target object from image data containing the target object; a three-dimensional reconstruction model for reconstructing the two-dimensional attitude data of the target object into three-dimensional attitude data of the target object; a regional hazard assessment module for predicting the three-dimensional attitude trajectory of the target object and constructing a dynamic hazard proximity area; and an early warning module for monitoring and warning of construction safety distances. Step 3: The construction video of the underground cavern is collected by the monitoring camera to create training samples; the collected video is segmented into segments, the video segments are labeled, and the pre-trained visual model is fine-tuned using the labeled video segments; the fine-tuned visual model is used to process the video segments and extract the video segments containing the target object. Step 4: Extract frames from the video clip containing the target object, process them into image sequence data, and label them. Use the labeled image sequence data to train the two-dimensional pose prediction model. Step 5: The trained 2D pose prediction model is used to predict the image sequence data to obtain the predicted 2D pose sequence data; the predicted 2D pose sequence data is used to train the 3D reconstruction model, and the 3D pose data output by the 3D reconstruction model is used as the intermediate 3D pose data. Step 6: Based on the camera parameters, use the pinhole-distortion projection model to project the intermediate 3D pose data into 2D pose data; incorporate the distance error between the 2D pose data predicted by the 2D pose prediction model and the 2D pose data projected by the pinhole-distortion projection model into the training loss of the 3D reconstruction model. Step 7: During construction, the monitoring camera collects real-time video of the construction site, segments the video into segments and inputs them into the fine-tuned visual model. The visual model extracts video segments containing the target object, extracts frames from the video segments containing the target object and processes them into image sequence data, and then inputs them into the trained two-dimensional pose prediction model. The predicted two-dimensional pose sequence data output by the two-dimensional pose prediction model is input into the trained three-dimensional reconstruction model, and the three-dimensional reconstruction model predicts the three-dimensional pose data of the target object in real time. Step 8: Real-time predicted 3D attitude data and position trajectory of the target object are synchronized to its digital twin in real time; the real-time 3D attitude coordinates of the target object are transformed from the unified coordinate system on site to the local coordinate system of the target object itself through coordinate transformation; the regional hazard assessment module uses Taylor series analysis to predict the trajectory of the target object and form a dynamic hazard adjacent area in real time. Step 9: The early warning module monitors the dynamic dangerous proximity zone distance of the digital twin in real time and implements early warning control to achieve real-time safety monitoring of the target object.

[0048] Preferably, step 1 may include the following method steps: Manually calibrating the calibration board at multiple locations within the monitoring camera area, including the four corners, center, and near, middle, and far points, is performed. Simultaneously, measuring tools are used to acquire the coordinates of multiple control points on-site, establishing the origin and a unified coordinate system for the field. .

[0049] Using existing tools and procedures, based on the pinhole-distortion projection model, and using multiple sets of collected data, the camera intrinsic parameters were analyzed. Distortion coefficient and camera external parameters Solve for camera extrinsic parameters. For each frame of multiple sets of calibration data, the corresponding data is calculated, and then the average value is taken to determine the result. ; ; ; In the formula: Represents camera intrinsic parameters, which are 3×3 matrices describing imaging geometry and pixel scale; Indicating camera extrinsic parameters involves unifying the coordinate system of the site. The parameters when transforming points in the coordinate system to the camera coordinate system; This represents a 3×3 rotation orthogonal matrix used to describe the orientation from the field unified coordinate system to the camera coordinate system; This represents a 3×1 translation vector used to describe the position of the origin of the unified field coordinate system in the camera coordinate system; The horizontal focal length in pixels of the camera; Represents the camera's vertical focal length in pixels; Indicates the horizontal coordinates of the camera's principal point; Indicates the longitudinal coordinates of the camera's principal point; This represents the camera skew coefficient, used to describe the non-orthogonality between pixel axes; Represents the lateral coordinates of the undistorted normalized image plane; Represents the normalized vertical coordinates of the undistorted image plane; Represents the normalized lateral coordinates of the image plane after distortion; Represents the normalized vertical coordinates of the image plane after distortion; Represents the normalized radius from the undistorted normalized image plane coordinates to the camera coordinates; , , Indicates the camera's radial distortion coefficient; , Indicates the camera's tangential distortion coefficient; Represents a point in a unified coordinate system on site; This represents a point in the camera coordinate system.

[0050] Preferably, step 2 may include the following method steps: Based on measurement data and structural parameters of construction equipment, a digital twin of the site is constructed; A two-dimensional pose prediction model is constructed based on Yolov11-pose, and a three-dimensional reconstruction model is constructed based on deep neural networks, convolutional neural networks, neural radiation field networks, or holographic generative networks.

[0051] Holographic Generating Networks (HGNs) are deep learning-driven computational holographic networks. Their core consists of three main categories: real-valued / complex-valued backbone networks, physical propagation constraint modules, and output encoding modules. Mainstream architectures also incorporate optimization units such as attention, residuals, and skip connections.

[0052] Real-valued networks are used to process real numbers such as intensity, grayscale, and depth. Their functions include: extracting real-domain features from 2D images, depth maps, and light field intensity, and performing initial phase prediction, refocusing, and feature dimensionality reduction.

[0053] Commonly used architectures for real-valued networks include: U-Net, ResNet, multi-scale residual blocks, downsampling / upsampling modules, and skip connections to preserve details.

[0054] Complex-valued networks are used to process the complex amplitude / phase of optical fields. They are the core of holography and their function is to directly learn the complex amplitude field (amplitude + phase), adapt to the output of SLM (spatial light modulator), and are more in line with optical physics than real-valued networks.

[0055] Commonly used structures for complex-valued networks include: complex-valued ResUNet (a deep learning model that combines residual networks and U-Net architecture), complex-valued convolution / activation / normalization, and dual-branch (amplitude + phase).

[0056] Preferably, the method for constructing a two-dimensional pose prediction model based on Yolov11-pose may include the following steps; A dual-backbone architecture is formed by introducing an auxiliary reversible branch into the backbone network of Yolov11-pose. The main branch is responsible for routine feature extraction and prediction, while the auxiliary branch injects complete input information during the loss calculation stage to improve the gradient and alleviate the information bottleneck. By preserving the gradient and local information and reducing the memory usage for training, the network's ability to extract image features is enhanced, and the accuracy and stability of the model are improved in scenes with dust, occlusion, and poor lighting conditions.

[0057] The first layer of the dual-backbone network uses augmented detail convolutional modules, which include horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules. During the training phase, the augmented detail convolutional modules are composed of horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules connected in parallel. During the working inference phase, the parallel horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules are equivalently merged into a standard convolution.

[0058] The workflow of the Yolov11-pose backbone network can be selected as follows: Figure 2 The process shown is as follows: Figure 2 The modules are described in the following table: The module function table of the Yolov11-pose backbone network:

[0059] Preferably, step 3 may include the following sub-steps: Step 3-1: The video slicing module uses Python to randomly segment the collected on-site monitoring video, with the segmentation time being greater than or equal to a set threshold, forming several video clips; the video clip annotation module annotates the segmented video clips accordingly, while the annotation format conversion module uses Python to convert the file format, thereby creating dataset A for fine-tuning the large-scale visual model; the dataset partitioning module uses Python to partition dataset A into a training set, a validation set, and a test set, with a ratio of 7:2:1 for each subset; when creating dataset A, the annotation content includes the target object in the video clip and the timestamp of the corresponding excavator image, which helps guide the large-scale visual model to identify the target object and whether the target object exists in the video frame; Step 3-2: Input dataset A into the visual big model to train and fine-tune the visual big model. After fine-tuning, the visual big model can identify target objects and output timestamps of valid video segments. Based on the timestamps output by the visual big model, the video slicing module is further called to segment the random monitoring video segments, thereby deleting redundant video segments that do not contain target objects and retaining valid video segments that contain target objects.

[0060] Preferably, step 4 may include the following method steps: The video frame extraction module extracts effective video segments from the output of the large visual model and processes them into image data. The image annotation module then uses this image data to create dataset B for training the 2D pose prediction model. The annotations include the bounding boxes and pose keypoints of the target objects. The dataset partitioning module divides dataset B into training, validation, and test sets. The 2D pose prediction model is trained using dataset B and outputs the 2D pose data of the target objects. The performance of the optimized model is evaluated using average precision, recall, and / or F1 score metrics to obtain and output robust 2D pose data of the target objects.

[0061] Preferably, step 6 may include the following sub-steps: Step 6-1: Using the known camera parameters, project the intermediate 3D pose data of the target object through a pinhole-distortion projection model to generate 2D pose data. Construct the reprojection error loss between the predicted 2D pose data and the projected 2D pose data of the target object, and its expression is as follows: ; In the formula: This represents the reprojection error loss, used to describe the pixel deviation between the projected 2D pose and the predicted 2D pose of the target object. Indicates the index of the pose key points of the target object; Indicates the target object's first The weights corresponding to each pose key point; Represents a robust loss function; Represents the two-dimensional attitude coordinates obtained by reprojection; This represents the two-dimensional attitude coordinates of the target object predicted by the two-dimensional attitude prediction model; This represents the L2 norm, used to describe the Euclidean distance between two points; Step 6-2: Introduce a bone length consistency constraint to help the hybrid loss function converge, the expression of which is as follows: ; In the formula: This represents the bone length consistency loss, used to describe the deviation loss of the change in link length between adjacent frames; This indicates the total number of image frames; Indicates the total number of links in the target object; Indicates the sequence number of the image frame; Indicates the serial number of the link in the target object; Indicates the first The target object in the frame image The length of the connecting rod; Indicates the first The target object in the frame image The length of the connecting rod.

[0062] Step 6-3: Weight the reprojection error loss and bone length consistency loss to form a hybrid loss function, which is used as the training loss for the 3D reconstruction model: Then we have: ; In the formula: The hyperparameter representing the degree of control constraint is used to balance the relative contributions of reprojection error loss and bone length consistency loss. The training loss objective function of the 3D reconstruction model; The hybrid loss function enables the 3D reconstruction model to learn to reduce the time variation of link length, thereby avoiding random jitter of key points of the target object.

[0063] Preferably, the three-dimensional reconstruction model may include several reconstruction modules connected in sequence, and each reconstruction module consists of four parts: a Conv1D layer, a Batch Normalization layer, a ReLU layer, and a Dropout layer.

[0064] Preferably, step 8 may include the following sub-steps: Step 8-1: Transform the final 3D attitude coordinates of the target object from the unified coordinate system on site to its own local coordinate system using the coordinate transformation formula, as follows:

[0065] In the formula: Indicates the sequence number of the key points of the target object's pose; This represents the angle between the local coordinate system of the target object and the unified coordinate system of the site along the X-axis. This represents the X-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system. This represents the Y-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system; This represents the Z-axis coordinate of the q-th attitude key point of the target object in the unified coordinate system on site; The X-axis represents the x-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; The Y-axis represents the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; This represents the Z-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; X represents the horizontal axis of the unified coordinate system on site; Y represents the vertical axis of the unified coordinate system on site; Z represents the deep axis of the unified on-site coordinate system; X q Let x be the horizontal axis of the local coordinate system of the target object; Y q The vertical axis of the local coordinate system of the target object; Z q This is the deep axis of the local coordinate system of the target object.

[0066] Step 8-2: Simplify the target object's motion posture into a motion posture described by changing joint angles, while considering the target object's reaction time. The trajectory of the target object's expected movement within the reaction time is considered as a dangerous adjacent area; a Taylor series is used to fit a trajectory with a higher probability based on the current motion state. The Taylor series expression is as follows: ; In the formula: Indicates the current time of the trajectory prediction; The index representing the order of the summation is a non-negative integer; This represents the time increment, which is equivalent to the reaction time of the target object. A function representing time; express The function at time The The first-order time derivative.

[0067] Step 8-3: Use Taylor series to predict the motion posture of the target object described by the variable joint angles, and obtain the dynamic danger proximity area of ​​the target object.

[0068] Step 8-4: Transmit the 3D attitude data and predicted trajectory of the target object to the digital twin in real time. The early warning module monitors the dynamic danger proximity area of ​​each target object in real time. When the dynamic danger proximity areas of any two or more target objects overlap, intersect, or intrude into each other, the early warning module sends an early warning and control command to the corresponding digital twin. The corresponding digital twin sends action instructions to the corresponding physical entity of the target object, and the physical entity executes the corresponding action according to the action instructions.

[0069] The present invention also provides an apparatus for a method of monitoring the construction safety of underground cavern groups based on a digital twin intelligent agent, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program and, when executing the computer program, implement the steps of the method of monitoring the construction safety of underground cavern groups based on a digital twin intelligent agent as described above.

[0070] The workflow and working principle of the present invention are further illustrated below using a preferred embodiment as an example: A method for construction safety monitoring of underground cavern complexes based on digital twin intelligent agents is proposed. This method constructs a unified coordinate system through on-site data acquisition and solves for camera parameters using a pinhole distortion model. It employs a fine-tuned VLM to retrieve valid segments of the on-site monitoring video containing excavators, and is based on an improved Yolov11-pose model (model architecture details are available in [link to model]). Figure 2 Two-dimensional attitude estimation of the excavator was performed, which included 7 key points for the excavator's two-dimensional attitude (see details). Figure 4 The three positions are: excavator body tail, boom-body connection point, boom-boom cylinder connection point, boom-stick connection point, stick-end connection point, stick-bucket connection point, bucket left tip, and bucket right tip. A deep neural network (containing multiple blocks, each block consisting of Conv1D, Batch Normalization, ReLU, and Dropout layers) is used to upscale the predicted 2D pose to an intermediate 3D pose and then reproject it back to 2D based on camera parameters to obtain the projected 2D pose. The distance error between the predicted and projected 2D poses is considered as the reprojection loss, and a bone length consistency loss is introduced to form a hybrid loss function. Based on the hybrid loss function, the deep neural network is continuously optimized during training to obtain the final 3D pose of the excavator. The 3D pose coordinates of the excavator are transformed to its own local coordinate system (see details). Figure 4 ), and simplified the excavator's motion constraints to 4 constraint angles (see details). Figure 5 Furthermore, motion constraint equations are constructed, and Taylor series expansion is used to predict the excavator's motion trajectory, thereby constructing the excavator's dynamic hazard proximity zone. Based on on-site data and excavator attitude data, a digital twin is constructed. The developed construction safety monitoring intelligent agent monitors the dynamic hazard proximity zones of multiple digital twins. Depending on the situation, corresponding commands are sent to the digital twins. After receiving the commands, the digital twins send instructions to the corresponding physical entities. The excavator in the real world receives and executes the instructions, realizing real-time closed-loop safety monitoring of excavators in complex underground cavern groups.

[0071] Through the above steps, this invention enables on-demand retrieval of valid segments from underground cavern monitoring videos, robust acquisition of the target object's 3D pose, and real-time mapping of this pose to a digital twin on-site. A safety monitoring intelligent agent then assesses the dynamic danger zone online and provides early warnings / control. This method not only significantly improves identification accuracy under complex conditions and enables proactive prevention of risks to target objects in underground cavern groups, but also reduces computing power and deployment costs, decreases the burden of manual inspections, and enhances the real-time nature and traceability of safety management. It possesses significant engineering application value and broad application prospects.

[0072] The method specifically includes the following steps: Step S1: Using a calibration board and measuring tools, perform calibration measurements in the surveillance camera area, establish a unified coordinate system on site, and use existing tools and programs to analyze and solve the camera parameters of the surveillance cameras. Based on the collected data, construct a digital twin of the site and simultaneously collect on-site surveillance video. Specifically, this includes: S11: Manually hold the calibration board and perform calibration at multiple points within the monitoring camera area, including the four corners, center, and near, middle, and far positions, while simultaneously collecting construction monitoring data for the corresponding time period. At the same time, use surveying tools (such as RTK) to measure the coordinates of multiple control points on site, and establish a unified coordinate system on site based on the collected data and the origin. .

[0073] S12: Using existing tools and programs, based on the pinhole-distortion projection model principle, analyze and solve multiple sets of manually calibrated data to obtain the camera intrinsic parameters of the surveillance camera. Distortion coefficient and camera external parameters The solution is performed, and the corresponding expression is as follows, where the camera extrinsic parameters are... The data for each frame should be calculated from multiple sets of calibration data, and then the average value should be taken to determine the result. ; ; ; In the formula: Represents camera intrinsic parameters, which are 3×3 matrices describing imaging geometry and pixel scale; Indicating camera extrinsic parameters involves unifying the coordinate system of the site. The parameters when transforming points in the coordinate system to the camera coordinate system; This represents a 3×3 rotation orthogonal matrix used to describe the orientation from the field unified coordinate system to the camera coordinate system; This represents a 3×1 translation vector used to describe the position of the origin of the unified field coordinate system in the camera coordinate system; The horizontal focal length in pixels of the camera; Represents the camera's vertical focal length in pixels; Indicates the horizontal coordinates of the camera's principal point; Indicates the longitudinal coordinates of the camera's principal point; This represents the camera skew coefficient, used to describe the non-orthogonality between pixel axes; Represents the lateral coordinates of the undistorted normalized image plane; Represents the normalized vertical coordinates of the undistorted image plane; Represents the normalized lateral coordinates of the image plane after distortion; Represents the normalized vertical coordinates of the image plane after distortion; Represents the normalized radius from the undistorted normalized image plane coordinates to the camera coordinates; , , Indicates the camera's radial distortion coefficient; , Indicates the camera's tangential distortion coefficient; Represents a point in a unified coordinate system on site; This represents a point in the camera coordinate system.

[0074] S13: Based on the on-site measurement data, construct a digital twin of the on-site environment and simultaneously collect on-site construction monitoring videos to provide training data for subsequent vision-based construction safety monitoring methods.

[0075] Step S2: Using the previously collected on-site construction monitoring videos, create a dataset (dataset A) for fine-tuning the pre-trained visual large model (VLM). Fine-tuning the pre-trained VLM allows the VLM model, applicable in general domains, to be generalized to the complex underground cavern environment and corresponding construction processes and mechanical resources involved in this invention, enabling the VLM to identify, retrieve, and output valid video clips (those with examples). Specifically, this includes: S21: The collected on-site monitoring videos are randomly segmented (the segmentation time must not be lower than a set threshold) to form several video clips. These clips are then labeled accordingly, and a Python program is used to convert the file format, thus forming a dataset (dataset A) for fine-tuning the VLM. Similarly, a Python program is used to partition dataset A into a training set, a validation set, and a test set, with a ratio of 7:2:1. When creating dataset A, the annotations include the excavator in the video clips and the timestamps corresponding to the presence of the excavator, which helps guide the VLM to identify the excavator and its presence or absence in the video footage.

[0076] S22: Use dataset A to train and fine-tune the VLM so that the VLM can identify excavators and output timestamps of valid video segments. Use Python to write a program to segment random monitoring video segments based on the timestamps output by the VLM, thereby deleting redundant video segments (segments that do not exist in the example) and outputting valid video segments.

[0077] Step S3: Use Python to write a program to extract frames from effective control video clips to form image data, and call the Labelme annotation tool to annotate the extracted images to create a dataset (dataset B) for training the improved Yolov11-pose model; similarly, dataset B is divided into training set, validation set and test set, with a ratio of 7:2:1 for each subset; when creating dataset B, the annotation content includes the target bounding box of the excavator and 7 pose key points.

[0078] Step S4: The traditional Yolov11-pose model is improved by introducing Detail-enhanced Convolution and auxiliary invertible branching to address issues such as overexposure / backlighting / darkness, dust blurring, and severe excavator occlusion due to numerous construction machines and a single monitoring perspective in underground cavern monitoring videos. The improved Yolov11-pose model is trained using dataset B to obtain robust 2D pose estimation of the excavator in complex environments. The architecture of the improved Yolov11-pose model can be seen in the appendix. Figure 1 Specifically, it includes: S41: Introducing the concept of auxiliary invertible branches, an auxiliary invertible branch is introduced on the Yolov11-pose backbone to form a dual backbone architecture. The original backbone focuses on global semantics and scale robustness, while the auxiliary backbone is used to preserve high-frequency details. By retaining gradient and detail information and reducing training memory usage, the network's ability to extract image features is enhanced, and the accuracy and robustness of the model in scenes with dust, blur, occlusion, or poor lighting conditions are improved.

[0079] S42: In the improved model, the first layer of the dual backbone network uses detail enhancement convolutions instead of ordinary convolutions. The detail enhancement convolutions consist of horizontal difference convolutions (H), vertical difference convolutions (V), diagonal difference convolutions (D), central difference convolutions (C), and an ordinary convolution. During the training phase, they are connected in parallel with multiple branches, and during the inference phase, the branches are folded into standard convolutions, which can enhance the network's ability to learn the detailed features of excavators under complex conditions. S43: Train the improved Yolov11-pose model using dataset B, evaluate the performance of the optimized model using metrics such as average precision, recall, and F1 score, obtain the robust two-dimensional pose of the excavator and output it as input to the deep neural network for subsequent pose enhancement and reprojection optimization.

[0080] Step S5: Input the predicted 2D pose of the excavator into a deep neural network to upscale it to 3D. Then, using the camera parameters solved in the previous steps, reproject the 3D pose of the excavator back to 2D to obtain its projected 2D pose. Construct a reprojection error loss using the distance between the predicted 2D pose and the projected 2D pose, and introduce a bone length consistency constraint to form a hybrid loss function to optimize the deep neural network, thereby obtaining the final 3D pose of the excavator. Specifically, this includes: S51: The predicted two-dimensional posture of the excavator is input into a deep neural network. The neural network consists of several blocks, each of which consists of four parts: a Conv1D layer, a Batch Normalization layer, a ReLU layer, and a Dropout layer, and outputs the intermediate three-dimensional posture of the excavator.

[0081] S52: Using the camera parameters obtained from the previous steps, the intermediate 3D pose of the excavator is reprojected back to the 2D pose using the pinhole-distortion model. The distance between the predicted 2D pose and the projected 2D pose of the excavator is used as the reprojection error loss, and its expression is as follows: ; In the formula: This represents the reprojection error loss, used to describe the pixel deviation between the projected 2D pose and the predicted 2D pose of the target object. Indicates the index of the pose key points of the target object; Indicates the target object's first The weights corresponding to each pose key point; Represents a robust loss function; Represents the two-dimensional attitude coordinates obtained by reprojection; This represents the two-dimensional attitude coordinates of the target object predicted by the two-dimensional attitude prediction model; This represents the L2 norm, used to describe the Euclidean distance between two points.

[0082] S53: To alleviate monocular depth and morphological ambiguity and stabilize loss convergence, a link length constraint is applied to the predicted 3D keypoints, and a bone length consistency constraint is introduced. Together with the reprojection loss mentioned above, they form a hybrid loss function, the expression of which is as follows: ; In the formula: This represents the bone length consistency loss, used to describe the deviation loss of the change in link length between adjacent frames; This indicates the total number of image frames; Indicates the total number of links in the target object; Indicates the sequence number of the image frame; Indicates the serial number of the link in the target object; Indicates the first The target object in the frame image The length of the connecting rod; Indicates the first The target object in the frame image The length of the connecting rod.

[0083] The value is 6, representing the following links: the link formed by attitude point boom_base and attitude point tail, with a bone length of... The link formed by attitude points boom_end and boom_base has a bone length of... The link formed by the attitude point stick_base and the attitude point boom_end has a bone length of... The link formed by attitude point stick_end and attitude point stick_base has a bone length of... The link formed by the attitude point bucket_left and the attitude point stick_end has a bone length of... The link formed by the attitude point bucket_right and the attitude point stick_end has a bone length of... .

[0084] S54: The reprojection error loss and bone length consistency loss are weighted and summed to form a hybrid supervision for the deep neural network that improves the 2D to 3D pose, which is then used for training to obtain the excavator's final 3D pose. ; In the formula: The hyperparameter representing the degree of control constraint is used to balance the relative contributions of reprojection error loss and bone length consistency loss. The range of values ​​is .

[0085] This represents the training loss objective function of the 3D reconstruction model.

[0086] Hybrid loss functions enable deep neural networks to learn to reduce the time variation of link length while learning basic principles, thus avoiding random jitter at key points of the excavator.

[0087] Step S6: Transform the excavator's final three-dimensional attitude coordinates from the unified coordinate system on site to its own local coordinate system (the origin of the local coordinate system is the attitude key point boom_base), simplify the description of the excavator's motion state and construct motion constraint equations. Use Taylor series expansion of the motion constraint equations to predict the trajectory of its attitude key points, forming the excavator's dynamic danger proximity area. Use the developed construction safety monitoring intelligent agent to monitor the excavator's dynamic danger proximity area in the digital twin system in real time, realizing closed-loop monitoring of "perception-assessment-early warning / control".

[0088] S61: The final three-dimensional attitude coordinates of the excavator are transformed from the unified coordinates on site to its own local coordinate system using a coordinate transformation formula, as follows: ; In the formula: Indicates the sequence number of the key points of the target object's pose; This represents the angle between the local coordinate system of the target object and the unified coordinate system of the site along the X-axis. This represents the X-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system. This represents the Y-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system; This represents the Z-axis coordinate of the q-th attitude key point of the target object in the unified coordinate system on site; The X-axis represents the x-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; The Y-axis represents the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; This represents the Z-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; X represents the horizontal axis of the unified coordinate system on site; Y represents the vertical axis of the unified coordinate system on site; Z represents the deep axis of the unified on-site coordinate system; X q Let x be the horizontal axis of the local coordinate system of the target object; Y q The vertical axis of the local coordinate system of the target object; Z q This is the deep axis of the local coordinate system of the target object.

[0089] S62: The excavator's motion posture is simplified to a motion posture described by variable joint angles. In this embodiment of the invention, it is easy to see that the key points of the excavator's posture can be approximated as X in its local coordinate system. q Y q On a plane, the motion posture constraints of the excavator can be simplified as follows: Figure 5 shown , , , The four angles represent: The link formed by attitude point boom_base and attitude point stick_base and the local coordinate system X q The included angle between the axes is expressed as: ; The link formed by attitude point stick_base and attitude point stick_end and the local coordinate system X q The included angle between the axes is expressed as: ; The plane formed by the attitude points stick_end, bucket_left, and bucket_right intersects with the local coordinate system X. q The included angle between the axes is expressed as: ; Excavator local coordinate system X q The angle between the axis and the X-axis of the unified coordinate system on site is expressed as: ; In the formula: This represents the Y-axis coordinate of the attitude point stick_base; This represents the Y-axis coordinate of the attitude point boom_base; This represents the X-axis coordinate of the attitude point stick_base; This represents the X-axis coordinate of the attitude point boom_base; This represents the Y-axis coordinate of the attitude point stick_end; This represents the Y-axis coordinate of the attitude point stick_base; This represents the X-axis coordinate of the attitude point stick_end; This represents the X-axis coordinate of the attitude point stick_base; This represents the Y-axis coordinate of the attitude point bucket_left; This represents the Y-axis coordinate of the attitude point bucket_right; This represents the X-axis coordinate of the attitude point bucket_left; This represents the X-axis coordinate of the attitude point bucket_right; This represents the Z-axis coordinate of the attitude point stick_base; This represents the Z-axis coordinate of the attitude point boom_base.

[0090] S63: Consider the reaction time of the operator when operating the excavator. The trajectory that the excavator might move within its reaction time is considered as a dangerous adjacent area; based on the continuous three-dimensional coordinates of the excavator's posture, the following can be calculated: , , , The angular velocity and angular acceleration of the excavator trajectory can be considered as separate functions for each angle, and the predicted trajectory can be predicted by the sum of its Taylor series. Furthermore, the higher-order terms of the Taylor series of the excavator trajectory are independent. In this case, only the first and second terms of the Taylor series need to be considered, i.e., the angular velocity. and angular acceleration The expressions for Taylor series, angular velocity, and angular acceleration are shown below: ; ; ; In the formula: Indicates the current time of the trajectory prediction; The index representing the order of the summation is a non-negative integer; This represents the time increment, which is equivalent to the reaction time of the target object. A function representing time; express The function at time The First-order time derivative; Indicates the time step; This represents the first derivative of the angle with respect to time, i.e., time interval. Angular velocity, in rad / s or ° / s; A function representing the change of angle with time, i.e., time interval. Angle; Indicates time Angle; Indicates time Angle This indicates the change in angle within the most recent time step; The second derivative of the angle with respect to time, i.e., time interval. Angular acceleration, in rad / s 2 or ° / s 2 ; time angular velocity; It represents the change in angular velocity within the most recent time step.

[0091] S64: Use Taylor series to predict the motion posture of the excavator described by the variable joint angles, and obtain the dynamic danger proximity area of ​​the excavator.

[0092] S65: The three-dimensional attitude and predicted trajectory of the excavator are transmitted to the digital twin in real time. The safety monitoring agent monitors the dynamic danger proximity areas of each excavator in real time. When the proximity overlaps, the safety monitoring agent sends an early warning / control command to the digital twin, and then the digital twin sends the command to the physical entity of the excavator, and the physical entity executes the command.

[0093] The functional modules such as the Yolov11-pose model, deep neural network, convolutional neural network, neural radiation field network, holographic generation network, horizontal difference convolution module, vertical difference convolution module, diagonal difference convolution module, central difference convolution module and convolution module, slicing module, annotation module, video slicing module, video segment annotation module, video frame extraction module, image annotation module, annotation format conversion module, dataset partitioning module, large visual model, two-dimensional pose prediction model, three-dimensional reconstruction model, and early warning module can all adopt the functional modules in the existing technology, or adopt the functional modules in the existing technology and construct them using conventional technical means.

[0094] Ultimately, through the above steps, real-time closed-loop monitoring of safety for targets in complex underground cavern groups can be achieved.

[0095] The embodiments described above are only used to illustrate the technical ideas and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The patent scope of the present invention should not be limited by these embodiments. That is, any equivalent changes or modifications made in accordance with the spirit disclosed in the present invention still fall within the patent scope of the present invention.

Claims

1. A method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents, characterized in that, This method includes the following steps: Step 1: Deploy on-site monitoring cameras and obtain relevant parameters of the on-site monitoring cameras and a unified coordinate system on-site through on-site calibration and measurement. Step 2: Construct a digital twin of the target object, a data processing module, and a safety monitoring intelligent agent. The data processing module includes: a video slicing module, a video segment annotation module, a video frame extraction module, an image annotation module, an annotation format conversion module, and a dataset partitioning module. The safety monitoring intelligent agent includes: a large visual model for filtering out valid video segments containing the target object from video segments; a two-dimensional attitude prediction model for predicting the two-dimensional attitude data of the target object from image data containing the target object; a three-dimensional reconstruction model for reconstructing the two-dimensional attitude data of the target object into three-dimensional attitude data of the target object; a regional hazard assessment module for predicting the three-dimensional attitude trajectory of the target object and constructing a dynamic hazard proximity area; and an early warning module for monitoring and warning of construction safety distances. Step 3: The construction video of the underground cavern is collected by the monitoring camera to create training samples; the collected video is segmented into segments, the video segments are labeled, and the pre-trained visual model is fine-tuned using the labeled video segments; the fine-tuned visual model is used to process the video segments and extract the video segments containing the target object. Step 4: Extract frames from the video clip containing the target object, process them into image sequence data, and label them. Use the labeled image sequence data to train the two-dimensional pose prediction model. Step 5: The trained 2D pose prediction model is used to predict the image sequence data to obtain the predicted 2D pose sequence data; the predicted 2D pose sequence data is used to train the 3D reconstruction model, and the 3D pose data output by the 3D reconstruction model is used as the intermediate 3D pose data. Step 6: Based on the camera parameters, use the pinhole-distortion projection model to project the intermediate 3D pose data into 2D pose data; incorporate the distance error between the 2D pose data predicted by the 2D pose prediction model and the 2D pose data projected by the pinhole-distortion projection model into the training loss of the 3D reconstruction model. Step 7: During construction, the monitoring camera collects real-time video of the construction site, segments the video into segments and inputs them into the fine-tuned visual model. The visual model extracts video segments containing the target object, extracts frames from the video segments containing the target object and processes them into image sequence data, and then inputs them into the trained two-dimensional pose prediction model. The predicted two-dimensional pose sequence data output by the two-dimensional pose prediction model is input into the trained three-dimensional reconstruction model, and the three-dimensional reconstruction model predicts the three-dimensional pose data of the target object in real time. Step 8: Real-time predicted 3D attitude data and position trajectory of the target object are synchronized to its digital twin in real time; the real-time 3D attitude coordinates of the target object are transformed from the unified coordinate system on site to the local coordinate system of the target object itself through coordinate transformation; the regional hazard assessment module uses Taylor series analysis to predict the trajectory of the target object and form a dynamic hazard adjacent area in real time. Step 9: The early warning module monitors the dynamic dangerous proximity zone distance of the digital twin in real time and implements early warning control to achieve real-time safety monitoring of the target object.

2. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 1 includes the following steps: Manually calibrating the calibration board at multiple locations within the monitoring camera area, including the four corners, center, and near, middle, and far points, is performed. Simultaneously, measuring tools are used to acquire the coordinates of multiple control points on-site, establishing the origin and a unified coordinate system for the field. ; Using existing tools and procedures, based on the pinhole-distortion projection model, and using multiple sets of collected data, the camera intrinsic parameters were analyzed. Distortion coefficient and camera external parameters Solve for camera extrinsic parameters. For each frame of multiple sets of calibration data, the corresponding data is calculated, and then the average value is taken to determine the result. ； ； ； In the formula: Represents camera intrinsic parameters, which are 3×3 matrices describing imaging geometry and pixel scale; Indicating camera extrinsic parameters involves unifying the coordinate system of the site. The parameters when transforming points in the coordinate system to the camera coordinate system; This represents a 3×3 rotation orthogonal matrix used to describe the orientation from the field unified coordinate system to the camera coordinate system; This represents a 3×1 translation vector used to describe the position of the origin of the unified field coordinate system in the camera coordinate system; The horizontal focal length in pixels of the camera; Represents the camera's vertical focal length in pixels; Indicates the horizontal coordinates of the camera's principal point; Indicates the longitudinal coordinates of the camera's principal point; This represents the camera skew coefficient, used to describe the non-orthogonality between pixel axes; Represents the lateral coordinates of the undistorted normalized image plane; Represents the normalized vertical coordinates of the undistorted image plane; Represents the normalized lateral coordinates of the image plane after distortion; Represents the normalized vertical coordinates of the image plane after distortion; Represents the normalized radius from the undistorted normalized image plane coordinates to the camera coordinates; , , Indicates the camera's radial distortion coefficient; , Indicates the camera's tangential distortion coefficient; Represents a point in a unified coordinate system on site; This represents a point in the camera coordinate system.

3. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 2 includes the following steps: Based on measurement data and structural parameters of construction equipment, a digital twin of the site is constructed; A two-dimensional pose prediction model is constructed based on Yolov11-pose, and a three-dimensional reconstruction model is constructed based on deep neural networks, convolutional neural networks, neural radiation field networks, or holographic generative networks.

4. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 3, characterized in that, The method for constructing a two-dimensional pose prediction model based on Yolov11-pose includes the following steps; Auxiliary reversible branches are introduced into the backbone network of Yolov11-pose to form a dual backbone architecture. The main branch is responsible for routine feature extraction and prediction, while the auxiliary branch injects complete input information during the loss calculation stage to improve the gradient and alleviate the information bottleneck. By preserving the gradient and local information and reducing the training memory usage, the network's ability to extract image features is enhanced, and the accuracy and stability of the model are improved in scenes with dust, occlusion, and poor lighting conditions. The first layer of the dual-backbone network uses augmented detail convolutional modules, which include horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules. During the training phase, the augmented detail convolutional modules are composed of horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules connected in parallel. During the working inference phase, the parallel horizontal difference convolutional modules, vertical difference convolutional modules, diagonal difference convolutional modules, central difference convolutional modules, and convolutional modules are equivalently merged into a standard convolution.

5. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 3 includes the following sub-steps: Step 3-1: The video slicing module uses Python to randomly segment the collected on-site monitoring video, with the segmentation time being greater than or equal to a set threshold, forming several video clips; the video clip annotation module annotates the segmented video clips accordingly, while the annotation format conversion module uses Python to convert the file format, thereby creating dataset A for fine-tuning the large-scale visual model; the dataset partitioning module uses Python to partition dataset A into a training set, a validation set, and a test set, with a ratio of 7:2:1 for each subset; when creating dataset A, the annotation content includes the target object in the video clip and the timestamp of the corresponding excavator image, which helps guide the large-scale visual model to identify the target object and whether the target object exists in the video frame; Step 3-2: Input dataset A into the visual big model to train and fine-tune the visual big model. After fine-tuning, the visual big model can identify target objects and output timestamps of valid video segments. Based on the timestamps output by the visual big model, the video slicing module is further called to segment the random monitoring video segments, thereby deleting redundant video segments that do not contain target objects and retaining valid video segments that contain target objects.

6. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 4 includes the following steps: The video frame extraction module extracts effective video segments output by the large visual model and processes them into image data. The image annotation module then uses the image data to create dataset B for training the 2D pose prediction model. The annotation content includes the bounding box of the target object and pose key points. The dataset partitioning module divides dataset B into training, validation, and test sets. It uses dataset B to train a two-dimensional pose prediction model, which outputs two-dimensional pose data of the target object. The optimized model performance is evaluated using average precision, recall, and / or F1 score metrics to obtain and output robust two-dimensional pose data of the target object.

7. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 6 includes the following sub-steps: Step 6-1: Using the known camera parameters, project the intermediate 3D pose data of the target object through a pinhole-distortion projection model to generate 2D pose data. Construct the reprojection error loss between the predicted 2D pose data and the projected 2D pose data of the target object, and its expression is as follows: ； In the formula: This represents the reprojection error loss, used to describe the pixel deviation between the projected 2D pose and the predicted 2D pose of the target object. Indicates the index of the pose key points of the target object; Indicates the target object's first The weights corresponding to each pose key point; Represents a robust loss function; Represents the two-dimensional attitude coordinates obtained by reprojection; This represents the two-dimensional attitude coordinates of the target object predicted by the two-dimensional attitude prediction model; This represents the L2 norm, used to describe the Euclidean distance between two points; Step 6-2: Introduce a bone length consistency constraint to help the hybrid loss function converge, the expression of which is as follows: ； In the formula: This represents the bone length consistency loss, used to describe the deviation loss of the change in link length between adjacent frames; This indicates the total number of image frames; Indicates the total number of links in the target object; Indicates the sequence number of the image frame; Indicates the serial number of the link in the target object; Indicates the first The target object in the frame image The length of the connecting rod; Indicates the first The target object in the frame image The length of the connecting rod; Step 6-3: Weight the reprojection error loss and bone length consistency loss to form a hybrid loss function, which is used as the training loss for the 3D reconstruction model: Then we have: ； In the formula: The hyperparameter representing the degree of control constraint is used to balance the relative contributions of reprojection error loss and bone length consistency loss. The training loss objective function of the 3D reconstruction model; The hybrid loss function enables the 3D reconstruction model to learn to reduce the time variation of link length, thereby avoiding random jitter of key points of the target object.

8. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, The 3D reconstruction model consists of several sequentially connected reconstruction modules. Each reconstruction module is composed of four parts: Conv1D layer, BatchNormalization layer, ReLU layer, and Dropout layer.

9. The method for monitoring the construction safety of underground cavern groups based on digital twin intelligent agents according to claim 1, characterized in that, Step 8 includes the following sub-steps: Step 8-1: Transform the final 3D attitude coordinates of the target object from the unified coordinate system on site to its own local coordinate system using the coordinate transformation formula, as follows: ； In the formula: Indicates the sequence number of the key points of the target object's pose; This represents the angle between the local coordinate system of the target object and the unified coordinate system of the site along the X-axis. This represents the X-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system. This represents the Y-axis coordinate of the q-th attitude key point of the target object in the unified field coordinate system; This represents the Z-axis coordinate of the q-th attitude key point of the target object in the unified coordinate system on site; The X-axis represents the x-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; The Y-axis represents the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; This represents the Z-axis of the q-th pose keypoint of the target object in its own local coordinate system. q Axis coordinates; X represents the horizontal axis of the unified coordinate system on site; Y represents the vertical axis of the unified coordinate system on site; Z represents the deep axis of the unified on-site coordinate system; X q Let x be the horizontal axis of the local coordinate system of the target object; Y q The vertical axis of the local coordinate system of the target object; Z q The deep axis of the local coordinate system of the target object; Step 8-2: Simplify the target object's motion posture into a motion posture described by changing joint angles, while considering the target object's reaction time. The trajectory of the target object's expected movement within the reaction time is considered as a dangerous adjacent area; a Taylor series is used to fit a trajectory with a higher probability based on the current motion state. The Taylor series expression is as follows: ； In the formula: Indicates the current time of the trajectory prediction; The index representing the order of the summation is a non-negative integer; This represents the time increment, which is equivalent to the reaction time of the target object. A function representing time; express The function at time The First-order time derivative; Step 8-3: Use Taylor series to predict the motion posture of the target object described by the variable joint angles, and obtain the dynamic danger proximity area of ​​the target object; Step 8-4: Transmit the 3D attitude data and predicted trajectory of the target object to the digital twin in real time. The early warning module monitors the dynamic danger proximity area of ​​each target object in real time. When the dynamic danger proximity areas of any two or more target objects overlap, intersect, or intrude into each other, the early warning module sends an early warning and control command to the corresponding digital twin. The corresponding digital twin sends action instructions to the corresponding physical entity of the target object, and the physical entity executes the corresponding action according to the action instructions.

10. A device for monitoring the construction safety of underground cavern groups based on a digital twin intelligent agent, comprising a memory and a processor, characterized in that, The memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the underground cavern group construction safety monitoring method based on a digital twin intelligent agent as described in any one of claims 1 to 9.

Images