A Three-Dimensional Deformation Monitoring System and Method for Tunnel Surrounding Rock Integrating Monocular Vision and Millimeter-Wave Radar

Abstract

This invention relates to a tunnel surrounding rock three-dimensional deformation monitoring system integrating monocular vision and millimeter-wave radar, including in-tunnel sensing equipment, tunnel entrance terminal, database and application server, and remote monitoring terminal. This invention also relates to a tunnel surrounding rock three-dimensional deformation monitoring method integrating monocular vision and millimeter-wave radar, including: S1, setting monitoring targets; S2, multimodal data acquisition; S3, visual and radar target detection; S4, visual and radar heterogeneous data matching; S5, solving for pixel displacement and radial displacement; S6, calculating the three-dimensional deformation of the surrounding rock; and S7, alarm judgment and result output. This invention can significantly improve the accuracy, real-time performance, and automation level of tunnel surrounding rock deformation monitoring, and is particularly suitable for scenarios with complex environments and high safety risks in the early stages of tunnel excavation; aiming to provide a monitoring method for tunnel surrounding rock deformation that is adaptable to the early stages of excavation and is efficient, accurate, and automated.

Description

Technical Field

[0001] This invention belongs to the field of tunnel construction safety monitoring technology, specifically relating to a three-dimensional deformation monitoring system and method for tunnel surrounding rock that integrates monocular vision and millimeter-wave radar. Background Technology

[0002] Due to the complex construction environment and variable geological conditions, tunnel engineering is highly susceptible to surrounding rock deformation during excavation, which can lead to serious accidents such as crown cracking, collapse, and excessive settlement. Especially in sections with poor geological conditions, the exposed and turbulent surrounding rock before the implementation of stabilization support represents the period of most severe deformation and instability, posing a significant safety threat. Therefore, real-time and accurate deformation monitoring of the surrounding rock during tunnel excavation is crucial for ensuring the safety of workers and maintaining the stability of the surrounding rock.

[0003] Currently, conventional methods for monitoring tunnel surrounding rock deformation still mainly rely on total stations, levels, and convergence instruments. These methods are not only inefficient but also highly dependent on manual operation, resulting in significant time lags in the monitoring data and making it difficult to achieve real-time, continuous deformation sensing. Furthermore, in high-risk and complex environments with low light, high dust levels, poor ventilation, or the presence of toxic or harmful gases, conventional manual methods are insufficient to effectively ensure the safety of workers. Although burying deformation monitoring sensors can achieve continuous monitoring, in the early stages of tunnel excavation or before the support structure is formed, due to unstable surrounding rock, limited installation space, and high operational risks, sensors are difficult to deploy effectively, often resulting in "window periods" in monitoring critical deformation stages.

[0004] Millimeter-wave radar, with its advantages of all-weather operation, strong anti-interference capability, and ability to penetrate dust and smoke, is gradually being applied to the field of geological disaster monitoring. However, it can only capture the projected component of deformation along the radial direction, and its ability to reconstruct the spatial displacement field is limited, which can easily lead to an underestimation of the actual deformation. On the other hand, monocular vision technology has the advantages of low cost and high resolution image acquisition capability, but due to the lack of depth information, its spatial resolution capability is insufficient, making it difficult to accurately reflect three-dimensional deformation.

[0005] Therefore, in order to fully leverage the advantages of both and overcome their respective limitations, this invention proposes a three-dimensional deformation monitoring system and method for tunnel surrounding rock that integrates monocular vision and millimeter-wave radar. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide a three-dimensional deformation monitoring system and method for tunnel surrounding rock that integrates monocular vision and millimeter-wave radar. This system can significantly improve the accuracy, real-time performance, and automation level of tunnel surrounding rock deformation monitoring, and is particularly suitable for scenarios with complex environments and high safety risks in the early stages of tunnel excavation. The aim is to provide a monitoring method for tunnel surrounding rock deformation that is adaptable to the early stages of excavation and is efficient, accurate, and automated.

[0007] The technical problem solved by this invention is achieved through the following technical solution:

[0008] A tunnel surrounding rock three-dimensional deformation monitoring system integrating monocular vision and millimeter-wave radar includes in-tunnel sensing equipment, tunnel entrance terminal, database and application server, and remote monitoring terminal;

[0009] The in-cave sensing equipment includes a millimeter-wave radar and a monocular camera. The millimeter-wave radar and the monocular camera are mounted on a base via connectors and are both connected to a local area network via an Ethernet interface. The in-cave sensing equipment collects radar data and image data of the monitoring area in real time. The video stream collected by the monocular camera is transmitted in real time via the RTSP protocol. The millimeter-wave radar outputs multi-dimensional monitoring data of echo intensity, phase, distance, and angle using a serial port or UDP protocol.

[0010] The tunnel entrance terminal is an industrial-grade PC or edge computing server, used to receive, parse and standardize multimodal raw data from the sensing devices inside the tunnel, unify data streams of different protocols into a standardized data structure, realize synchronous acquisition and preprocessing of multi-source data, and upload the structured data to the database and application server.

[0011] The database and application server serve as the core data processing platform of the system, responsible for the joint spatial and temporal calibration of multimodal data, heterogeneous data fusion, 3D deformation calculation, and monitoring result management. This enables efficient storage, time-series querying, intelligent analysis, and result output of monitoring data. It supports the automatic generation of structured monitoring files for target detection and deformation analysis both locally and in the cloud, and maintains complete synchronization and status logs. Based on the alarm thresholds preset in the system management interface, it automatically alarms and judges the real-time calculated 3D deformation and related indicators. Once the monitoring data exceeds the limits, it immediately generates alarm information and pushes it synchronously to the remote monitoring terminal, achieving automatic alarm and emergency response.

[0012] The remote monitoring terminal is used for real-time visualization of monitoring data, historical data backtracking, alarm information reception and remote management. The construction site and regulatory departments can use the client to monitor the deformation status of the surrounding rock in real time and respond to alarms immediately.

[0013] During the initial deployment or operation of the system, spatial and temporal joint calibration is required. Spatial calibration involves setting up a calibration board in the monitoring area and using a dedicated program to collect observation data from both a monocular camera and millimeter-wave radar. The terminal device calculates the rigid transformation parameters (rotation matrix and translation vector) between the two coordinate systems. The calibration parameters are stored in the form of configuration files or databases on the application server or the local terminal device and can be imported, queried, and updated through the system management interface for subsequent data coordinate transformation and spatial alignment. Temporal synchronization uses low-frequency sensors as a reference and interpolates the timestamps of high-frequency sensor data to achieve spatiotemporal consistency of multi-source data.

[0014] A method for monitoring three-dimensional deformation of tunnel surrounding rock integrating monocular vision and millimeter-wave radar, comprising the steps of the aforementioned system for monitoring three-dimensional deformation of tunnel surrounding rock integrating monocular vision and millimeter-wave radar, is as follows:

[0015] S1. Monitoring target setting: Dedicated targets with high visual contrast and strong radar echo capability are deployed in the monitoring area. Target information can be queried and managed in the system management interface.

[0016] S2. Multimodal data acquisition: Simultaneously acquire raw observation data from millimeter-wave radar and monocular camera of each target. After standardization processing by the terminal, the data is uploaded to the database and application server in real time.

[0017] S3. Visual and radar target detection: The server uses visual image processing methods to identify and extract visual detection boxes of monitored targets from image data, while simultaneously performing point cloud target extraction and continuous tracking from radar data.

[0018] S4. Visual and radar heterogeneous data matching: Based on the joint spatial and temporal calibration parameters, the radar detection results are projected onto the visual coordinate system, and based on the spatial position and feature intensity information, an accurate correspondence between the visual detection box and the radar observation point cloud is established, and a unique ID is assigned to each monitoring target.

[0019] S5. Solving for pixel displacement and radial displacement: Calculate the pixel displacement and radial displacement of the monitored target respectively;

[0020] S6. Calculation of three-dimensional deformation of surrounding rock: By integrating multimodal observation data and based on geometric relationships and optimization algorithms, the three-dimensional deformation of the monitoring target is calculated and structured monitoring results are generated.

[0021] S7. Alarm Judgment and Result Output: The server automatically judges the 3D deformation and related indicators according to the preset alarm threshold. If the limit is exceeded, an alarm message is immediately generated and pushed to the remote monitoring terminal. At the same time, the monitoring results are output, and historical query and data archiving are supported.

[0022] Furthermore, the dedicated target of S1 adopts a fluorescent corner reflector structure with fluorescent stripes on its surface, and the main body is an aluminum corner reflector; after the target is deployed, relevant target information can be queried and managed through the system management interface, including but not limited to:

[0023] 1) Initial target deployment parameters (deployment location, physical properties, deployment time, person in charge);

[0024] 2) A unique ID automatically generated after target detection and matching;

[0025] 3) Real-time and historical deformation monitoring results corresponding to the target;

[0026] 4) Other system records related to the target status (such as alarm information, detection history, parameter correction records).

[0027] Moreover, S3 specifically refers to:

[0028] S31, Obtain the precise detection frame coordinates of the monitoring target.

[0029] The server uses the initial target image as the template image. For each candidate region of the acquired monitoring images The normalized correlation coefficient (NCC) method was used to calculate similarity. The similarity calculation formula is as follows:

[0030] ;

[0031] in: Template image; Image of the region to be matched; This represents the displacement of the sliding window; The width and height of the template image; and These are the average pixel values ​​of the template and the region to be matched, respectively.

[0032] By traversing all candidate regions, the location with the highest similarity is determined as the initial location of the monitoring target. Based on the initial detection results, the similarity response is finely located at the sub-pixel level using spline interpolation within the detection area to obtain the precise detection box coordinates of the monitoring target in the image.

[0033] S32, acquire radar point cloud of monitored target

[0034] The server analyzes the data collected by the millimeter-wave radar to obtain the reflection intensity of each target point. Phase information radial distance and spatial angle (azimuth) With pitch angle For multiple time-series frames of data, the above parameters are extracted to form the target feature vector. By using gating mechanisms and target association algorithms, target feature vectors in consecutive frames are associated and matched to track the same monitoring target at different times and output radar point clouds and their parameters.

[0035] Moreover, S4 specifically refers to:

[0036] S41, Radar point cloud projection and coordinate transformation

[0037] Based on the spatial transformation parameters obtained from the joint calibration of the system, the server automatically transforms the radar point cloud data from the radar coordinate system and projects it to the visual image coordinate system, obtaining the projection coordinates of each radar point in the image. ;

[0038] S42, Space-Intensity Joint Matching

[0039] For each visual detection bounding box, obtain its center point coordinates. and normalized average gray value For each radar projection point, extract its normalized reflection intensity. ;

[0040] The following spatial-intensity joint matching degree function is used for association determination:

[0041] ;

[0042] in: It is the first The coordinates of the projection points of each radar point in the image coordinate system; For the first The coordinates of the center point of each target detection box; Normalized radar reflection intensity; This represents the normalized average gray value of the detection frame.

[0043] in: This represents the spatial distance tolerance threshold. These are the weighting coefficients;

[0044] The judgment condition is: if ( (For the matching threshold), then determine the radar point. With the target detection box Related;

[0045] S43, Target Unique ID Assignment

[0046] For each successfully associated monitoring target, its position station on the tunnel axis is calculated based on the radar spatial location and measurement data. Within the cross-section corresponding to the same station number, each target is numbered sequentially in a clockwise direction. The system automatically assigns a unique identifier ID (in the format "..."). - The system will input the target ID, coordinates, deployment parameters, etc. into the system database to support subsequent querying, modification and tracking.

[0047] Moreover, S5 specifically refers to:

[0048] The precise bounding box coordinates of the monitored target in the initial and subsequent frames are extracted respectively. By calculating the coordinate difference between the two frames, the pixel displacement of the target in the image plane is obtained. For radar point cloud sets that successfully match the visual inspection boxes ( (For the matching point index set), extract the phase information of each radar point at the initial time and subsequent time, denoted as . and According to the radar operating wavelength Calculate the radial displacement of the radar point using the following formula. :

[0049] ;

[0050] After calculating the radial displacement of all matched radar points, the final radial displacement of the monitored target is calculated using the following weighted average formula. :

[0051] ;

[0052] in: For the first The reflection intensity of each radar point; These are the corresponding weighting factors.

[0053] Moreover, S6 specifically refers to:

[0054] S61, based on pixel displacement, initial depth, and camera intrinsic parameters, the following geometric model is used to monitor the pixel coordinate changes of the target. With three-dimensional space deformation Related:

[0055] ;

[0056] in: The initial depth of the target point, estimated using radar observation, is:

[0057] ;

[0058] To measure the initial radial distance for radar; , These are the azimuth and elevation angles measured by radar, respectively. , This refers to the camera's focal length parameter; To monitor the initial pixel coordinates of the target;

[0059] The coordinates of the camera's principal point;

[0060] S62, based on the radial geometry of the radar, establish the radial displacement. With three-dimensional space deformation Geometric relationship between them:

[0061] ;

[0062] in: To monitor the initial three-dimensional coordinates of the target and Calculated from pixel coordinates and initial depth:

[0063] ;

[0064] S63, Based on the above relationships, define a joint objective function to include the three-dimensional deformation. Let this be the optimization variable, and the objective function be:

[0065] ;

[0066] in: ; , , These are the visual and radar geometric residuals, respectively. , , , Weights for each item;

[0067] S64, regarding the initial depth variable in the objective function Set boundary constraints, i.e. Only The range of values ​​is , where This represents the initial depth measurement error range;

[0068] S65 employs the trust-region reflection algorithm to optimize the objective function and outputs the optimal three-dimensional deformation solution. The results are automatically generated into structured monitoring files and stored in a database for easy retrieval and analysis of historical data.

[0069] Furthermore, S7 specifically involves: the server automatically alarming and judging the above-mentioned three-dimensional spatial deformation and related monitoring indicators based on the alarm threshold preset in the system management interface; if the deformation amount or change rate of any target exceeds the set threshold, the system immediately generates alarm information and pushes it in real time through the remote monitoring terminal to realize automatic alarm and emergency response; at the same time, all monitoring and alarm information is archived in the database, supporting users to query, count and analyze alarm history.

[0070] The advantages and beneficial effects of this invention are as follows:

[0071] 1. Improved monitoring accuracy and real-time performance: This invention integrates monocular vision and millimeter-wave radar to achieve high-precision, continuous, and real-time monitoring of three-dimensional deformation of surrounding rock.

[0072] 2. Enhanced environmental adaptability: This invention can operate stably under complex working conditions such as high dust and low light, and is especially suitable for special stages such as the initial stage of excavation, when conventional safety monitoring equipment cannot be deployed, and when support is not yet completed, which significantly expands the scope of application of surrounding rock monitoring.

[0073] 3. Reduce human safety risks: Automate and remotely control the monitoring process, reduce personnel entry into dangerous work areas, reduce human error and operational risks, and improve the safety level of the construction site. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the integrated monitoring system of the present invention;

[0075] Figure 2 This is a flowchart of the present invention;

[0076] Figure 3 This is a schematic diagram of the target structure of the present invention. Detailed Implementation

[0077] The present invention will be further described in detail below through specific embodiments. The following embodiments are merely descriptive and not limiting, and should not be used to limit the scope of protection of the present invention.

[0078] The tunnel surrounding rock three-dimensional deformation monitoring system integrating monocular vision and millimeter-wave radar of the present invention, such as Figure 1 As shown, it consists of four parts: in-cave sensing equipment, in-cave terminal, database and application server, and remote monitoring terminal.

[0079] (a) In-cave sensing equipment

[0080] In a preferred embodiment of the present invention, the cave sensing device includes a monocular camera and a millimeter-wave radar. Preferably, the monocular camera is an industrial-grade CMOS camera (e.g., using a SONY IMX283 sensor), equipped with an F1.4 fixed-focus lens, preferably with a focal length of 50 mm (or an 8-75 mm lens depending on actual needs), and a sampling frequency of up to 30 FPS, suitable for temperatures from −10°C to 60°C; the millimeter-wave radar is an 80.5 GHz FMCW millimeter-wave radar with a bandwidth of 2 GHz, a maximum monitoring distance of up to 300 m, a distance monitoring accuracy better than 0.15 m, a radial displacement monitoring accuracy of up to 0.1 mm, a horizontal angle resolution of 1°, a pitch angle resolution of 1.5°, a monitoring frequency of 30 Hz, and can operate in environments from −40°C to 75°C.

[0081] The millimeter-wave radar and monocular camera are securely mounted on a unified base using specialized custom connectors (such as aluminum alloy structures or high-strength composite materials). Mechanical adjustment ensures that their line-of-sight is parallel and consistent, improving the spatial registration accuracy of heterogeneous data. The mounting base can be fixed using adsorption or expansion bolts, facilitating flexible deployment in different areas such as tunnel arches and sidewalls.

[0082] Both the millimeter-wave radar and the monocular camera are connected to a unified local area network via an Ethernet interface, enabling high-speed data interaction with the tunnel entrance terminal. The video stream acquired by the monocular camera is transmitted to the tunnel entrance terminal in real time using the RTSP protocol; the millimeter-wave radar outputs multi-dimensional monitoring data such as echo intensity, phase, radial distance, and spatial angle in real time via serial port or UDP protocol.

[0083] (ii) Tunnel entrance terminal

[0084] The tunnel entrance terminal is an industrial-grade PC or edge computing server, pre-installed with customized data acquisition and processing software. Connected to the tunnel's sensing devices via Ethernet, the terminal automatically identifies the network addresses and data ports of monocular cameras and millimeter-wave radar equipment. Its data processing program supports multi-threaded asynchronous acquisition, real-time parsing, format conversion, and standardization of received raw data streams from different protocols, uniformly organizing them into JSON or a custom binary data structure, enabling synchronous acquisition and preprocessing of multi-source data. The processed structured data is automatically uploaded to a database and application server via Ethernet, achieving seamless integration with upper-level data management and analysis platforms. The terminal also supports automatic data caching and breakpoint resume functionality, ensuring the continuity and reliability of tunnel monitoring data. All data acquisition and transmission processes can be configured and monitored via the system management interface, facilitating system maintenance and troubleshooting.

[0085] (III) Database and Application Server

[0086] The database and application server utilize enterprise-grade servers and relational databases. The server automatically receives standardized multimodal data uploaded from the portal terminal, completing spatial and temporal joint calibration, heterogeneous data fusion, and 3D deformation calculation. All monitoring data and calculation results are efficiently stored in the database, supporting rapid querying by time, target, and other criteria. The server can automatically generate structured monitoring files such as target detection and deformation analysis, and archive status logs. The system management interface supports alarm threshold settings. The server automatically judges real-time 3D deformation and related indicators, immediately generating alarm information when limits are exceeded and pushing it to the remote monitoring terminal in real time via the network, realizing automatic alarm and emergency linkage.

[0087] (iv) Remote monitoring terminal

[0088] The remote monitoring terminal is a PC-based or web-based visualization software platform. The client connects to the application server via Ethernet or 4G / 5G networks, allowing users to view surrounding rock deformation monitoring data, 3D deformation analysis results, and alarm status in real time. The system interface supports data curve, model, or table display, and users can query historical data and alarm records by target, time interval, and other criteria. The client has alarm pop-ups and push notifications, facilitating timely responses from construction sites and regulatory authorities. The system also supports remote parameter configuration and access control, enabling remote operation and management of the monitoring system.

[0089] Before the system is first installed or run, joint spatial and temporal calibration must be completed. For spatial calibration, a standard calibration board is used as the target. The board is manually moved to different spatial locations, and at least 20 frames of images and point cloud data are collected by a monocular camera and millimeter-wave radar, respectively. The terminal at the tunnel entrance runs a dedicated calibration algorithm to process the collected multi-frame data, automatically calculating the rigid transformation parameters (including rotation matrices and translation vectors) between the camera and radar coordinate systems to achieve accurate spatial coordinate mapping. These calibration parameters can be stored in configuration files or a database and can be imported, queried, and updated in the system management interface for subsequent spatial alignment of multimodal data. For temporal synchronization, the system uses the data collected by sensors with lower acquisition frequencies (such as cameras) as the time reference. Data collected by high-frequency sensors (such as millimeter-wave radar) is synchronized using timestamp interpolation to ensure spatiotemporal consistency of multi-source data at the same time.

[0090] A method for monitoring three-dimensional deformation of tunnel surrounding rock integrating monocular vision and millimeter-wave radar, implemented using the aforementioned system, is as follows: Figure 2 As shown, the specific steps include:

[0091] S1. Monitoring Target Setup: Select key areas to be observed within the monitoring area and deploy dedicated monitoring targets. Preferably, the target employs a fluorescent corner reflector structure with fluorescent stripes on its surface to enhance visual detection contrast. The main body is an aluminum corner reflector to enhance the echo signal strength of the millimeter-wave radar. The target structure is as follows: Figure 3 As shown. The target deployment location and number can be flexibly adjusted according to monitoring needs, ensuring that both the vision system and radar system can stably detect the same target. After each target is deployed, staff can enter deployment parameters through the system management interface, including deployment location, physical attributes (such as model and size), deployment time, and responsible person. After target detection and data matching are completed (i.e., step S4), the system automatically assigns a unique ID to each target and associates the ID with target deployment parameters, spatial coordinates, and other information, entering it into the system database. Real-time deformation monitoring data and historical data for each target (output in step S6) can be queried at any time through the system management interface. The system also supports querying and managing target status-related information, such as alarm information, detection history, and parameter corrections, achieving full lifecycle information management. All data supports export and access control, facilitating engineering traceability and quality control.

[0092] S2. Multimodal Data Acquisition: The monocular camera and millimeter-wave radar in the system are connected to the tunnel entrance terminal via Ethernet, automatically and synchronously acquiring raw observation data of each monitored target. The monocular camera periodically acquires high-definition video streams, with an acquisition frequency that can be set to 1~30 frames / second; the millimeter-wave radar outputs monitoring data including echo intensity, phase, distance, and angle at a frequency of 30 Hz. The terminal equipment performs protocol parsing and format conversion on the raw data, structuring it and automatically uploading it to the database and application server. The application server categorizes and stores the uploaded data, creating indexes according to fields such as target number and acquisition time, supporting real-time centralized management and subsequent efficient queries, providing a data foundation for subsequent data fusion and 3D deformation processing.

[0093] S3, visual and radar target detection:

[0094] S31, Obtain the precise detection frame coordinates of the monitoring target.

[0095] The server saves the initial target image as a template image. In each frame of the image, a sliding window approach is used to traverse the candidate regions across the entire image. For each candidate region... With template image According to the following Formula for calculating similarity:

[0096] ;

[0097] in, Template image; Image of the region to be matched; This represents the displacement of the sliding window; The width and height of the template image; and The average pixel values ​​of the template and the region to be matched are respectively.

[0098] Select The location of the maximum value is used as the initial detection box area of ​​the monitoring target. Spline interpolation (such as cubic spline) is performed on the similarity distribution within this area to achieve fine positioning of the detection box coordinates at the sub-pixel level, and output the detection results and confidence level.

[0099] S32, acquire radar point cloud of monitored target

[0100] At the corresponding time, the raw data acquired by the millimeter-wave radar is analyzed to extract the reflection intensity of each target point. Phase information radial distance and spatial angle (azimuth) With pitch angle The parameters are combined to form the target feature vector. For multi-frame data, a gating mechanism (such as setting a maximum motion range threshold) and a target association algorithm (such as Hungarian matching or Kalman filtering) are used to track and number the data points of the same physical target in different frames, and output the radar point cloud parameters detected in each frame.

[0101] S4, visual and radar heterogeneous data matching:

[0102] S41, Radar point cloud projection and coordinate transformation

[0103] Based on the spatial calibration parameters obtained during system deployment, the server automatically transforms the radar point cloud data from the radar coordinate system and projects it to the visual image coordinate system, obtaining the projection coordinates of each radar point in the image. .

[0104] S42, Space-Intensity Joint Matching

[0105] For each visual detection bounding box, obtain its center point coordinates. and normalized average gray value For each radar projection point, extract its normalized reflection intensity. ;

[0106] The following spatial-intensity joint matching degree function is used for association determination:

[0107] ;

[0108] in: It is the first The coordinates of the projection points of each radar point in the image coordinate system; For the first The coordinates of the center point of each target detection box; Normalized radar reflection intensity; This represents the normalized average gray value of the detection frame. This represents the spatial distance tolerance threshold. These are the weighting coefficients.

[0109] The judgment condition is: if ( (For the matching threshold), then determine the radar point. With the target detection box Related.

[0110] S43, Target Unique ID Assignment

[0111] For each successfully associated monitoring target, its position station on the tunnel axis is calculated based on the radar spatial location and measurement data. Within the cross-section corresponding to the same station number, each target is numbered sequentially in a clockwise direction. The system automatically assigns a unique identifier ID (in the format "..."). - The system will input the target ID, coordinates, deployment parameters, etc. into the system database to support subsequent querying, modification and tracking.

[0112] S5. Pixel and Radial Displacement Calculation: The system automatically extracts the precise detection box coordinates of the target in the initial and subsequent frame images. By calculating the coordinate difference between the two frames, the pixel displacement of the target in the image plane is obtained. For radar point cloud sets that successfully match the visual inspection boxes ( (For the matching point index set), extract the phase information of each radar point at the initial time and subsequent time, denoted as . and According to the radar operating wavelength Calculate the radial displacement of the radar point using the following formula. :

[0113] ;

[0114] After calculating the radial displacement of all matched radar points, the final radial displacement of the monitored target is calculated using the following weighted average formula. :

[0115] ;

[0116] in: For the first The reflection intensity of each radar point; These are the corresponding weighting factors.

[0117] S6. Calculation of three-dimensional deformation of surrounding rock:

[0118] S61, based on pixel displacement, initial depth, and camera intrinsic parameters, the following geometric model is used to monitor the pixel coordinate changes of the target. With three-dimensional space deformation Related:

[0119] ;

[0120] in: The initial depth of the target point, estimated using radar observation, is:

[0121] ;

[0122] To measure the initial radial distance for radar; , These are the azimuth and elevation angles measured by radar, respectively. , This refers to the camera's focal length parameter; To monitor the initial pixel coordinates of the target; These are the coordinates of the camera's principal point.

[0123] S62, based on the radial geometry of the radar, establish the radial displacement. With three-dimensional space deformation Geometric relationship between them:

[0124] ;

[0125] in: To monitor the initial three-dimensional coordinates of the target, and Calculated from pixel coordinates and initial depth:

[0126] ;

[0127] S63, Based on the above relationships, define a joint objective function to include the three-dimensional deformation. Let this be the optimization variable, and the objective function be:

[0128] ;

[0129] in: ; , , These are the visual and radar geometric residuals, respectively. , , , The weights are for each item.

[0130] S64, regarding the initial depth variable in the objective function Set boundary constraints, i.e. Only The range of values ​​is , where This represents the initial depth measurement error range.

[0131] S65 employs the trust-region reflection algorithm to optimize the objective function and outputs the optimal three-dimensional spatial deformation solution. The results are automatically generated into structured monitoring files and stored in a database for easy retrieval and analysis of historical data.

[0132] S7. Alarm Judgment and Result Output: The server automatically judges the above-mentioned three-dimensional spatial deformation and related monitoring indicators based on the alarm thresholds preset in the system management interface. If the deformation or rate of change of any target exceeds the set threshold, the system immediately generates an alarm message and pushes it in real time through the remote monitoring terminal to realize automatic alarm and emergency response; at the same time, all monitoring and alarm information is archived in the database, supporting users to query, count and analyze alarm history.

[0133] The deformation calculation error under different distances and deformation amplitudes was tested according to the method of the present invention, as shown in Table 1. As can be seen from Table 1, the average error is mostly within 1 mm, and the overall accuracy is better than 2 mm, which fully demonstrates the good adaptability and robustness of the present invention under multiple working conditions.

[0134] Table 1 Statistical results of errors for different distances and deformation amplitudes

[0135]

[0136] Although embodiments and drawings of the present invention have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the scope of the present invention is not limited to the contents disclosed in the embodiments and drawings.

Claims

1. A method for monitoring three-dimensional deformation of tunnel surrounding rock by integrating monocular vision and millimeter-wave radar, characterized in that: The monitoring system used in the method includes in-cave sensing equipment, in-cave terminal, database and application server, and remote monitoring terminal; The in-cave sensing equipment includes a millimeter-wave radar and a monocular camera. The millimeter-wave radar and the monocular camera are mounted on a base via connectors and are both connected to a local area network via an Ethernet interface. The in-cave sensing equipment collects radar data and image data of the monitoring area in real time. The video stream collected by the monocular camera is transmitted in real time via the RTSP protocol. The millimeter-wave radar outputs multi-dimensional monitoring data of echo intensity, phase, distance, and angle using a serial port or UDP protocol. The tunnel entrance terminal is an industrial-grade PC or edge computing server, used to receive, parse and standardize multimodal raw data from the sensing devices inside the tunnel, unify data streams of different protocols into a standardized data structure, realize synchronous acquisition and preprocessing of multi-source data, and upload the structured data to the database and application server. The database and application server, serving as the core data processing platform of the system, are responsible for the joint spatial and temporal calibration of multimodal data, heterogeneous data fusion, 3D deformation calculation, and monitoring result management. They enable efficient storage, time-series querying, intelligent analysis, and result output of monitoring data. They support the automatic generation of structured monitoring files for target detection and deformation analysis on both local and cloud platforms, and maintain complete synchronization and status logs. Based on the alarm thresholds preset in the system management interface, they automatically perform alarm judgments on the real-time calculated 3D deformation and related indicators. Once the monitoring data exceeds the limits, alarm information is immediately generated and synchronously pushed to the remote monitoring terminal, realizing automatic alarm and emergency linkage. The remote monitoring terminal is used for real-time visualization of monitoring data, historical data backtracking, alarm information reception and remote management. The construction site and regulatory departments can use the client to monitor the deformation status of the surrounding rock in real time and respond to alarms immediately. During the initial deployment or operation of the system, spatial and temporal joint calibration is required. Spatial calibration involves setting up a calibration board in the monitoring area and using a dedicated program to collect observation data from both a monocular camera and millimeter-wave radar. The terminal device calculates the rigid transformation parameters between the two coordinate systems. The calibration parameters are stored in the form of configuration files or databases on the application server or the local terminal device and can be imported, queried, and updated through the system management interface for subsequent data coordinate transformation and spatial alignment. Temporal synchronization uses low-frequency sensors as a reference and interpolates the timestamps of high-frequency sensor data to achieve spatiotemporal consistency of multi-source data. The steps of the method are as follows: S1. Monitoring target setting: Dedicated targets with high visual contrast and strong radar echo capability are deployed in the monitoring area. Target information can be queried and managed in the system management interface. S2. Multimodal data acquisition: Simultaneously acquire raw observation data from millimeter-wave radar and monocular camera of each target. After standardization processing by the terminal, the data is uploaded to the database and application server in real time. S3. Visual and radar target detection: The server uses visual image processing methods to identify and extract visual detection boxes of monitored targets from image data, while simultaneously performing point cloud target extraction and continuous tracking from radar data. S4. Visual and radar heterogeneous data matching: Based on the joint spatial and temporal calibration parameters, the radar detection results are projected onto the visual coordinate system, and based on the spatial position and feature intensity information, an accurate correspondence between the visual detection box and the radar observation point cloud is established, and a unique ID is assigned to each monitoring target. S5. Solving for pixel displacement and radial displacement: Calculate the pixel displacement and radial displacement of the monitored target respectively; S6. Calculation of three-dimensional deformation of surrounding rock: By integrating multimodal observation data and based on geometric relationships and optimization algorithms, the three-dimensional deformation of the monitoring target is calculated and structured monitoring results are generated. S7. Alarm Judgment and Result Output: The server automatically judges the 3D deformation and related indicators according to the preset alarm threshold. If the limit is exceeded, an alarm message is immediately generated and pushed to the remote monitoring terminal. At the same time, the monitoring results are output, and historical query and data archiving are supported. Specifically, S5 is: The precise bounding box coordinates of the monitored target in the initial and subsequent frames are extracted respectively. By calculating the coordinate difference between the two frames, the pixel displacement of the target in the image plane is obtained. For radar point cloud sets that successfully match the visual inspection boxes , To match the point index set, the phase information of each radar point at the initial and subsequent times is extracted and denoted as follows: and According to the radar operating wavelength Calculate the radial displacement of the radar point using the following formula. : ； After calculating the radial displacement of all matched radar points, the final radial displacement of the monitored target is calculated using the following weighted average formula. : ； in: For the first The reflection intensity of each radar point; For the corresponding weighting factors; Specifically, S6 is: S61, based on pixel displacement, initial depth, and camera intrinsic parameters, the following geometric model is used to monitor the pixel coordinate changes of the target. With three-dimensional space deformation Related: ； in: The initial depth of the target point, estimated using radar observation, is: ； To measure the initial radial distance for radar; , These are the azimuth and elevation angles measured by radar, respectively. , This refers to the camera's focal length parameter; To monitor the initial pixel coordinates of the target; The coordinates of the camera's principal point; S62, based on the radial geometry of the radar, establish the radial displacement. With three-dimensional space deformation Geometric relationship between them: ； in: To monitor the initial three-dimensional coordinates of the target, and Calculated from pixel coordinates and initial depth: ； S63, Based on the above relationships, define a joint objective function to include the three-dimensional deformation. Let this be the optimization variable, and the objective function be: ； in: ; , , These are the geometric residuals for visual and radar operations, respectively. , , , Weights for each item; S64, regarding the initial depth variable in the objective function Set boundary constraints, i.e. Only The range of values ​​is , where This represents the initial depth measurement error range; S65 employs the trust-region reflection algorithm to optimize the objective function and outputs the optimal three-dimensional deformation solution. The results are automatically generated into structured monitoring files and stored in a database for easy retrieval and analysis of historical data.

2. The method for monitoring three-dimensional deformation of tunnel surrounding rock by integrating monocular vision and millimeter-wave radar according to claim 1, characterized in that: The dedicated target of S1 adopts a fluorescent corner reflector structure with fluorescent stripes on the surface and an aluminum corner reflector as the main body. After the targets are deployed, target-related information can be queried and managed through the system management interface, including: 1) Initial target deployment parameters, including deployment location, physical properties, deployment time, and person in charge; 2) A unique ID automatically generated after target detection and matching; 3) Real-time and historical deformation monitoring results corresponding to the target; 4) Other system records related to the target status, including alarm information, detection history, and parameter correction records.

3. The method for monitoring three-dimensional deformation of tunnel surrounding rock by integrating monocular vision and millimeter-wave radar according to claim 1, characterized in that: Specifically, S3 is: S31, Obtain the precise detection frame coordinates of the monitoring target. The server uses the initial target image as the template image. For each candidate region of the acquired monitoring images The normalized correlation coefficient method is used to calculate similarity. The similarity calculation formula is as follows: ； in, Template image; Image of the region to be matched; This represents the displacement of the sliding window; The width and height of the template image; and These are the average pixel values ​​of the template and the region to be matched, respectively. By traversing all candidate regions, the location with the highest similarity is determined as the initial location of the monitoring target. Based on the initial detection results, the similarity response is finely located at the sub-pixel level using spline interpolation within the detection area to obtain the precise detection box coordinates of the monitoring target in the image. S32, acquire radar point cloud of monitored target The server analyzes the data collected by the millimeter-wave radar to obtain the reflection intensity of each target point. Phase information radial distance And spatial angles, including azimuth angles. With pitch angle For multiple time-series frames of data, the above parameters are extracted to form the target feature vector. By using gating mechanisms and target association algorithms, target feature vectors in consecutive frames are associated and matched to track the same monitoring target at different times and output radar point clouds and their parameters.

4. The method for monitoring three-dimensional deformation of tunnel surrounding rock by integrating monocular vision and millimeter-wave radar according to claim 1, characterized in that: Specifically, S4 is: S41, Radar point cloud projection and coordinate transformation Based on the spatial transformation parameters obtained from the joint calibration of the system, the server automatically transforms the radar point cloud data from the radar coordinate system and projects it to the visual image coordinate system, obtaining the projection coordinates of each radar point in the image. ; S42, Space-Intensity Joint Matching For each visual detection bounding box, obtain its center point coordinates. and normalized average gray value For each radar projection point, extract its normalized reflection intensity. ; The following spatial-intensity joint matching degree function is used for association determination: ； in: It is the first The coordinates of the projection points of each radar point in the image coordinate system; For the first The coordinates of the center point of each target detection box; Normalized radar reflection intensity; This represents the normalized average gray value of the detection frame. This represents the spatial distance tolerance threshold. These are the weighting coefficients; The judgment condition is: if , To match the threshold, determine the radar point. With the target detection box Related; S43, Target Unique ID Assignment For each successfully associated monitoring target, its position station on the tunnel axis is calculated based on the radar spatial location and measurement data. Within the cross-section corresponding to the same station number, each target is numbered sequentially in a clockwise direction. The system automatically assigns a unique identifier ID, the format of which is " - The system also records the target ID, coordinates, and deployment parameters into the system database, supporting subsequent queries, modifications, and tracking.

5. The method for monitoring three-dimensional deformation of tunnel surrounding rock by integrating monocular vision and millimeter-wave radar according to claim 1, characterized in that: Specifically, S7 involves the server automatically alarming and judging the above-mentioned three-dimensional spatial deformation and related monitoring indicators based on the alarm threshold preset in the system management interface. If the deformation amount or change rate of any target exceeds the set threshold, the system immediately generates alarm information and pushes it in real time through the remote monitoring terminal to realize automatic alarm and emergency response. At the same time, all monitoring and alarm information is archived in the database, supporting users to query, count and analyze alarm history.

Images