Tunnel surrounding rock three-dimensional deformation monitoring system and method integrating monocular vision and millimeter wave radar

Abstract

The invention relates to a tunnel surrounding rock three-dimensional deformation monitoring system integrating monocular vision and millimeter wave radar. The tunnel surrounding rock three-dimensional deformation monitoring system comprises in-tunnel sensing equipment, a tunnel opening terminal, a database and application server and a remote monitoring end. The invention also relates to a tunnel surrounding rock three-dimensional deformation monitoring method fusing monocular vision and millimeter wave radar. The method comprises the following steps: S1, setting a monitoring target; s2, multi-modal data acquisition is carried out; s3, visual and radar target detection; s4, performing visual and radar heterogeneous data matching; s5, solving pixel displacement and radial displacement; s6, calculating the three-dimensional deformation of the surrounding rock; and S7, alarm judgment and result output. According to the invention, the precision, the real-time performance and the automation level of tunnel surrounding rock deformation monitoring can be obviously improved, and the method is especially suitable for scenes with complex environment and high safety risk in the early stage of tunnel excavation; the invention aims to provide an efficient, accurate and automatic monitoring means capable of adapting to the early stage of excavation for tunnel surrounding rock deformation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of tunnel engineering construction safety monitoring, and particularly relates to a tunnel surrounding rock three-dimensional deformation monitoring system and method fusing monocular vision and millimeter wave radar. BACKGROUND

[0002] Due to complex construction environment and variable geological conditions, tunnel engineering is prone to surrounding rock deformation during excavation, which may induce serious accidents such as vault cracking, collapse and excessive settlement. Especially after excavation in poor geological sections and before the implementation of stable support, the surrounding rock is exposed and disturbed violently, which is the most unstable period of deformation and poses a great threat to safety. Therefore, real-time and accurate deformation monitoring of surrounding rock during tunnel excavation is crucial to protect the lives of workers and ensure the stability of surrounding rock.

[0003] At present, the conventional tunnel surrounding rock deformation monitoring method still mainly relies on total station, level, convergence meter, etc., which not only has low detection efficiency, but also highly depends on manual operation, resulting in obvious time lag of monitoring data and difficulty in realizing real-time and continuous deformation perception. In addition, in the high-risk complex environment of low illumination, high dust, poor ventilation or toxic and harmful gases, conventional manual methods are also difficult to effectively protect the safety of workers. Although buried deformation monitoring sensors can achieve continuous monitoring, in the initial stage of tunnel excavation or when the support structure has not yet formed, due to the instability of surrounding rock, limited installation space and high operation risk, the sensors are difficult to be effectively deployed, resulting in a "window period" of monitoring in the key deformation stage.

[0004] Millimeter wave radar has gradually been applied to geological disaster monitoring due to its all-weather, strong anti-interference ability and ability to penetrate dust and smoke, but it can only capture the projection component of deformation along the radial direction, and the reconstruction ability of spatial displacement field is limited, which may lead to underestimation of the actual deformation. Monocular vision technology has low cost and high resolution image acquisition capability, but it lacks depth information and has insufficient spatial resolution, making it difficult to accurately reflect three-dimensional deformation.

[0005] Therefore, in order to fully utilize the advantages of both and overcome their limitations, the present application proposes a tunnel surrounding rock three-dimensional deformation monitoring system and method fusing monocular vision and millimeter wave radar. SUMMARY

[0006] The present application aims to overcome the shortcomings of the prior art and provide a tunnel surrounding rock three-dimensional deformation monitoring system and method fusing monocular vision and millimeter wave radar, which can significantly improve the accuracy, real-time performance and automation level of tunnel surrounding rock deformation monitoring, especially suitable for the initial stage of tunnel excavation, complex environment and high safety risk scenarios. The present application aims to provide a monitoring method that is efficient, accurate and automated for tunnel surrounding rock deformation in the initial stage of excavation.

[0007] The technical problem of the present application is solved by the following technical scheme: A tunnel surrounding rock three-dimensional deformation monitoring system fusing monocular vision and millimeter wave radar, comprising an in-cave sensing device, a portal terminal, a database and an application server, and a remote monitoring end; The in-cave sensing device comprises a millimeter wave radar and a monocular camera, the millimeter wave radar and the monocular camera are installed on a base through a connecting piece and are connected to a local area network through an Ethernet interface; the in-cave sensing device collects radar data and image data of a monitoring area in real time, the video stream collected by the monocular camera is transmitted in real time through an RTSP protocol, and the millimeter wave radar outputs multi-dimensional monitoring data of echo intensity, phase, distance and angle through a serial port or a UDP protocol; The portal terminal is an industrial PC or an edge computing server, which is used for receiving, analyzing and standardizing processing of multi-modal original data from the in-cave sensing device, unifying data streams of different protocols into a standardized data structure, realizing synchronous collection and preprocessing of multi-source data, and uploading the structured data to the database and the application server; The database and the application server serve as a core data processing platform of the system, responsible for spatial and temporal joint calibration of multi-modal data, heterogeneous data fusion, three-dimensional deformation calculation and monitoring result management, realizing efficient storage, time sequence query, intelligent analysis and result output of monitoring data; supporting automatic generation of local and cloud target detection, deformation analysis and other structured monitoring files, and maintaining complete synchronization and state logs; according to the alarm threshold preset by the system management interface, automatically alarming and judging the three-dimensional deformation and related indexes calculated in real time, and once the monitoring data exceeds the limit, alarm information is immediately generated and pushed to the remote monitoring end, realizing automatic alarm and emergency linkage; The remote monitoring end is used for real-time visualization of monitoring data, historical data backtracking, alarm information receiving and remote management, and the construction site and supervision department can master the surrounding rock deformation state in real time through the client and respond to the alarm in the first time; During initial deployment or operation of the system, spatial and temporal joint calibration is required, spatial calibration is performed by setting a calibration board in the monitoring area, and observation data of the monocular camera and the millimeter wave radar are collected respectively by using a special program, rigid transformation parameters (rotation matrix and translation vector) between coordinate systems of the two are calculated by the terminal device, the calibration parameters are stored in the application server or the local terminal device in the form of a configuration file or a database, and can be imported, queried and updated through the system management interface, used for subsequent data coordinate conversion and spatial alignment; time synchronization is based on a low-frequency sensor, and time synchronization is realized by time stamp interpolation of high-frequency sensor data.

[0008] A tunnel surrounding rock three-dimensional deformation monitoring method fusing monocular vision and millimeter wave radar, adopting a tunnel surrounding rock three-dimensional deformation monitoring system fusing monocular vision and millimeter wave radar, steps of the method are: S1, monitoring target setting: a special target with high visual contrast and strong radar echo capability is arranged in a monitoring area, and target information is inquired and managed in a system management interface; S2, multi-modal data acquisition: millimeter wave radar and monocular camera original observation data of each target are synchronously acquired, and the data is uploaded to a database and an application server in real time after terminal standardization processing; S3, visual and radar target detection: the server side identifies and extracts a visual detection frame of the monitoring target by a visual image processing method, and simultaneously performs point cloud target extraction and continuous tracking on radar data; S4, visual and radar heterogeneous data matching: the radar detection result is projected to a visual coordinate system relying on space and time joint calibration parameters, and an accurate corresponding relationship between the visual detection frame and the radar observation point cloud is established based on spatial position and feature intensity information, and a unique ID identifier is allocated to each monitoring target; S5, pixel displacement and radial displacement solving: pixel displacement and radial displacement of the monitoring target are respectively calculated; S6, surrounding rock three-dimensional deformation calculation: multi-modal observation data are fused, three-dimensional deformation of the monitoring target is calculated based on geometric relationship and an optimization algorithm, and structured monitoring results are generated; S7, alarm judgment and result output: the server side automatically performs alarm judgment on three-dimensional deformation and related indexes according to a preset alarm threshold, generates alarm information and pushes to a remote monitoring end immediately if there is overrun, simultaneously outputs monitoring results, supports historical inquiry and data archiving.

[0009] Moreover, the special target of S1 adopts a fluorescent corner reflector structure, a surface is provided with a fluorescent strip, and a main body is an aluminum corner reflector; after the target is arranged, target related information is inquired and managed through a system management interface, including but not limited to: 1) target initial arrangement parameters (arrangement position, physical property, arrangement time, person in charge); 2) a unique ID automatically generated after target detection and matching; 3) real-time and historical deformation monitoring results corresponding to the target; 4) other system records related to the state of the target (such as alarm information, detection history, parameter correction record).

[0010] Moreover, S3 specifically is: S31, obtaining accurate detection frame coordinates of the monitoring target The server side takes an initial monitoring target image as a template image For each candidate region of the acquired monitoring images The normalized correlation coefficient (NCC) method was used to calculate similarity. The similarity calculation formula is as follows: ; in: Template image; Image of the region to be matched; This represents the displacement of the sliding window; The width and height of the template image; and These are the average pixel values ​​of the template and the region to be matched, respectively. By traversing all candidate regions, the location with the highest similarity is determined as the initial location of the monitoring target. Based on the initial detection results, the similarity response is finely located at the sub-pixel level using spline interpolation within the detection area to obtain the precise detection box coordinates of the monitoring target in the image. S32, acquire radar point cloud of monitored target The server analyzes the data collected by the millimeter-wave radar to obtain the reflection intensity of each target point. Phase information radial distance and spatial angle (azimuth) With pitch angle For multiple time-series frames of data, the above parameters are extracted to form the target feature vector. By using gating mechanisms and target association algorithms, target feature vectors in consecutive frames are associated and matched to track the same monitoring target at different times and output radar point clouds and their parameters.

[0011] Moreover, S4 specifically refers to: S41, Radar point cloud projection and coordinate transformation Based on the spatial transformation parameters obtained from the joint calibration of the system, the server automatically transforms the radar point cloud data from the radar coordinate system and projects it to the visual image coordinate system, obtaining the projection coordinates of each radar point in the image. ; S42, Space-Intensity Joint Matching For each visual detection bounding box, obtain its center point coordinates. and normalized average gray value For each radar projection point, extract its normalized reflection intensity. ; The following spatial-intensity joint matching degree function is used for association determination: ; in: It is the first The coordinates of the projection points of each radar point in the image coordinate system; For the first The coordinates of the center point of each target detection box; Normalized radar reflection intensity; This represents the normalized average gray value of the detection frame. in: This represents the spatial distance tolerance threshold. These are the weighting coefficients; The judgment condition is: if ( (For the matching threshold), then determine the radar point. With the target detection box Related; S43, Target Unique ID Assignment For each successfully associated monitoring target, its position station on the tunnel axis is calculated based on the radar spatial location and measurement data. Within the cross-section corresponding to the same station number, each target is numbered sequentially in a clockwise direction. The system automatically assigns a unique identifier ID (in the format "..."). - The system will input the target ID, coordinates, deployment parameters, etc. into the system database to support subsequent querying, modification and tracking.

[0012] Moreover, S5 specifically refers to: The precise bounding box coordinates of the monitored target in the initial and subsequent frames are extracted respectively. By calculating the coordinate difference between the two frames, the pixel displacement of the target in the image plane is obtained. For radar point cloud sets that successfully match the visual inspection boxes ( (For the matching point index set), extract the phase information of each radar point at the initial time and subsequent time, denoted as . and According to the radar operating wavelength Calculate the radial displacement of the radar point using the following formula. : ; After calculating the radial displacement of all matched radar points, the final radial displacement of the monitored target is calculated using the following weighted average formula. : ; in: For the first The reflection intensity of each radar point; These are the corresponding weighting factors.

[0013] Further, the S6 is specifically: S61, according to the pixel displacement, the initial depth and the camera internal parameter, the pixel coordinate change of the monitoring target is changed by using the following geometric model And the three-dimensional space deformation Correlation: ; Wherein: The initial depth of the target point is estimated by using radar observation: ; The initial radial distance measured by the radar is: , The azimuth angle and the pitch angle measured by the radar are respectively: , The camera focal length parameter is: The initial pixel coordinate of the monitoring target is: The camera principal point coordinate is: S62, according to the radial geometric relationship of the radar, the radial displacement And the three-dimensional space deformation Geometric relationship between them: ; Wherein: The initial three-dimensional coordinate of the monitoring target And Is calculated from the pixel coordinate and the initial depth: ; S63, according to the above relationship, the joint objective function is defined, the three-dimensional deformation amount Is set as the optimization variable, and the objective function is: ; Wherein: ; , , The visual and radar geometric relationship residual are respectively: , , , The weight of each item is: S64, the initial depth variable in the objective function Boundary constraints are set, that is Only in the range of , wherein, The initial depth measurement error range is: S65, the trust domain reflection algorithm is used to optimize and solve the objective function, and the optimal three-dimensional deformation solution is output And the result is automatically generated a structured monitoring file, stored in the database, facilitating subsequent historical data retrieval and analysis.

[0014] Moreover, the S7 is specifically: the server end according to the alarm threshold value preset by the system management interface, the three-dimensional space deformation and related monitoring index are automatically alarmed and judged;If any target deformation or change rate exceeds the set threshold, the system immediately generates alarm information, and real-time push is carried out through the remote monitoring end, automatic alarm and emergency response are realized;At the same time, all monitoring and alarm information are archived in the database, which supports the user to query, statistics and analysis of alarm history.

[0015] The advantages and beneficial effects of the present application are: 1、Improve monitoring accuracy and real-time performance: the present application combines monocular vision and millimeter wave radar to realize high-precision, continuous real-time monitoring of three-dimensional deformation of surrounding rock.

[0016] 2、Enhance environmental adaptability: the present application can operate stably in complex working conditions such as large dust and low illumination, and is especially suitable for special stages such as initial excavation, unable to lay conventional safety monitoring equipment, and incomplete support, which significantly expands the application range of surrounding rock monitoring.

[0017] 3、Reduce artificial safety risk: realize the automation and remote of monitoring process, reduce the personnel into dangerous operation surface, reduce the human error and operation risk, improve the safety guarantee level of construction site. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a schematic diagram of the fusion monitoring system of the present application; Figure 2 is a flow chart of the present application; Figure 3 is a schematic diagram of the target structure of the present application. DETAILED DESCRIPTION

[0019] The present application will be further described in detail below through specific embodiments, the following embodiments are only descriptive, not limiting, and the protection scope of the present application cannot be limited by this.

[0020] The tunnel surrounding rock three-dimensional deformation monitoring system of the present application fuses monocular vision and millimeter wave radar, as shown in Figure 1 , including in-hole sensing device, hole terminal, database and application server, remote monitoring end four parts.

[0021] (I) In-hole sensing device In a preferred embodiment of the present application, the in-hole sensing device comprises a monocular camera and a millimeter wave radar. Preferably, the monocular camera is an industrial-grade CMOS camera (such as using a SONY IMX283 sensor), equipped with a F1.4 fixed-focus lens, preferably with a focal length of 50 mm (or alternatively, a lens with a focal length in the range of 8-75 mm can be selected according to actual needs), with a capture frequency of up to 30 FPS, suitable for temperatures from -10°C to 60°C; the millimeter wave radar is an 80.5 GHz operating frequency FMCW millimeter wave radar, with a bandwidth of 2 GHz, a maximum monitoring distance of up to 300 m, a distance monitoring accuracy of better than 0.15 m, a radial displacement monitoring accuracy of up to 0.1 mm, a horizontal angle resolution of 1°, a pitch angle resolution of 1.5°, a monitoring frequency of 30 Hz, and can work in an environment from -40°C to 75°C.

[0022] The millimeter wave radar and the monocular camera are firmly mounted on a unified base through a dedicated custom connector (such as an aluminum alloy structure or a high-strength composite material), and the visual axes of the two are guaranteed to be parallel and consistent through mechanical adjustment, thereby improving the spatial registration accuracy of heterogeneous data. The mounting base can be fixed by suction or expansion bolts, facilitating flexible deployment in different areas such as the arch of the tunnel and the side wall.

[0023] The millimeter wave radar and the monocular camera are both connected to a unified local area network through an Ethernet interface, enabling high-speed data interaction with the portal terminal. The video stream captured by the monocular camera is transmitted in real time to the portal terminal using the RTSP protocol; the millimeter wave radar outputs multi-dimensional monitoring data such as echo intensity, phase, radial distance, and spatial angle in real time through a serial port or UDP protocol.

[0024] (II) Portal terminal The portal terminal is an industrial-grade PC or edge computing server, and is pre-installed with customized data acquisition and processing software. The portal terminal is connected to the in-hole sensing device through Ethernet, and can automatically identify the network address and data port of the monocular camera and millimeter wave radar devices. The data processing program running on the portal terminal supports multi-threaded asynchronous acquisition, can analyze, format convert, and standardize different protocol raw data streams received in real time, and uniformly organizes them into JSON or custom binary data structures, thereby realizing synchronous acquisition and preprocessing of multi-source data. The structured data after processing can be automatically uploaded to the database and application server through Ethernet, realizing seamless connection with the upper-layer data management and analysis platform. The terminal also supports automatic data caching and breakpoint resume functions, ensuring the continuity and reliability of tunnel monitoring data. All data acquisition and transmission processes can be parameter-configured and state-monitored through the system management interface, facilitating system maintenance and troubleshooting.

[0025] (III) Database and application server The database and application server adopt enterprise-level servers and relational databases. The server automatically receives the standardized multi-modal data uploaded by the opening terminal, completes the joint calibration of space and time, heterogeneous data fusion and three-dimensional deformation calculation. All monitoring data and calculation results are efficiently stored in the database, supporting fast query according to time, target and other conditions. The server can automatically generate structured monitoring files such as target detection and deformation analysis, and archive state logs. The system management interface supports alarm threshold setting. The server automatically alarms and judges the real-time three-dimensional deformation and related indicators. When the limit is exceeded, an alarm information is immediately generated and pushed to the remote monitoring end in real time through the network, realizing automatic alarm and emergency linkage.

[0026] (Four) Remote monitoring end The remote monitoring end is a visual software platform based on PC or Web. The client connects with the application server through Ethernet or 4G / 5G network, supporting users to view real-time surrounding rock deformation monitoring data, three-dimensional deformation analysis results and alarm status. The system interface supports data curve, model or table display. Users can query historical data and alarm records according to target, time interval and other conditions. The client has alarm pop-up and push functions, which is convenient for the construction site and supervision department to respond in time. The system also supports remote parameter configuration and permission management, realizing remote operation and management of the monitoring system.

[0027] Before the system is installed or operated for the first time, joint calibration of space and time needs to be completed. During space calibration, a standard calibration board is used as a target. The calibration board is manually moved to different spatial positions, and at least 20 frames of image and point cloud data are collected by monocular camera and millimeter wave radar respectively. The opening terminal runs a special calibration algorithm to process the collected multi-frame data, automatically calculates the rigid transformation parameters (including rotation matrix and translation vector) between the camera and radar coordinate systems, and realizes accurate mapping of spatial coordinates. The above calibration parameters can be stored through configuration file or database, and can be imported, queried and updated in the system management interface, which is used for subsequent multi-modal data space alignment. In terms of time synchronization, the system takes the sensor with lower collection frequency (such as camera) as the time reference. The data collected by high-frequency sensors (such as millimeter wave radar) are synchronized through timestamp interpolation method, ensuring the spatio-temporal consistency of multi-source data at the same time.

[0028] A three-dimensional deformation monitoring method for tunnel surrounding rock combining monocular vision and millimeter wave radar is realized by means of the foregoing system, and the method flow is as shown in Figure 2 The specific steps include: S1, monitoring target setting: select the key parts that need to be observed in the monitoring area, and lay out special monitoring targets. Preferably, the target adopts a fluorescent corner reflector structure, and the surface is provided with a fluorescent strip to enhance the visual detection contrast. The main body is an aluminum corner reflector to enhance the echo signal strength of the millimeter wave radar. The target structure is shown in Figure 3 . The target layout position and quantity can be flexibly adjusted according to the monitoring requirements to ensure that the visual system and the radar system can stably detect the same target. After the target is laid out, the staff can input the layout parameters through the system management interface, including the layout position, physical properties (such as model, size), layout time, responsible person and other information. After the target detection and data matching are completed (i.e. S4 step), the system automatically allocates a unique ID for each target, and associates the ID with the target layout parameters, spatial coordinates and other information, and inputs them into the system database. The real-time deformation monitoring data and historical data of each target (S6 step output) can be queried at any time through the system management interface. The system also supports the query and management of target state related information, such as alarm information, detection history, parameter correction, etc., to realize the information management of the whole life cycle. All data support export and permission management, which is convenient for engineering traceability and quality control.

[0029] S2, multi-modal data acquisition: the monocular camera and the millimeter wave radar in the system are connected with the portal terminal through Ethernet, and automatically synchronously acquire the original observation data of each monitoring target. The monocular camera acquires high-definition video stream at a frequency of 1-30 frames per second; the millimeter wave radar outputs monitoring data including echo intensity, phase, distance, angle, etc. at a frequency of 30 Hz. The terminal device performs protocol analysis and format conversion on the original data, and automatically uploads the structured data to the database and application server. The application server stores the uploaded data in categories, indexes them according to the target number, acquisition time and other fields, supports real-time centralized management and subsequent efficient query, and provides data basis for subsequent data fusion and three-dimensional deformation processing.

[0030] S3, visual and radar target detection: S31, obtaining the accurate detection box coordinates of the monitoring target The server saves the initial monitoring target image as a template image , and in each image, a sliding window is used to traverse the candidate regions in the whole image. For each candidate region , the similarity is calculated with the template image according to the following formula: ; wherein, is the template image; is the region image to be matched; displacement of the sliding window; width and height of the template image; and pixel mean of the template and the region to be matched respectively selecting the position of the maximum value as the preliminary detection box region of the monitoring target, performing spline interpolation (such as cubic spline) on the similarity distribution in the region to realize fine positioning of the detection box coordinates at the sub-pixel level, and outputting the detection result and the confidence.

[0031] S32, obtaining the radar point cloud of the monitoring target At the corresponding moment, the original data collected by the millimeter wave radar is parsed, and the reflection intensity , phase information , radial distance , and spatial angle (azimuth angle and pitch angle ) of each target point are extracted. The parameters are combined into a target feature vector . For multiple frames of data, a gating mechanism (such as setting a maximum motion range threshold) and a target association algorithm (such as Hungarian matching or Kalman filtering) are used to track and number the data points of the same physical target in different frames, and the radar point cloud parameters detected in each frame are output.

[0032] S4, matching of visual and radar heterogeneous data: S41, radar point cloud projection and coordinate conversion According to the spatial calibration parameters obtained during system deployment, the server automatically converts and projects the radar point cloud data from the radar coordinate system to the visual image coordinate system to obtain the projection point coordinates of each radar point in the image .

[0033] S42, joint space-intensity matching For each visual detection box, the center point coordinates and the normalized average gray value are obtained; for each radar projection point, its normalized reflection intensity is extracted. The following joint space-intensity matching degree function is used for correlation determination: ; Wherein: is the projection point coordinates of the i-th radar point in the image coordinate system; is the center point coordinates of the i-th target detection box; is the normalized radar reflection intensity; is the normalized detection box average gray value; is the normalized radar reflection intensity; is the normalized detection box average gray value; This represents the spatial distance tolerance threshold. These are the weighting coefficients.

[0034] The judgment condition is: if ( (For the matching threshold), then determine the radar point. With the target detection box Related.

[0035] S43, Target Unique ID Assignment For each successfully associated monitoring target, its position station on the tunnel axis is calculated based on the radar spatial location and measurement data. Within the cross-section corresponding to the same station number, each target is numbered sequentially in a clockwise direction. The system automatically assigns a unique identifier ID (in the format "..."). - The system will input the target ID, coordinates, deployment parameters, etc. into the system database to support subsequent querying, modification and tracking.

[0036] S5. Pixel and Radial Displacement Calculation: The system automatically extracts the precise detection box coordinates of the target in the initial and subsequent frame images. By calculating the coordinate difference between the two frames, the pixel displacement of the target in the image plane is obtained. For radar point cloud sets that successfully match the visual inspection boxes ( (For the matching point index set), extract the phase information of each radar point at the initial time and subsequent time, denoted as . and According to the radar operating wavelength Calculate the radial displacement of the radar point using the following formula. : ; After calculating the radial displacement of all matched radar points, the final radial displacement of the monitored target is calculated using the following weighted average formula. : ; in: For the first The reflection intensity of each radar point; These are the corresponding weighting factors.

[0037] S6. Calculation of three-dimensional deformation of surrounding rock: S61, based on pixel displacement, initial depth, and camera intrinsic parameters, the following geometric model is used to monitor the pixel coordinate changes of the target. With three-dimensional space deformation Related: ; wherein: is the initial depth of the target point, which is estimated by radar observation: ; is the initial radial distance of the radar measurement; , are the azimuth and elevation angles measured by the radar, respectively; , is the camera focal length parameter; is the initial pixel coordinate of the monitoring target; is the camera principal point coordinate.

[0038] S62, according to the radial geometric relationship of the radar, the geometric relationship between the radial displacement and the three-dimensional space deformation is established: ; wherein: is the initial three-dimensional coordinate of the monitoring target, and are calculated from the pixel coordinate and the initial depth: ; S63, according to the above relationship, the joint objective function is defined, and the three-dimensional deformation is set as the optimization variable, and the objective function is: ; wherein: ; , , are the geometric relationship residuals of vision and radar, respectively; , , , are the weights of each item.

[0039] S64, the initial depth variable in the objective function is set as a boundary constraint, that is, only takes values in the range of , wherein, is the initial depth measurement error range.

[0040] S65, the trust region reflection algorithm is used to optimize and solve the objective function, and the optimal three-dimensional space deformation solution is output, and the result is automatically generated into a structured monitoring file and stored in a database, which is convenient for subsequent historical data retrieval and analysis.

[0041] S7, alarm judgment and result output: the server end automatically judges the three-dimensional space deformation and related monitoring indexes according to the alarm threshold preset by the system management interface. If the deformation or change rate of any target exceeds the set threshold, the system immediately generates an alarm information and pushes it in real time through the remote monitoring end, realizing automatic alarm and emergency response; at the same time, all monitoring and alarm information is archived in the database, supporting user query, statistics and analysis of alarm history.

[0042] According to the method of the application, the deformation calculation error under different distances and deformation amplitudes is obtained, and is specifically shown in Table 1. As shown in Table 1, the average error is within 1 mm, and the overall accuracy is better than 2 mm, which fully illustrates the good adaptability and robustness of the application under multiple working conditions.

[0043] Table 1: Error statistics results of different distances and different deformation amplitudes

[0044] Although the embodiments of the application and the drawings are disclosed for the purpose of illustration, those skilled in the art can understand that various substitutions, changes and modifications are possible without departing from the spirit and scope of the application and the appended claims, therefore, the scope of the application is not limited to the disclosed contents of the embodiments and the drawings.

Claims

1. A tunnel surrounding rock three-dimensional deformation monitoring system fusing monocular vision and millimeter wave radar, characterized in that: It comprises in-hole sensing devices, portal terminals, databases and application servers, and remote monitoring terminals. The in-hole sensing devices comprise millimeter wave radars and monocular cameras, which are installed on a base through connecting members and connected to a local area network through Ethernet interfaces; the in-hole sensing devices collect radar data and image data of a monitoring area in real time, the video stream collected by the monocular cameras is transmitted in real time through an RTSP protocol, and the millimeter wave radars output multi-dimensional monitoring data of echo intensity, phase, distance and angle through a serial port or a UDP protocol; The portal terminals are industrial-grade PCs or edge computing servers, which are used to receive, analyze and standardize the multi-modal raw data from the in-hole sensing devices, unify the data streams of different protocols into a standardized data structure, realize the synchronous collection and preprocessing of multi-source data, and upload the structured data to the databases and application servers; The databases and application servers serve as the core data processing platform of the system, responsible for the spatial and temporal joint calibration of multi-modal data, the fusion of heterogeneous data, the three-dimensional deformation calculation and the management of monitoring results, realizing the efficient storage, time sequence query, intelligent analysis and result output of monitoring data; supporting the automatic generation of local and cloud target detection, deformation analysis and other structured monitoring files, and maintaining complete synchronization and status logs; according to the alarm threshold preset by the system management interface, automatically alarming and judging the three-dimensional deformation and related indicators calculated in real time, and once the monitoring data exceeds the limit, generating alarm information and synchronously pushing it to the remote monitoring terminal to realize automatic alarm and emergency linkage; The remote monitoring terminal is used for real-time visualization of monitoring data, historical data backtracking, alarm information receiving and remote management, and the construction site and supervision departments can master the surrounding rock deformation state in real time through the client and respond to the alarm in the first time; During the initial deployment or operation of the system, spatial and temporal joint calibration is required, spatial calibration is performed by setting calibration plates in the monitoring area, and observation data of the monocular camera and the millimeter wave radar are collected respectively by using a special program, and the rigid transformation parameters (rotation matrix and translation vector) between the coordinate systems of the two are calculated by the terminal device, the calibration parameters are stored in the application server or the local terminal device in the form of a configuration file or a database, and can be imported, queried and updated through the system management interface, used for subsequent data coordinate conversion and spatial alignment; time synchronization is based on low-frequency sensors, and the time stamps of high-frequency sensor data are interpolated to realize the spatio-temporal consistency of multi-source data.

2. A tunnel surrounding rock three-dimensional deformation monitoring method fusing monocular vision and millimeter wave radar, characterized by: The method comprises the following steps: S1, monitoring target setting: special targets with high visual contrast and strong radar echo capability are arranged in the monitoring area, and the target information is queried and managed in the system management interface; S2, multi-modal data collection: millimeter wave radar and monocular camera raw observation data of each target are collected synchronously, and the data are uploaded to the databases and application servers in real time after standardized processing by the terminal. S3, visual and radar target detection: the server side identifies and extracts the visual detection frame of the monitoring target through visual image processing method, and extracts and continuously tracks the point cloud target of the radar data; S4, visual and radar heterogeneous data matching: relying on the space and time joint calibration parameters, the radar detection results are projected into the visual coordinate system, and based on the spatial position and feature intensity information, the accurate correspondence between the visual detection frame and the radar observation point cloud is established, and a unique ID identifier is assigned to each monitoring target; S5, pixel displacement and radial displacement solving: the pixel displacement and radial displacement of the monitoring target are calculated respectively; S6, surrounding rock three-dimensional deformation calculation: fusion of multi-modal observation data, based on geometric relationship and optimization algorithm, the three-dimensional deformation of the monitoring target is calculated and the structured monitoring result is generated; S7, alarm judgment and result output: the server side automatically judges the three-dimensional deformation and related indicators according to the preset alarm threshold, and if there is an overrun, alarm information is immediately generated and pushed to the remote monitoring end, and the monitoring result is output, supporting historical query and data archiving.

3. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: The special target of S1 adopts a fluorescent corner reflector structure, and a fluorescent strip is arranged on the surface, and the main body is an aluminum corner reflector; after the target is laid, the related information of the target is queried and managed through the system management interface, including but not limited to: 1) initial laying parameters of the target (laying position, physical properties, laying time, responsible person); 2) unique ID automatically generated after target detection and matching; 3) real-time and historical deformation monitoring results corresponding to the target; 4) other system records related to the state of the target (such as alarm information, detection history, parameter correction record).

4. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: S3 is specifically: S31, obtaining the accurate detection frame coordinates of the monitoring target The server end takes the initial monitoring target image as a template image , each candidate region of the collected monitoring image , similarity calculation is performed by using a normalized correlation coefficient method (NCC), and a similarity calculation formula is as follows: ； wherein, is a template image; is a region image to be matched; is a displacement of the sliding window; is a width and height of the template image; and are respectively a mean of pixels of the template and the region image to be matched; By traversing all candidate regions, the position with the highest similarity is determined as the preliminary position of the monitoring target, and for the preliminary detection result, a spline interpolation method is used to finely position the similarity response at a sub-pixel level within the detection region to obtain the accurate detection frame coordinates of the monitoring target in the image; S32, obtaining the radar point cloud of the monitoring target For the data collected by the millimeter wave radar, the server side analyzes and obtains the reflection intensity of each target point , phase information , radial distance , and spatial angle (azimuth angle and elevation angle ), for multiple time sequence frame data, the above parameters are extracted respectively to form a target feature vector , using a gating mechanism and a target association algorithm, the target feature vectors in the continuous frames are associated and matched, the tracking of the same monitoring target at different times is realized, and the radar point cloud and its parameters are output.

5. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: S4 is specifically: S41, radar point cloud projection and coordinate conversion According to the spatial transformation parameters obtained by the system joint calibration, the server end automatically converts and projects the radar point cloud data from the radar coordinate system to the visual image coordinate system to obtain the projection point coordinates of each radar point in the image ; S42, space-intensity joint matching For each visual detection frame, obtain the center point coordinates and the normalized average gray value ; for each radar projection point, extract its normalized reflection intensity ; The following space-intensity joint matching degree function is used for correlation determination: ； wherein: is the projection point coordinate of the th radar point in the image coordinate system; is the center point coordinate of the th target detection box; is the normalized radar reflection intensity; is the normalized detection box average gray value; is the spatial distance tolerance threshold; is the weight coefficient; The judgment condition is: if ( is a matching threshold value), it is determined that the radar point is associated with the target detection frame . S43, target unique ID assignment For each successfully associated monitoring target, its position post number on the tunnel axis is calculated according to the radar spatial position and measurement data ; in the same section corresponding to the post number, each target is sequentially numbered in the clockwise direction , the system automatically assigns a unique identification ID (format is " - " ), and the target ID, coordinates, layout parameters, etc. are entered into the system database, supporting subsequent query, modification and tracking.

6. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: S5 is specifically: The accurate detection frame coordinates of the monitoring target in the initial frame and the subsequent frame images are extracted respectively, the pixel displacement of the target in the image plane is obtained by calculating the coordinate difference between the two frames ; for the radar point cloud set matched with the visual detection frame (matching point index set), the phase information of each radar point at the initial time and the subsequent time is extracted, respectively denoted as and , according to the radar working wavelength , the radial displacement of the radar point is calculated according to the following formula :​ ； After calculating the radial displacement of all matched radar points, the final radial displacement of the monitoring target is calculated according to the following weighted average formula : ； wherein: is the reflection intensity of the th radar point; is the corresponding weight factor.

7. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: S6 is specifically: S61, according to the pixel displacement, the initial depth and the camera intrinsic parameters, the pixel coordinate change of the monitoring target is calculated by using the following geometric model correlation with three-dimensional space deformation correlation: ； where: is the initial depth of the target point, estimated using radar observations. ； initial radial distance for radar measurement; , azimuth and elevation angles measured by radar, respectively; , focal length parameter of camera; initial pixel coordinates of monitoring target; principal point coordinates of camera; S62, according to the radar radial geometry, to establish radial displacement with three-dimensional space deformation between the geometric relationship: ； wherein: to monitor the initial three-dimensional coordinates of the target, and are calculated from the pixel coordinates and the initial depth. ； S63, according to the above relationship, define the joint objective function, the three-dimensional deformation Let the optimization variable, the objective function is: ； wherein: ; , , are respectively visual, radar geometric relation residuals; , , , are respective weights; S64, an initial depth variable in the objective function Setting a boundary constraint, i.e. Only in a range, wherein, is an initial depth measurement error range; S65, the target function is solved by using the trust region reflective algorithm, and the optimal three-dimensional deformation solution is output And the results are automatically generated into a structured monitoring file and stored in a database for subsequent historical data retrieval and analysis.

8. The tunnel surrounding rock three-dimensional deformation monitoring method of fusing monocular vision and millimeter wave radar according to claim 1, characterized in that: S7 is specifically: the server side automatically judges the three-dimensional space deformation and related monitoring indicators according to the alarm threshold preset by the system management interface; if the deformation amount or change rate of any target exceeds the set threshold, the system immediately generates an alarm information and pushes it to the remote monitoring end in real time, realizing automatic alarm and emergency response; at the same time, all monitoring and alarm information are archived in the database, supporting user query, statistics and analysis of alarm history.

Images