A high-density pixel tracking method and system for real-time monitoring of deformation of rock-soil mass

Abstract

The application relates to the technical field of geological disaster monitoring and early warning, and discloses a high-density pixel tracking real-time monitoring method and system for rock-soil mass deformation. The technical scheme is as follows: a monitoring video is collected and uploaded or cached; format conversion, time alignment and buffer management are completed; automatic point distribution and high-density pixel-level continuous tracking are carried out to generate a coordinate matrix; a model is trained and optimized; three-dimensional coordinates are obtained by using depth estimation; clustering analysis, threshold judgment and alarm triggering are carried out; and visual display and report generation are realized. The application has the advantages of strong equipment compatibility, high automation, large monitoring density, strong real-time performance, no need for physical targets, flexible deployment and the like, can be widely applied to complex geological scenes such as slopes, tunnels and landslide bodies, can significantly improve the timeliness and accuracy of geological disaster early warning, can reduce operation and maintenance costs, and has good engineering popularization value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of geological disaster monitoring and early warning, in particular to a real-time monitoring method and system for rock-soil mass deformation based on sliding window management and high-density pixel-level feature point tracking technology, which is suitable for continuous deformation monitoring in various rock-soil engineering environments such as landslides, slopes, tunnel surrounding rock, foundation pits and underground structures. BACKGROUND

[0002] Rock-soil deformation monitoring is an important means to prevent geological disasters such as landslides and collapses, especially in complex environments such as mountainous areas, karst topography or deep tunnels. Landslides are the most widespread and destructive form of slope failure, widely existing in complex geological regions such as mountains and karst, which can destroy vegetation, affect regional ecological balance, and cause significant loss of life and property along the road. In the construction phase of tunnel engineering, rock-soil deformation may cause mud bursting and water gushing at the working face; during the operation period, long-term stress may cause fatigue failure of the tunnel lining, threatening traffic safety.

[0003] Deformation monitoring is an important means to realize geological disaster warning and prediction of rock-soil mass, but existing methods still have obvious shortcomings when faced with complex engineering environments and sudden disaster processes. Current commonly used monitoring technologies, such as laser radar, GNSS and image method based on artificial target setting, generally have problems such as expensive equipment, tedious layout, sparse point distribution and poor real-time performance.

[0004] For example, Chinese patent application CN114942422A proposes a three-dimensional point cloud monitoring method based on laser radar, which has certain accuracy, but due to high equipment cost, it is difficult to promote and apply in a large area. For another example, Chinese patent application CN118670262A uses unmanned aerial vehicle images to cooperate with artificial or physical targets for deformation detection, which improves the convenience of data acquisition, but in harsh outdoor environments, the layout and maintenance of targets are difficult, the monitoring density is limited, and the target layout cannot be fully covered. This method cannot achieve high-density and continuous long-term monitoring.

[0005] More importantly, existing methods often have difficulty in simultaneously meeting the three core requirements of low cost, strong real-time performance and high accuracy: either the cost is too high to be widely applied, or the technology is limited to intermittent monitoring, making it impossible to achieve continuous data collection. Once sudden deformation or disaster occurs during the monitoring period, it is often difficult to identify and warn in time, seriously weakening the effectiveness of disaster prevention and control.

[0006] Therefore, there is an urgent need for a geotechnical deformation real-time monitoring technology that does not rely on physical targets, supports high-density automatic point distribution, can run on ordinary industrial cameras or embedded platforms, and has pixel-level tracking capability and spatial coordinate reconstruction function. The scheme proposed in the application realizes all-weather, continuous and non-contact deformation monitoring in complex environments by developing a sliding window mechanism, high-density pixel-level feature point tracking and depth estimation technology. It takes into account the economy of system deployment, the reliability of monitoring accuracy and the real-time response efficiency, and is suitable for various geological disaster scenes such as tunnels, slopes and landslides, and has wide engineering application prospect and popularization value. SUMMARY

[0007] In order to solve the technical problems existing in the prior art, the purpose of the present application is to provide a geotechnical deformation real-time monitoring method and system with the advantages of low cost, high density, strong real-time performance and high precision, in order to solve the shortcomings of the existing monitoring technology in terms of complex deployment, high cost, sparse point distribution, poor real-time performance and inability to continuous monitoring.

[0008] To achieve the above purpose, the present application provides the following technical scheme:

[0009] A high-density pixel tracking geotechnical deformation real-time monitoring system, comprising:

[0010] A data acquisition module for acquiring video data of a monitoring area;

[0011] A depth estimation module for converting pixel coordinates into three-dimensional space coordinates;

[0012] A pixel tracking module for automatically deploying high-density monitoring points in the monitoring area and performing continuous pixel-level tracking;

[0013] A data processing module for time alignment, deformation calculation and alarm judgment of tracking data and depth data;

[0014] A data visualization module for dynamically displaying deformation information, triggering alarms and generating monitoring reports.

[0015] Further, the data acquisition module supports monocular, binocular or multi-view imaging devices, has three running modes of cloud, edge or local, and supports real-time uploading, local caching and breakpoint resume of video data.

[0016] Further, the pixel tracking module comprises:

[0017] A monitoring point initialization unit for defining a monitoring area in the first frame of image and generating a grid monitoring point;

[0018] A feature extraction and tracking unit for multi-scale feature extraction and iterative position correction based on a deep learning network;

[0019] A sliding window processing unit is used to process the video stream in blocks to ensure tracking continuity and memory optimization.

[0020] Further, the sliding window processing unit divides the video stream into overlapping windows, each window having a length of L frames, and adjacent windows overlapping by L / 2 frames, and only tracking states of the current window and part of the previous window are kept each time.

[0021] Further, the depth estimation module supports monocular depth estimation, binocular stereo matching or active ranging, and converts the two-dimensional coordinates output by the pixel tracking module into three-dimensional world coordinates.

[0022] The application also includes a high-density pixel tracking real-time monitoring method for rock and soil deformation, and the steps are as follows:

[0023] S1: Collect video data of the monitoring area and perform time synchronization and storage management;

[0024] S2: Perform format conversion, time alignment and buffer management on the video data;

[0025] S3: Automatically set up monitoring points in the first frame of image, use a sliding window mechanism for high-density pixel-level tracking, and generate a pixel coordinate time sequence matrix;

[0026] S4: Train and optimize the pixel tracking model to improve its robustness in complex environments;

[0027] S5: Convert the pixel coordinates into three-dimensional space coordinates using a depth estimation method;

[0028] S6: Perform time sequence alignment, displacement calculation, clustering analysis and threshold judgment on the three-dimensional coordinates to trigger an alarm;

[0029] S7: Dynamically visualize the deformation data, generate a monitoring report and output warning information.

[0030] Further, the sliding window mechanism in step S3 divides the video stream into multiple overlapping windows, each window having a length of L frames, and adjacent windows overlapping by L / 2 frames, and the position of the monitoring point is iteratively corrected and the pixel coordinate matrix is output.

[0031] Further, the depth estimation method in step S5 includes:

[0032] In binocular mode, the disparity is calculated through camera calibration and stereo matching to obtain depth information;

[0033] In monocular mode, the depth is predicted by a pre-trained depth estimation network, and the scale is calibrated;

[0034] In the active ranging mode, the distance information is obtained by laser ranging, structured light or millimeter wave radar.

[0035] Further, the model training in step S4 uses multi-scene synthetic data or measured videos, adopts supervised learning, contrast learning or iterative optimization method, and supports cloud, edge or local deployment after training.

[0036] Further, the visualization display in step S7 includes three-dimensional deformation field, heat map, motion trajectory and sliding window dynamic display, supports user interaction operation and multi-platform access.

[0037] The beneficial effects of the present application are:

[0038] Compared with the prior art, the high-density pixel tracking real-time monitoring method and system for rock-soil mass deformation have the following technical characteristics and beneficial effects:

[0039] (1) Flexible device selection and strong compatibility: high-precision industrial equipment, general monitoring cameras or other imaging devices can be selected according to the site conditions, supporting high-precision monitoring and meeting the low-cost application requirements;

[0040] (2) No physical target and high automation: avoiding manual intervention such as target layout and maintenance, the system automatically lays out and tracks, reducing operation and maintenance costs;

[0041] (3) High-density monitoring and strong real-time performance: pixel-level tracking combined with sliding window mechanism improves monitoring frequency and point coverage, realizes second-level update and continuous data acquisition;

[0042] (4) Accurate three-dimensional deformation reconstruction: the spatial motion information of each monitoring point is obtained through the depth estimation module to construct the complete dynamic evolution process of the rock-soil mass surface;

[0043] (5) Widely applicable and suitable for multi-scene deployment: widely applicable to geological disaster monitoring areas such as slope landslide, tunnel surrounding rock and surface crack area, and also suitable for complex environments such as transportation, building and underground space.

[0044] In summary, the present application significantly improves the real-time performance, accuracy and deployment flexibility of rock-soil mass deformation monitoring, breaks the dependence of traditional technology on expensive equipment and periodic measurement, and provides a new type of solution that can be implemented and popularized for intelligent monitoring, early warning and engineering safety management of geological disasters. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to make the technical solutions of the embodiments of the present application clearer, the present application will be described in detail below in combination with the drawings and detailed embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative effort. Among them:

[0046] Figure 1 Module block diagram for the monitoring system in the present application;

[0047] Figure 2 Technical roadmap for the real-time monitoring method in the present application;

[0048] Figure 3 Structural schematic diagram of the pixel tracking module in the present application;

[0049] Figure 4 Working mode diagram of the sliding window of the pixel tracking module;

[0050] Figure 5 Visual result display diagram of the deformation monitoring system. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. The following will be described in combination with the drawings and embodiments Figures 1 to 5 The real-time monitoring method and system for rock-soil body deformation based on high-density pixel tracking are further described.

[0052] Embodiment 1

[0053] The present application proposes a real-time monitoring system for rock-soil body deformation based on sliding window mechanism and high-density pixel-level feature tracking. The system constructs a monitoring framework that can flexibly run on various camera platforms (including high-precision industrial cameras and low-cost general monitoring cameras), and has good universality and adaptability. The overall framework of the system is divided into five modules: data acquisition module, depth estimation module, pixel tracking module, data processing module and data visualization and alarm module, and the working process set architecture is as shown in Figure 1 .

[0054] The functions of each module are as follows:

[0055] Part 1 Data acquisition module: deploy monocular, binocular or other imaging equipment, and data transmission device. Get the video data of the monitoring area through network or local storage.

[0056] Part 2 Depth estimation module: multiple distance acquisition methods are adopted, including but not limited to monocular or binocular depth estimation methods, and laser ranging, etc., to estimate the depth information of the monitoring area according to the video stream data transmitted by the data acquisition module, and to obtain the real physical world space coordinates of the area.

[0057] Part 3 Pixel tracking module: contains a trained high-density pixel tracking neural network model. The module arranges and initializes the monitoring points and the monitoring density, adopts a sliding window design for pixel tracking, tracks the position of each monitoring point over time, and outputs the pixel coordinates of each monitoring frame of the monitoring point in the image.

[0058] Part 4 Data processing module: processes the data reception and time alignment of Part 2 depth estimation and Part 3 pixel tracking, and performs deformation calculation and motion analysis, sets up an alarm mechanism, and performs data storage and management.

[0059] Part 5 Data visualization module: used for dynamic visualization of monitoring and analysis data, alarm display, report generation, system integration and expandability, etc.

[0060] The technical advantages of the present application include:

[0061] (1) Compatible with multiple types of camera devices: the system supports both high-resolution industrial cameras for fine monitoring needs and general network surveillance cameras, suitable for cost-limited or existing monitoring system scenarios, enabling differentiated deployment;

[0062] (2) Automatic point placement and high-density pixel-level tracking: automatically generates high-density grid monitoring points in the monitoring image without the need for physical target placement, enabling non-contact large-scale point placement and continuous tracking;

[0063] (3) Sliding window mechanism: frame data is cached and managed through a sliding window, ensuring monitoring continuity while reducing system memory and computing resource consumption;

[0064] (4) Multi-mode depth estimation module: supports monocular vision depth estimation, binocular stereo matching, laser ranging, and other depth estimation methods, allowing selection of appropriate methods to obtain three-dimensional space coordinates based on site conditions and device capabilities;

[0065] (5) Deformation analysis and visualization module: the system can calculate the three-dimensional displacement, speed, and motion trajectory of the monitoring points in real time, and display and analyze the results on the visualization interface, supporting abnormal alarm and data backtracking.

[0066] The method and steps used by the monitoring system are as follows:

[0067] Preferably, the Part 1 data acquisition module is used to acquire video data of the monitoring area and perform time synchronization and storage management. Mono, binocular or other imaging devices can be deployed to adjust the shooting angle, focal length and resolution to cover the monitoring area. Video data can be uploaded to the cloud in real time through the network, or temporarily stored locally when the network is disconnected, and automatically supplemented when the network is restored. The system supports local, edge or cloud processing, video is transmitted through encryption and time stamping, and preliminary preprocessing is performed, and the server can dynamically allocate computing resources to the pixel tracking and depth estimation modules.

[0068] Preferably, the Part 2 depth estimation module converts the two-dimensional pixel coordinates output by the pixel tracking module into three-dimensional space coordinates, realizing the positioning of the monitoring points in the real space. It supports monocular depth estimation, binocular stereo matching or active ranging, etc., to ensure that each frame of pixel corresponds to a space coordinate. The module can be deployed in the cloud, edge or local, and the output three-dimensional coordinates are used for subsequent displacement calculation and deformation analysis.

[0069] Preferably, the Part 3 pixel tracking module is used to initialize the monitoring points and realize continuous pixel-level tracking. The system defines the region of interest in the first frame of image, automatically generates grid monitoring points, and records the initial position and timestamp. The module extracts features through but not limited to CNN, combines attention mechanism or Transformer network structure for iterative correction, and fuses neighborhood pixel correlation to improve stability and accuracy in complex environments. Sliding window processing is used for video, reducing memory occupation, supporting long-time continuous monitoring. Training uses multi-scene data, with good generalization ability, which can be deployed in the cloud, edge or local. The output pixel coordinates are used for three-dimensional reconstruction and deformation analysis.

[0070] Preferably, the Part 4 data processing module receives the data output by the pixel tracking and depth estimation modules, aligns by timestamp, and completes spatial position matching. It calculates the displacement, velocity, trajectory and local deformation clustering of the monitoring points, and assists in identifying abnormal areas. The data can be stored and accessed by the visualization module, and can trigger alarms and generate deformation trend and risk level reports. The module adapts to multiple monitoring points, multiple frame rates and multiple platform inputs, ensuring continuous and stable output.

[0071] Preferably, the Part 5 data visualization and alarm module dynamically displays the processed three-dimensional coordinates and deformation information, supporting three-dimensional deformation field, trajectory, heat map and sliding window display. User interaction can be performed to adjust the monitoring point density, viewing angle and time window. According to the displacement rate, acceleration or accumulated deformation compared with the threshold value, the alarm is triggered, and periodic monitoring reports can be generated, including deformation trend, risk level and high-risk area labeling. The module supports cloud, edge or local deployment, and can flexibly adjust the strategy according to different monitoring scenarios, realizing multi-user access and system integration.

[0072] Example 2

[0073] The main steps of the monitoring method of the present application include:

[0074] S1 Data acquisition: Install data acquisition modules at the monitoring site, including monocular, binocular or other imaging devices and data transmission devices. According to the site environment, adjust the shooting angle, focal length and resolution to ensure coverage of the entire monitoring area. The collected video data can be uploaded to the cloud server in real time through 4G / 5G, optical fiber or wired network, or temporarily stored in local or edge devices when the network is limited, and automatically supplemented after the network is restored to ensure data integrity. Time stamping and preliminary formatting are performed during data uploading or storage to prepare for subsequent processing.

[0075] S2 Preliminary data processing: The collected video stream is pre-processed in the cloud, edge or local server, including data format conversion, time alignment, frame selection and buffer management, to ensure that the subsequent pixel tracking and depth estimation modules can efficiently and stably process the data.

[0076] S3 Pixel tracking: Define the monitoring area in the first frame of image and automatically generate a grid of adjustable density monitoring points. The system uses a point tracking network based on deep learning to track each monitoring point with high precision and across frames, records its pixel coordinates in each frame, and finally forms a coordinate time sequence matrix with dimensions TxNx2, providing a basis for subsequent displacement calculation. This module uses convolutional neural networks to extract image features and combines attention mechanisms or Transformer structures for global modeling, iteratively corrects and fuses neighborhood pixel correlations, and realizes stable pixel tracking with high density, continuity and multiple frames. It also has strong adaptability to common noise, light changes and local deformation in slope scenes.

[0077] S4 Model training and optimization: Training data uses multi-scene synthetic data or measured video data, covering different light, texture, monitoring distance and environmental conditions to ensure the robustness of the model in complex scenes. Training methods can use supervised learning, contrastive learning or iterative optimization methods to make the network learn the correspondence between pixels and dynamic change rules, and optimize tracking accuracy. In terms of model deployment, the trained model can be deployed on the cloud, edge or local device to process monitoring data in real time according to computing resources and network conditions, providing continuous and reliable predictions for the pixel tracking module.

[0078] S5 Depth estimation: input the two-dimensional coordinates output by the pixel tracking module into the depth estimation module, and generate a three-dimensional world coordinate matrix TxNx3 of the monitoring points by using monocular estimation, binocular stereo matching or active ranging, etc. In the binocular mode, the depth is obtained by parallax calculation; in the monocular mode, the depth information is predicted by the trained depth network model; in the active ranging mode, the target distance is directly obtained. The internal and external parameters of the device are used to realize the mapping of pixel coordinates to spatial coordinates.

[0079] S6 Monitoring data analysis: the system receives the data output by the pixel tracking and depth estimation modules, and aligns them according to the video frame timestamps. Based on the three-dimensional coordinate sequence, the displacement, velocity, acceleration and motion direction of the monitoring points are calculated, and the local deformation area can be clustered and trend evaluated. The processed data is stored in sequence, which can be accessed locally, at the edge or in the cloud, and the displacement rate, acceleration or aggregated deformation condition is compared with the preset threshold to trigger real-time alarms.

[0080] S7 Data visualization and early warning: the system dynamically visualizes the data analysis results, including the three-dimensional displacement trajectory of the monitoring points, the deformation field heat map and the motion direction arrow, and supports continuous display in a sliding window and user interaction adjustment (such as observation angle, time window and monitoring point density). According to the displacement rate, acceleration or aggregated deformation condition of the monitoring points compared with the preset threshold, the system can trigger real-time alarms (SMS, email, platform push, etc.), and automatically generate safety tips and operation suggestions. At the same time, periodic monitoring reports are generated, including deformation trend chart, risk level evaluation and high-risk area labeling, which can be exported for engineering management and decision-making. The module supports cloud, edge or local deployment, adapts to different monitoring scenarios, realizes multi-user access and system integration.

[0081] The present application as a whole comprises five modules: data acquisition module, depth estimation module, pixel tracking module, data processing module and data visualization module, and the monitoring method technical route is as shown in Figure 2 Each module cooperates with each other to form a complete, continuous and operable rock-soil deformation monitoring system.

[0082] Example 3

[0083] In order for those skilled in the art to more clearly understand the technical solutions of the present application, the present application will be described in detail in conjunction with an example of a rock-soil deformation real-time monitoring system based on a general binocular camera.

[0084] It should be noted that the following examples are only used to illustrate the principles and methods of the present application and do not constitute a limitation on the scope of protection of the present application. It should be understood that the present application is not only applicable to slope monitoring scenarios, but also applicable to tunnels, tunnels, foundation pits, underground structures, landslide bodies, goaf and other types of geotechnical engineering fields. The camera used is not limited to a binocular camera, and a high-precision industrial camera, a general network monitoring camera or other types of imaging equipment can be selected according to the actual monitoring accuracy requirements. Any structural replacement, module adjustment, function extension or algorithm deformation made within the spirit and principles of the present application shall be deemed to fall within the scope of protection of the present application.

[0085] Part 1: Build a data acquisition module

[0086] In the field deployment stage of the monitoring area, the spatial layout of the binocular camera is planned according to the structure characteristics of the geotechnical body and the monitoring target. The camera spacing is calculated according to the monitoring range by the baseline formula:

[0087] (1)

[0088] Where Z is the monitoring distance, f is the focal length, and D is the parallax accuracy requirement. The camera spacing of the data acquisition module is determined according to formula (1). To eliminate binocular parallax error, stereo calibration is performed before deployment to obtain the camera intrinsic parameter matrix K and the rotation and translation matrix T.

[0089] The camera is fixed by a stabilizing gimbal to ensure that the viewing angle covers the overall view of the monitoring area. The pitch angle is recommended to be to avoid backlight interference. The camera parameters are set as follows: resolution ≥ 2560x1440, frame rate ≥ 30Fps, and support H.265 encoding compression to reduce transmission bandwidth.

[0090] The data acquisition module of the present application supports multiple operating modes:

[0091] Cloud mode: Video stream is uploaded to the cloud server in real time through 4G / 5G or optical fiber network. Video stream is encrypted and transmitted, and the cloud server receives, stores and processes it. When the network is disconnected, it is automatically cached to the local storage (≥256GB SSD).

[0092] Edge computing mode: An edge computing box is deployed on site to perform real-time preprocessing, feature extraction, pixel tracking and preliminary depth estimation on the video stream. The edge computing module can calculate according to the monitoring point density and window length to reduce the dependence on network bandwidth and generate preliminary deformation information locally.

[0093] Local mode: In areas where the network is unreliable or security is limited, data can be completely processed and stored locally, and the edge computing box or local server can generate monitoring reports, deformation charts and warning information without uploading to the cloud.

[0094] The camera is equipped with a protective cover and a built-in temperature and humidity sensor, which can automatically start heating or defogging function in extreme environment. The camera has zooming capability, which can realize the switching of overall and local monitoring. This module supports high-precision industrial cameras, general monitoring cameras or other imaging devices, which can be flexibly selected according to the actual precision requirements.

[0095] Part 2: Realize depth estimation module

[0096] The depth estimation module is used to convert pixel coordinates into three-dimensional world coordinates, realizing accurate acquisition of the spatial position of the monitoring point. The present application supports a variety of depth acquisition methods, including but not limited to binocular stereo vision, monocular depth estimation, structured light, laser ranging or other applicable imaging and ranging technologies. It can be flexibly selected according to the site environment, precision requirements and cost constraints.

[0097] In this embodiment, binocular vision scheme is adopted. First, the camera intrinsic matrix and the extrinsic rotation and translation matrix are obtained according to the calibration. By matching the corresponding points in the binocular camera imaging, the parallax of the monitoring point is calculated and the world coordinates are further obtained.

[0098] The depth estimation module can work in the following modes:

[0099] Cloud mode: the video stream collected is uploaded to the cloud server, and the depth estimation and three-dimensional coordinate calculation are performed by the cloud, which is suitable for the scene with smooth network.

[0100] Edge computing mode: deploy edge computing box on site to complete depth estimation and preliminary three-dimensional coordinate calculation in real time, which can significantly reduce transmission delay and bandwidth pressure, and support local processing of sliding window.

[0101] Local mode: in the network limited or security sensitive area, the depth estimation can be completed completely in the local, including video preprocessing, pixel tracking coordinate input and world coordinate calculation, and the preliminary deformation data and warning information can be directly generated.

[0102] The three-dimensional world coordinates obtained by this module can be input into the data processing module to realize the calculation of the spatial displacement, velocity and deformation trajectory of the monitoring point.

[0103] This module flexibly compatible with different hardware and depth acquisition technologies, and can complete data processing in cloud, edge or local multiple running modes, ensuring high precision, real-time and scalability of the system.

[0104] Part 3: Realize pixel tracking module

[0105] (1) Monitoring point initialization: As shown in the pixel tracking neural network architecture as shown in Figure 3 , a region of interest (ROI) is defined in the first frame of image, and a grid monitoring point is automatically generated and initialized for all frame monitoring point positions, recording the initial pixel coordinates and time stamp . All frame monitoring point positions are initialized as the first frame monitoring point position . In this way, the initial consistency and continuity of the monitoring point are ensured, providing a reliable basis for subsequent tracking.

[0106] (2) Feature extraction and pixel tracking:

[0107] Feature extraction and pixel tracking are modeled by, but not limited to, the following network and method:

[0108] ① Multi-scale feature extraction: The module extracts texture, edge and local detail features of the monitoring point at different scales through the vision transformer (ViT), providing rich information for tracking.

[0109] ② Iterative correction: The extracted features are input into the Transformer or iterative network structure, and through multiple rounds of feature fusion and pixel position update, the position of each monitoring point is continuously corrected, thereby reducing drift and improving tracking accuracy.

[0110] ③ Correlation feature fusion: During tracking, the module uses the correlation features of the monitoring point and the neighboring pixels to optimize the position, enhancing the robustness in texture repetition, occlusion or illumination change scenarios.

[0111] ④ High-density continuous tracking: Support for high-density grid monitoring points tracking, maintain position stability through iterative correction, achieve high-precision tracking of continuous video frames.

[0112] (3) Sliding window processing

[0113] Sliding window strategy is used to process video stream: video frames are divided into overlapping windows, only the current window and part of the previous window frames are retained each time; window overlap ensures tracking continuity and avoids drift accumulation; block processing of video reduces memory occupancy and ensures real-time tracking of long videos.

[0114] (4) Model training and generalization

[0115] ① Training data: Use synthetic data sets or multi-scene measured data to train the network to cover different light, texture, monitoring distance and environmental conditions, and enhance the model generalization ability.

[0116] ②Training method: Through supervised learning, contrastive learning or iterative optimization strategy, the network learns the correspondence between pixels and the change rule of dynamic characteristics, realizing continuous tracking of high-density monitoring points.

[0117] ③Adaptability: After training and optimization, the model can stably operate in different geotechnical environments (slope, tunnel, foundation pit, underground structure, landslide body, goaf, etc.), providing accurate and continuous pixel coordinate sequences.

[0118] (5) Model deployment

[0119] The pixel tracking model after training and optimization can be flexibly deployed on different computing platforms to adapt to various monitoring scenarios and resource conditions:

[0120] ①Cloud deployment: The model is loaded on a cloud server, which receives video streams from the data acquisition module in real time for processing; the cloud can use powerful computing resources for high-density, multi-point, and long-time video frame tracking, which is suitable for large-scale monitoring scenarios; the output monitoring point pixel coordinate sequence can be directly transmitted to the depth estimation module to generate three-dimensional spatial coordinates, and used for deformation analysis, visualization and alarm.

[0121] ②Edge deployment: The model is deployed in edge computing devices (such as edge computing boxes) to process video streams directly on the acquisition site; edge processing reduces transmission delay and is suitable for environments with high real-time requirements or limited network conditions; it supports local real-time deformation calculation and immediate alarm, and can synchronize the processing results to the cloud server for centralized management and storage.

[0122] Local deployment: The model can also be deployed on a local server or workstation to realize a completely independent monitoring system; it is suitable for monitoring scenarios with limited network or high data privacy requirements; local deployment can combine sliding window strategy and high-density pixel tracking to realize long-time continuous monitoring and data caching, ensuring uninterrupted monitoring even if the network is disconnected.

[0123] Although this embodiment exemplarily adopts CNN and Transformer architecture combination for training and optimization, the patent protection scope is not limited to specific network structure or training method. Any feature extraction and iterative correction method that can realize high-density, continuous, pixel-level tracking, including convolutional network, attention mechanism, Transformer, related feature fusion and training optimization strategy, etc., belongs to the protection scope of the invention.

[0124] Part 4: Data processing module

[0125] The data processing module is mainly responsible for receiving, aligning, analyzing and visualizing the monitoring data output by the pixel tracking module and the depth estimation module, thus achieving continuous, reliable and operable deformation information output of the rock mass. This module includes data receiving devices, servers and processing algorithms, and realizes the following functions:

[0126] ① Data reception and time alignment

[0127] The module receives real-time pixel coordinate sequences of monitoring points from the pixel tracking module and three-dimensional coordinate data output by the depth estimation module;

[0128] Align the pixel coordinates and three-dimensional coordinates according to the video frame timestamp to ensure that the spatial position and pixel position of each monitoring point at the same time point correspond consistently;

[0129] Support multi-source data fusion, including data output by different cameras and different collection devices, to ensure the unity and completeness of the monitoring results.

[0130] ② Deformation calculation and motion analysis

[0131] Based on the time series of spatial coordinates, calculate the displacement, velocity, acceleration and motion direction of each monitoring point;

[0132] Perform spatial clustering analysis on the monitoring point group to identify local deformation concentration areas or potential risk areas;

[0133] Support continuous time window analysis to extract long-term deformation trends and short-term mutation information for real-time early warning.

[0134] ③ Data storage and management

[0135] Store and index the processed monitoring data, including spatial coordinate sequences, displacement, velocity and alarm records;

[0136] Support local storage, edge caching and cloud synchronization to ensure the integrity and reliability of long-term monitoring data;

[0137] Provide an interface for the visualization module to access, facilitating dynamic display of monitoring results.

[0138] ④ Alarm and threshold judgment

[0139] The module compares the displacement rate, acceleration or aggregated deformation degree of the monitoring point with the preset threshold value; once the safety threshold is exceeded, the system can trigger real-time alarms (SMS, email or platform notifications);

[0140] Support automatic risk level assessment and report generation, output deformation trend chart, heat map and high-risk area labeling.

[0141] ⑤ Extensibility and adaptability

[0142] The data processing module can adapt to multi-camera, multi-monitoring point, and high-frame-rate video input, and supports sliding window and incremental update strategy to ensure the real-time performance and stability of long-time continuous monitoring.

[0143] It is compatible with different types of pixel tracking and depth estimation algorithms, and can adapt to various monitoring systems and hardware platforms.

[0144] Part 5: Data visualization and alarm module

[0145] This module has dynamic presentation and operational display, and realizes real-time alarm and monitoring report generation. This module includes web or client display system, visualization engine and alarm interface, with the following functions:

[0146] ① Dynamic visualization

[0147] Receive the time series of three-dimensional space coordinates of monitoring points and the calculated deformation, velocity and other indicators output by the data processing module;

[0148] Automatically draw the spatial displacement trajectory of the monitoring points, and support three-dimensional deformation field rendering, use heat map, arrow or color coding to identify high-risk areas and displacement size;

[0149] Support real-time update display, show the changes of continuous monitoring frames through sliding window, and realize dynamic demonstration of visualization;

[0150] Provide user interaction functions, such as adjusting monitoring point density, viewing angle, time window and other parameters.

[0151] ② Alarm mechanism

[0152] According to the displacement rate, acceleration or accumulated deformation of continuous monitoring points, judge with the preset threshold value;

[0153] Once the threshold value is exceeded, the system can trigger real-time alarm, including SMS, email, platform push and other ways;

[0154] Support custom threshold and alarm logic, and automatically generate safety tips and operation suggestions combined with regional risk level.

[0155] ③ Report and analysis

[0156] Automatically generate periodic monitoring reports, including deformation trend chart, risk level assessment, high-risk area labeling and historical change data;

[0157] Export reports and visualization charts for easy decision-making and construction adjustment by engineering management personnel, construction units or monitoring personnel.

[0158] ④ System integration and scalability

[0159] The module can be seamlessly integrated with a cloud, edge or locally deployed monitoring system, enabling remote access and multi-user sharing;

[0160] Supports dynamic adjustment of visualization strategies and alarm logic for different monitoring scenarios (slopes, tunnels, foundation pits, landslide bodies, goaf, etc.), ensuring system versatility and flexibility.

[0161] The specific operation method is as follows:

[0162] S1, the data acquisition step includes:

[0163] Step S101, first, the monitoring area is surveyed on site, and the layout scheme and installation position of the imaging device are determined in combination with the rock mass geological conditions, illumination conditions and spatial distribution. For binocular or multi-view imaging devices, the baseline distance and relative attitude of each imaging unit are further adjusted and calibrated accurately to ensure that the subsequent three-dimensional measurement and reconstruction accuracy meets the design requirements.

[0164] Step S102, in the data transmission process, different transmission strategies are selected according to the network conditions of the monitoring site. When the network conditions are good, the collected video data is uploaded to the cloud server in real-time streaming mode through the 4G / 5G communication link or the optical fiber private network, and the code rate is dynamically adjusted to adapt to the network bandwidth to prevent frame loss caused by network fluctuations. In the case of limited or interrupted network, the video data is stored in the edge computing device or the local cache unit, and a circular or segmented storage mechanism is established according to the time sequence to avoid data loss caused by cache overflow. The system supports the breakpoint resume function, and after the network is restored, the timestamp information of the uploaded data is automatically compared, and the missing data segments are sequentially retransmitted to ensure the integrity and continuity of the time sequence.

[0165] Step S103, in the data uploading or storage link, high-precision timestamps are attached to the collected video data, which can be generated based on GPS time service or NTP network time service to realize time alignment of multi-device and multi-node data. Form a standardized data stream that meets the requirements of subsequent image processing and pattern recognition algorithms.

[0166] S2, the preliminary data processing step includes:

[0167] Step S201, the received original video data is parsed and converted, and video streams of different devices and different compression formats are unified into a preset standard data format, such as a specific resolution, frame rate and encoding method, to reduce the decoding overhead of subsequent processing modules and ensure data compatibility.

[0168] Step S202, time alignment processing is performed on the multi-path video stream. By reading the additional high-precision timestamp information, the multi-path video stream is synchronized in the time dimension, providing time-consistent data input for subsequent multi-view fusion and three-dimensional reconstruction.

[0169] Step S203, buffer management is performed on the video stream, a multi-level buffer queue is established, and the input data is queued, cached and flow controlled to ensure continuous input and no frame loss phenomenon in subsequent algorithm processing. In the scene of high concurrency or burst data volume, by dynamically adjusting the cache depth or enabling priority scheduling, the key frames and important monitoring area data are preferentially transmitted, and the stability and real-time performance of the overall processing system are improved.

[0170] S3, the pixel tracking step comprises:

[0171] Step S301, first determine the monitoring area boundary in the first frame image, the boundary can be obtained by manual interaction calibration, automatic target detection algorithm or preset coordinate file. After determining the monitoring area, the system automatically generates a grid monitoring point set covering the entire monitoring area according to the area size and the user-set grid density parameter, and the distribution of the monitoring points can be adjusted to meet the needs of different precision and calculation overhead.

[0172] Step S302, the system calls a point tracking network based on deep learning to track the monitoring points frame by frame. Specifically, the received video stream is disassembled into continuous frame images and labeled with the space-time index of each frame, and is processed by a sliding window mode as shown in Figure 4 . The video frame is first loaded into the sliding window, and the input video stream

[0173] (2)

[0174] wherein V represents the input video sequence, I t represents the t frame image, t represents the time index of the image frame;

[0175] The total number of frames is , and the sliding window is divided into:

[0176] (3)

[0177] wherein Wj represents the set of continuous image frames contained in the jth sliding window, wherein L is the window length (default L=8 frames), j is the window index , adjacent windows overlap L / 2 frames to ensure tracking continuity; ┗ • ┛ is the floor operation.

[0178] The processed video frames will be discarded from memory, only the tracking state of the last L / 2 frames of the current window and the previous window is retained to solve the problem of insufficient memory in long video tracking. The input sliding window video is processed by feature extraction and Transformer network architecture m times to correct the monitoring point position:

[0179] (4)

[0180] wherein is the multi-scale feature extracted by CNN, is the RAFT related feature, the iteration number m, represents the monitoring point pixel coordinates after the m+1th iteration, represents the monitoring point pixel coordinates estimation after the mth iteration. The sliding distance of the window is L / 2 frames each time, and the monitoring point pixel coordinate matrix is output after sliding window processing wherein represents the monitoring point pixel coordinate matrix output by the th sliding window, represents the real number field, N represents the number of monitoring points.

[0181] Further, in step S4, the model training and optimization step includes: first, constructing a multi-scene and multi-condition training data set, which can be generated from synthetic data based on physical simulation and rendering. The training data covers various lighting conditions, texture complexity, monitoring distances and environmental interference factors to ensure that the model has strong generalization ability and robustness in different scenes. For synthetic data, various camera angles, lens distortion, noise levels and dynamic deformation processes can be simulated through parameterized adjustment, making the training samples more similar to the actual slope monitoring environment.

[0182] In the training process, various training paradigms such as supervised learning, semi-supervised learning or contrastive learning can be used. Supervised learning directly minimizes the error between the predicted position and the true position by giving pixel-level correspondence labels; contrastive learning guides the network to learn discriminative feature representation between pixels by constructing positive and negative sample pairs; iterative optimization method gradually optimizes the prediction result frame by frame in a self-supervised manner, so that the network gradually learns the temporal consistency and dynamic change rule.

[0183] After the training is completed, the model is quantized, pruned, and optimized for inference acceleration to be efficiently deployed on cloud servers, edge computing devices, or local embedded processors. According to the computing resources and network bandwidth conditions of the monitoring system, the model can be run centrally in the cloud to achieve batch processing, or it can be sunk to edge devices to achieve low-latency real-time tracking. The model can support online updating and incremental training during operation, fine-tuning the network parameters with the latest monitoring data to adapt to long-term changes in the field environment and continuously improve the prediction performance, providing continuous, reliable, and high-precision prediction results for the pixel tracking module.

[0184] S5, the depth estimation step includes: inputting the two-dimensional coordinate time sequence matrix output by the pixel tracking module into a depth estimation module to obtain three-dimensional spatial position information of the monitoring points. Specifically, for the binocular imaging mode, first, the internal and external parameters of the binocular camera are calibrated to obtain the baseline distance, principal point coordinates, and distortion coefficients; then the disparity of the corresponding monitoring points in the left and right views is calculated through a binocular stereo matching algorithm, and the depth value is obtained by using the triangulation principle. For the monocular imaging mode, a monocular depth estimation network based on deep learning is used, a single frame of image is input into the trained depth prediction model, and the relative depth map corresponding to each pixel is output, and the absolute scale of the depth result is calibrated through a scale recovery method or a reference calibration point. For the active ranging mode, the system directly measures the distance information of the monitoring points through laser ranging, structured light, or millimeter wave radar, and combines it with the pixel coordinates to obtain the complete spatial position

[0185] In this embodiment, a binocular stereo vision depth estimation method is used. Specifically, the coordinates of the monitoring points are input into the depth estimation module, and for each monitoring point, the camera intrinsic matrix K and the extrinsic matrix T are used for coordinate system transformation, where the camera intrinsic matrix K is:

[0186]

[0187] 、 fxx and fyy represent the equivalent focal lengths in the horizontal and vertical directions of the camera, respectively. , Cxx and Cyy are the principal point coordinates of the camera, i.e., the pixel coordinates of the intersection of the optical axis and the imaging plane in the image. α is the skew factor, which is generally 0 (only appears when the pixel is not rectangular).

[0188] The extrinsic matrix T is:

[0189]

[0190] wherein R is a 3x3 rotation matrix representing the rotation relationship of the world coordinate system in the current camera coordinate system, is a 3x1 translation vector, representing the position of the world origin in the current camera coordinate system. The extrinsic matrix T is a 4x4 homogeneous transformation matrix, describing the transformation relationship between the camera coordinate system and the world coordinate system. The above parameters are camera intrinsic and extrinsic parameters, which can be obtained by calibration.

[0191] (5)

[0192] wherein is the pixel coordinate value at time t, is the world coordinate of the monitoring point at the current time. is the depth value corresponding to the pixel point coordinate at the current time.

[0193] S6, the monitoring data analysis step comprises: the system receives the three-dimensional coordinate time sequence matrix output from the pixel tracking module and the depth estimation module, and aligns and synchronizes the data according to the frame timestamp information, ensures that the spatial coordinates of each monitoring point at different time steps are one-to-one corresponding, and forms a complete and continuous time sequence trajectory data set.

[0194] Specifically, according to the world coordinates of the monitoring point changing with time, the spatial motion amount of the measuring point can be obtained:

[0195] (6)

[0196] wherein, , represents the current time and the initial time; ΔP represents the spatial motion amount of the measuring point in the monitoring time period, 、 、 represents the spatial coordinates of the monitoring point at the current time calculated by formula (5), 、 、 represents the spatial coordinates of the monitoring point at the initial time;

[0197] The deformation rate is:

[0198] (7)

[0199] The deformation monitoring system flow result is as shown in Figure 5 , a continuous motion trajectory is generated by using cubic spline interpolation, and an acceleration sudden change point is marked. The displacement, velocity and acceleration results processed are stored in the form of structured data, which can be saved in a local server, an edge computing node or a cloud database, supporting remote access and visual display.

[0200] Further, in step S7, the data visualization and early warning step includes: dynamically visualizing the displacement, velocity, acceleration and clustering analysis results output by the monitoring data analysis module, and presenting them in real time on the user terminal, monitoring platform or three-dimensional visualization interface. Specifically, the system generates a three-dimensional curve of the displacement trajectory of the monitoring point according to the three-dimensional coordinate time sequence matrix, uses an interpolation method to draw a continuous deformation field, and displays the displacement magnitude distribution of the local area in the form of a heat map. At the same time, the motion direction arrow is superimposed on the monitoring point position, which intuitively shows the motion trend of the monitoring point.

[0201] In order to facilitate user interaction analysis, the system provides an adjustable time sliding window function, which supports users to freely switch observation time periods, adjust monitoring point sampling density, change observation angles and scaling ratios, thereby realizing multi-scale and multi-angle visualization analysis. For the case of parallel monitoring of multiple scenes, the system can realize multi-window display and data comparison, which facilitates engineers to simultaneously track the stability conditions of different monitoring areas.

[0202] In the early warning link, the system compares the displacement rate, acceleration and accumulated deformation indicators calculated in real time with the preset threshold value, and if the set safety limit is exceeded, an alarm event is automatically triggered. The alarm information can be sent in various ways, including SMS, email, monitoring platform push, sound and light warning, etc., and can be accompanied by the risk level of the current monitoring area, recommended disposal measures and emergency operation suggestions, so that management personnel can take timely intervention measures to reduce disaster risks. In addition, the system supports periodic automatic generation of monitoring reports for subsequent engineering review, decision support and archival use.

[0203] This step can be deployed and run in the cloud, edge or local environment, supports multi-user concurrent access and cross-platform system integration, is convenient for interfacing with existing monitoring platforms, geological disaster warning systems or construction scheduling systems, realizes centralized management and joint scheduling of data, and thus improves the collaborative efficiency and intelligent level of the overall monitoring system.

[0204] The present application aims to solve the problems of high cost, sparse point distribution, poor real-time performance and inability to continuously monitor in the prior art, and to realize low-cost, high-precision, strong real-time and high-density continuous monitoring. The system consists of five modules: data acquisition, depth estimation, pixel tracking, data processing and data visualization. The former is responsible for video acquisition and transmission, the depth estimation module completes the mapping of pixels to three-dimensional space coordinates, the pixel tracking module realizes automatic deployment and continuous tracking of high-density monitoring points, the data processing module aligns, analyzes and deforms the multi-source data and triggers alarms, and the data visualization module realizes dynamic display and report generation of three-dimensional deformation field, trajectory and risk level.

[0205] Obviously, the above embodiments are merely example for clearly illustrating but not limitation to the embodiments. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, all the embodiments need not and can not be enumerated. The obvious changes or variations derived from this still fall within the protection scope of the present application.

Claims

1. A real-time monitoring system for soil and rock deformation using high-density pixel tracking, characterized in that, include: The data acquisition module is used to acquire video data of the monitored area; The depth estimation module is used to convert pixel coordinates into three-dimensional spatial coordinates; The pixel tracking module is used to automatically deploy high-density monitoring points in the monitoring area and perform continuous pixel-level tracking; The data processing module is used to perform time alignment, deformation calculation, and alarm judgment on tracking data and depth data; The data visualization module is used to dynamically display deformation information, trigger alarms, and generate monitoring reports; Perform the following steps: S1: Collect video data of the monitoring area and perform time synchronization and storage management; S2: Performs format conversion, time alignment, and buffer management on video data; S3: Automatically deploy monitoring points in the first frame image, use a sliding window mechanism for high-density pixel-level tracking, and generate a pixel coordinate temporal matrix. The specific steps are as follows: S31. First, determine the boundary of the monitoring area in the first frame image. The boundary is obtained through manual interactive calibration, automatic target detection algorithm or preset coordinate file. S32. After determining the monitoring area, the system automatically generates a gridded monitoring point set covering the entire monitoring area based on the area size and the grid density parameters set by the user. The distribution of the monitoring points can be adjusted to meet the needs of different precision and computational overhead. S33. The system calls a deep learning-based point tracking network to track the monitoring points frame by frame. The received video stream is decomposed into continuous frame images and each frame is marked with a spatiotemporal index. The process is performed using a sliding window mode. The video frames are first loaded into the sliding window. Input video stream: (2) Where V represents the input video sequence. This represents the image of frame n. t Indicates the time index of the image frame; Total number of frames is The sliding window is divided into: (3) in This represents the set of consecutive image frames contained in the nth sliding window, where L is the window length (default L=8 frames) and j is the window index. Adjacent windows overlap by L / 2 frames to ensure tracking continuity; This is a round-down operation; S34. The processed video frames are discarded from memory, retaining only the tracking state of the current window and the last L / 2 frames of the previous window to address the memory shortage issue in long video tracking; the input sliding window video undergoes feature extraction and m corrections of the monitoring point positions using the Transformer network architecture: (4) in Multi-scale features extracted for CNN For RAFT-style correlation features, the number of iterations is m. This represents the pixel coordinates of the monitoring point after the (k+1)th iteration. This represents the estimated pixel coordinates of the monitoring points after the nth iteration; the window slides a distance of L / 2 frames each time, and the output is the pixel coordinate matrix of the monitoring points after sliding window processing. ,in This represents the pixel coordinate matrix of the monitoring points output by the nth sliding window. Represents the real number field. N Indicates the number of monitoring points; S4: Train and optimize the pixel tracking model to improve its robustness in complex environments; S5: Use depth estimation methods to convert pixel coordinates into three-dimensional spatial coordinates; S6: Perform time-series alignment, displacement calculation, cluster analysis, and threshold judgment on 3D coordinates, and trigger an alarm; S7: Dynamically visualize deformation data, generate monitoring reports, and output early warning information.

2. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The data acquisition module supports monocular, binocular, or multi-view imaging devices and has three operating modes: cloud, edge, or local. It supports real-time uploading, local caching, and resume download of video data.

3. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The pixel tracking module includes: The monitoring point initialization unit is used to delineate the monitoring area and generate gridded monitoring points in the first frame image; The feature extraction and tracking unit performs multi-scale feature extraction and iterative position correction based on a deep learning network. The sliding window processing unit is used to process the video stream in blocks, ensuring tracking continuity and memory optimization.

4. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The depth estimation module supports monocular depth estimation, binocular stereo matching, or active ranging, and converts the two-dimensional coordinates output by the pixel tracking module into three-dimensional world coordinates.

5. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The sliding window mechanism described in step S3 divides the video stream into multiple overlapping windows, each window being L frames long, with adjacent windows overlapping by L / 2 frames. The position of the monitoring point is iteratively corrected and a pixel coordinate matrix is ​​output.

6. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The depth estimation method described in step S5 includes: In binocular mode, disparity is calculated through camera calibration and stereo matching, thereby obtaining depth information; In monocular mode, depth is predicted using a pre-trained depth estimation network, and scale calibration is performed. In active ranging mode, distance information is obtained through laser ranging, structured light, or millimeter-wave radar.

7. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, In step S4, model training uses multi-scenario synthetic data or real-world videos, employing supervised learning, comparative learning, or iterative optimization methods. Once training is complete, the model can be deployed in the cloud, at the edge, or locally.

8. The real-time monitoring system for soil and rock deformation with high-density pixel tracking according to claim 1, characterized in that, The visualization in step S7 includes a three-dimensional deformation field, heat map, motion trajectory and dynamic display of sliding window, supporting user interaction and access from multiple platforms.

Images