Highway slope dangerous rock collapse early warning method and system based on computer vision

Abstract

This invention relates to the field of high-speed monitoring technology, specifically to a computer vision-based method and system for early warning of dangerous rockfalls on highway slopes. The method includes the following steps: S1, real-time acquisition of monitoring data, including visible light images, thermal images, laser point clouds, and environmental parameters, using sensing devices deployed along the highway slope; S2, detection of dangerous rocks and cracks in the visible light images using a deep learning target detection model, extracting cumulative displacement as a first state variable from the detection results, and simultaneously using a time-series tracking algorithm to track the trajectory of dangerous rocks in consecutive frames, calculating the instantaneous displacement rate as a second state variable; S3, description of the nonlinear coupling evolution relationship between the first and second state variables using a Lotka-Volterra type differential equation system; S4, online fitting using the real-time acquired time-series data via nonlinear least squares method.

Description

Technical Field

[0001] This invention relates to the field of high-speed monitoring technology, specifically to a method and system for early warning of dangerous rockfalls on highway slopes based on computer vision. Background Technology

[0002] In the field of early warning for dangerous rockfalls on highway slopes, existing technologies mainly rely on three methods: manual inspection, traditional sensor monitoring, and ordinary video surveillance. Manual inspection involves maintenance personnel periodically patrolling the slope, visually observing cracks, loose rocks, or abnormal deformations. While this method is intuitive, it has significant blind spots, especially on steep slopes or in densely vegetated areas, where inspectors cannot approach or clearly see the true condition of the dangerous rock. Furthermore, manual inspection cannot provide continuous real-time monitoring; if rock loosening or rapid displacement occurs between inspections, it often goes undetected, leading to a severe delay in early warning response. Traditional sensor monitoring involves embedding displacement gauges, crack gauges, inclinometers, piezometers, and GPS receivers at key locations on the slope. These sensors can provide high-precision single-point deformation data; for example, crack gauges can measure width changes at the millimeter level. However, their limitation is that sensors can only cover a limited number of points; if dangerous rock appears in areas without sensors, it will be completely undetectable. For highway slopes stretching for several kilometers, densely deploying sensors is not only costly but also difficult to construct and maintain. The laying of power and communication lines also disturbs the slope environment. While conventional video surveillance systems are widely deployed along highways, most are only used for security or traffic monitoring, lacking intelligent analysis capabilities. Monitoring requires real-time human supervision, and operators may miss critical moments due to distraction during rockfalls or landslides. Even systems incorporating motion detection algorithms are highly susceptible to interference from swaying leaves, changes in lighting, and passing vehicles, resulting in numerous false alarms. Furthermore, pure visible light cameras suffer severe image quality degradation at night, in heavy rain, or in dense fog, making it almost impossible to detect minute movements of distant rocks. While multispectral or thermal imaging equipment can improve environmental adaptability, their high cost and inability to address the core issues of target recognition and behavior prediction when used alone are significant drawbacks. In summary, existing technologies suffer from incomplete monitoring coverage, insufficient real-time performance, poor environmental adaptability, low intelligence, high false alarm rates, and an inability to balance cost and effectiveness. These shortcomings mean that current early warning systems for dangerous rockfalls on highway slopes still rely heavily on manual experience, making it difficult to achieve all-weather, highly reliable automated early warning. Summary of the Invention

[0003] The technical problem solved by this invention is to provide a computer vision-based method and system for early warning of dangerous rockfall on highway slopes. This system can achieve adaptive perception and accurate early warning of the evolution trend of dangerous rockfall by adopting a nonlinear dynamic model of rockfall displacement and rate and dynamically adjusting the early warning threshold.

[0004] The basic solution provided by this invention is a computer vision-based early warning method for dangerous rockfall on highway slopes, comprising the following steps: S1. Real-time monitoring data, including visible light images, thermal imaging images, laser point clouds, and environmental parameters, is collected by sensing devices deployed along the highway slopes. S2. Utilize a deep learning object detection model to detect unstable rocks and cracks in the visible light image, and extract the cumulative displacement from the detection results as the first state variable. Simultaneously, a time-series tracking algorithm is used to track the trajectory of the unstable rock in consecutive frames, and the instantaneous displacement rate is calculated as the second state variable. ; S3. Describe the first state variable using a Lotka-Volterra type differential equation system. With the second state variable The nonlinear coupling evolution relationship:

[0005] in, For the crack's own propagation rate, The displacement release inhibits crack propagation. The natural decay rate of displacement release. The coefficient by which crack propagation promotes displacement release; S4. Utilizing real-time data acquisition and Time series data are fitted and estimated online using the nonlinear least squares method. , , , The parameter value is determined and the parameter is periodically updated based on new data within the sliding time window. S5. According to the aforementioned dynamic model, in the phase plane Calculate the predefined boundary of the collapse hazard zone and the current state point. The shortest distance d to the boundary of the danger zone is used, and the warning sensitivity threshold is dynamically adjusted according to the following formula:

[0006] in Basic warning threshold, To maximize the reduction, σ is the bandwidth parameter; the lower the warning sensitivity threshold, the lower the distance d. S6. Input the current rockfall motion characteristics and environmental parameters into the risk assessment model, calculate the collapse risk probability p, and when p > When an event is identified as a dangerous incident, the system will trigger the corresponding early warning information push and emergency response measures according to the preset early warning level.

[0007] The principle and advantages of this invention are as follows: First, multiple sensors deployed on the slope collect visible light images, thermal images, laser point clouds, and environmental parameters such as rainfall and temperature. Taking a high highway slope as an example, after the system continuously acquires images, it extracts the cumulative displacement and instantaneous displacement rate of the unstable rock, which are used as two state variables. Then, a pair of differential equations, namely the Lotka-Volterra equations, are established, using four parameters to describe the mutual inhibition and excitation relationship between crack propagation and displacement release. Using the real-time acquired data, these four parameters are fitted online using the nonlinear least squares method, enabling the model to adapt to the current mechanical behavior of the slope. On the phase plane formed by the cumulative displacement and instantaneous velocity, the boundary of the collapse hazard area is predefined, and the shortest distance from the current state point to this boundary is calculated. The warning sensitivity threshold is dynamically adjusted according to this distance: the closer the distance, the lower the threshold, meaning it is easier to trigger a warning. Finally, the current motion characteristics and environmental parameters are input into the risk assessment model to obtain the collapse risk probability. When the probability exceeds the dynamic threshold, a warning is issued and traffic control equipment is activated.

[0008] Compared to existing technologies that use fixed warning thresholds, this solution adaptively adjusts the alarm sensitivity based on the actual evolution trend of the dangerous rock, avoiding frequent false alarms in safe conditions and preventing missed alarms due to excessively high thresholds when danger is imminent. For example, traditional systems may issue false alarms when the slope is slowly creeping due to a low fixed threshold, while this solution, with a higher threshold at a greater distance from the danger boundary, will not issue false alarms. Conversely, when the displacement rate suddenly increases, the distance rapidly shortens, the threshold decreases, and the system can promptly detect the danger. This adaptive mechanism based on a dynamic model significantly improves the accuracy and timeliness of the warning.

[0009] In the mechanical mechanism of slope collapse caused by unstable rockfall, there is a mutually restrictive and mutually stimulating relationship between crack propagation and displacement release. Continuous crack propagation increases the accumulation of strain energy within the unstable rock mass, providing an energy source for subsequent displacement release, equivalent to the positive effect of the crack term in the equation inducing the increase in displacement rate. Simultaneously, when the unstable rock begins to produce significant displacement, the displacement release process partially consumes the potential for crack propagation, thus inhibiting the crack propagation rate, corresponding to the negative feedback of the displacement term in the equation to the crack. Furthermore, without the supply of new cracks, displacement release will naturally decay due to rock mass friction and resistance, consistent with the negative decay term of the displacement rate in the equation. This coupling relationship is mathematically highly similar to the predator-prey interaction in ecology, therefore, it is reasonable to apply the Lotka-Volterra equation to the field of slope monitoring. Taking a highway slope that has experienced multiple minor collapses as an example, monitoring data shows that the crack width gradually increases from zero, at which point the displacement rate is almost zero. When the crack expands to a certain extent, the displacement rate begins to rise, while the crack expansion rate slows down. After the displacement rate reaches its peak, a collapse occurs, followed by a decrease in both crack width and displacement rate. The entire process exhibits self-limiting oscillatory characteristics, which can be approximated by a set of nonlinear differential equations. By fitting the four core parameters of the equations online using real-time collected cumulative displacement and instantaneous displacement rate data, the system can capture the current dynamic characteristics of the slope and predict whether it is trending towards a dangerous area. Compared to purely data-driven black-box models, the Lotka-Volterra equation has a simple structure, clear physical meaning of its parameters, and requires fewer training samples, making it particularly suitable for continuous operation on edge computing nodes. Therefore, using this equation as the dynamic core for early warning of slope rockfall is feasible and is the key difference between this invention and existing static threshold methods.

[0010] Furthermore, S2 includes the following steps: S21. Input the acquired visible light image into the improved YOLO deep learning model. The model adopts a training strategy without nonmaximum suppression and a dual label assignment mechanism to output the bounding box and target category of the dangerous rock target. S22. Track the image coordinates of the center point of the bounding box of the same dangerous rock target in consecutive frames, and convert the image coordinates into three-dimensional world coordinates by combining the pre-calibrated camera intrinsic and extrinsic parameters and laser point cloud ranging information. Calculate from the initial time Cumulative displacement up to the current time t As the first state variable:

[0011] S23. Use the DeepSORT algorithm to track and maintain the trajectory of the dangerous rock targets detected in consecutive frames, assign a unique tracking ID to each dangerous rock target, and record its three-dimensional world coordinates in each frame. S24. Based on the tracking trajectory, the displacement vectors between adjacent time frames are differentially analyzed to calculate the instantaneous displacement rate. As a second state variable:

[0012] For time intervals, when there are multiple targets being tracked, the rock mass with the largest displacement rate is selected as the dominant rock mass, and its rate is used as the second state variable.

[0013] The system extracts the first state variable (cumulative displacement) and the second state variable (instantaneous displacement rate) from visible light images. First, an improved YOLO deep learning model is used to detect unstable rock targets. This model removes the non-maximum suppression step and employs a dual-labeling mechanism, improving detection speed and accuracy. For the same unstable rock target in consecutive frames, the system tracks the image coordinates of its bounding box center point. Combining pre-calibrated camera parameters and laser point cloud ranging information, the image coordinates are converted into real-world 3D coordinates. Cumulative displacement refers to the total distance the unstable rock has moved in 3D space from the initial observation time to the current time, calculated as the square root of the sum of squares of the coordinate differences. Simultaneously, the DeepSORT algorithm assigns a unique number to each unstable rock and records its coordinates for each frame. The instantaneous displacement rate is obtained by removing bits between adjacent frames and dividing by the time interval. If multiple unstable rocks are tracked simultaneously, the one with the highest rate is selected as the dominant unstable rock. Taking a slope as an example, a loose rock slowly slides down over several hours, with the cumulative displacement gradually increasing from 0 to 50 mm, while the instantaneous rate may only be a few millimeters per hour.

[0014] Existing technologies often rely solely on manual observation or single-point displacement gauges to acquire displacement data, failing to accurately obtain continuous trajectories and real-time rates in three-dimensional space. This solution achieves non-contact, high-precision displacement measurement through visual tracking and laser point cloud fusion, avoiding the safety risks and construction difficulties associated with installing sensors on unstable rocks. Furthermore, the DeepSORT algorithm ensures that tracking can be resumed even after the unstable rock is partially obscured or temporarily disappears, improving data continuity and reliability. These precise state variables provide high-quality input for subsequent dynamic models, forming the foundation of the entire early warning method.

[0015] Furthermore, S3 includes the following steps: S31. Construct the standard Lotka-Volterra equations to describe the first state variables. With the second state variable The coupling evolution relationship:

[0016] S32. Rainfall is collected in real time using rain gauges deployed on the slope. Temperature is collected in real time by a temperature sensor. Using the rainfall and temperature as external driving terms, the equation is expanded as follows:

[0017] in, and The environmental coupling coefficients to be calibrated represent the accelerating effect of rainfall on crack propagation and the stimulating effect of temperature change on displacement release, respectively. S33. Based on the geological survey report of the slope, set the parameters. , , , , and The initial value; where, and The initial values ​​are set based on the degree of weathering and joint development of the slope rock mass. and The initial value is set based on the friction angle and cohesion of the slope. and The initial value is set based on the statistical relationship of historical rainfall-induced landslides.

[0018] First, a pair of differential equations in standard form are constructed, where the rate of change of cumulative displacement equals the self-expansion rate multiplied by the displacement minus the inhibition coefficient multiplied by the product of displacement and velocity, and the rate of change of instantaneous velocity equals the negative decay rate multiplied by the velocity plus the promotion coefficient multiplied by the product of displacement and velocity. These four parameters correspond to physical processes such as crack propagation, displacement inhibition, velocity decay, and displacement initiation, respectively. Based on this, real-time rainfall and temperature are introduced as external driving terms through rain gauges and temperature sensors. The extended equations are: the rate of change of cumulative displacement increased by the rainfall multiplied by a coupling coefficient, and the rate of change of instantaneous velocity increased by the temperature multiplied by another coupling coefficient. Rainfall accelerates crack propagation, and temperature changes cause thermal expansion and contraction of the rock mass, thus initiating displacement. These two environmental factors are important external conditions for inducing collapse. Taking a slope after continuous heavy rain as an example, the increased rainfall causes an additional increase in the rate of change of cumulative displacement, and the displacement growth predicted by the model will be faster than in the absence of rain, approaching the danger boundary earlier. While existing pure data-driven models such as LSTM can also consider rainfall, they require a large number of historical collapse samples to learn the relationship between rainfall and displacement and lack physical interpretability. The extended equations of this scheme are based on well-defined physical mechanisms, requiring only a few parameters to characterize the impact of environmental factors. The initial values ​​of these parameters can be set based on the weathering degree, joint development, friction angle, cohesion, and historical rainfall-landslide statistics from the slope's geological survey report. This approach, combining physical laws with data-driven methods, enables the model to provide reasonable predictions even on new slopes or in data-sparse scenarios, while also facilitating parameter understanding and adjustment by engineers.

[0019] Furthermore, S4 includes the following steps: S41. Establish a time window of fixed length L, and store the observation data of consecutive moments in the serial port. ,in, , The sampling interval for the state variables; S42. Parameter estimation is performed on the data within the sliding window using a nonlinear least squares algorithm. The optimization objective is to minimize the sum of squared errors between the model predictions and the actual observations, and the parameter vector is solved. The estimated value; S43. Perform parameter estimation once according to the preset cycle.

[0020] A fixed-length time window is established, for example, taking the observation data from the most recent 30 minutes. Within the window, continuous cumulative displacement and instantaneous velocity data pairs are stored at sampling intervals. A nonlinear least squares algorithm is employed, aiming to minimize the sum of squared errors between the rate of change predicted by the model's differential equation and the rate of change calculated from the actual observation data, thereby solving for the parameter vector. Taking a creeping slope as an example, the initial parameters are set according to the geological report, but the actual mechanical properties of the rock mass may differ from the report. The system re-performs parameter estimation every 5 or 10 minutes, with the sliding window continuously moving forward, discarding the oldest data and adding the latest data. The estimated parameters in this way can reflect changes in the recent evolution of the slope. For example, after continuous rainfall, the coefficient representing the acceleration effect of rainfall may increase, and the system will automatically capture this change. Many existing monitoring systems use fixed model parameters, which are not updated once deployed, and cannot adapt to changes in the mechanical properties of the slope caused by weathering, rainfall, or earthquakes. The online identification mechanism of this scheme ensures that the model always fits the current slope state. At the same time, periodic updates ensure a controllable computational load, making it suitable for running on edge computing devices. Furthermore, users can set an emergency update to be triggered when the fitting residuals exceed a certain threshold, preventing severe model mismatch. This dynamic parameter adjustment capability is not available in traditional static analysis or fixed threshold methods, significantly improving the accuracy of long-term monitoring.

[0021] Furthermore, S5 includes the following steps: S51. According to the dynamic model, in the phase plane Define the collapse hazard zone ; S52. Calculate the state point at the current moment. Euclidean distance from the boundary of the danger zone:

[0022] in The set of points representing the boundary of the danger zone; if the current state point is already within the danger zone, d=0. S53. Calculate the dynamic early warning threshold based on the Euclidean distance d: .

[0023] First, a collapse hazard zone is defined on a two-dimensional phase plane composed of cumulative displacement and instantaneous velocity. This definition can be based on the envelope of state point trajectories before historical collapse events, or on stability analysis of model equilibrium points. Taking a high slope as an example, assuming the cumulative displacement reaches 100 mm and the instantaneous velocity exceeds 20 mm / s, most historical collapse events have occurred near this area, and the system stores these boundary point sets. Then, the shortest Euclidean distance from the current state point to the boundary of the hazard zone is calculated. If the current cumulative displacement is 30 mm and the velocity is 5 mm / s, the point is far from the boundary, resulting in a large distance. If the displacement increases to 80 mm and the velocity rises to 15 mm / s, the distance decreases. In the formula for calculating the dynamic warning threshold, the base threshold is set to 0.7, the maximum reduction is set to 0.3, and the bandwidth parameter is set to 0.1. When the distance is large, the exponent term approaches 0, and the threshold remains at 0.7; when the distance approaches 0, the exponent term approaches 1, and the threshold drops to 0.4. This means that the closer to the hazard zone, the easier it is for the system to trigger a warning. Most existing technologies use fixed probability thresholds, such as 0.5 or 0.7, applying the same standard regardless of the current state of the slope. This method may trigger unnecessary alarms due to noise when the slope is far from danger, while delaying alarms due to a high threshold when the slope is actually close to danger. Our proposed solution uses a dynamic threshold that automatically adjusts based on real-time distance, achieving a reasonable distribution of sensitivity. Furthermore, because the distance calculation is based on a physical dynamics model, it is more robust and interpretable than methods relying solely on statistical data.

[0024] Furthermore, S6 includes the following steps: S61. Input the first state variable and its rate of change, the second state vector and its rate of change, and environmental parameters into the risk assessment model to obtain its collapse risk probability. S62. Integrate collapse risk probability with dynamic early warning thresholds The system compares and determines the safety status, and issues warnings based on the safety status.

[0025] The risk assessment model is input together with the current state variables: cumulative displacement and its rate of change (first state variable), instantaneous displacement rate and its rate of change (second state variable), and real-time environmental parameters such as rainfall, temperature, and humidity. This model is a pre-trained binary classifier, employing algorithms such as logistic regression or random forest. Training data comes from historical slope monitoring records and actual collapse event labels. The model outputs a collapse risk probability between 0 and 1. For example, in a slope monitoring scenario, the cumulative displacement is 45 mm increasing at a rate of 2 mm per hour, the instantaneous rate is 8 mm per second and accelerating, and the rainfall is 30 mm per hour. The model calculates a risk probability of 0.65. The dynamic warning threshold, calculated in previous steps, is 0.6. Since 0.65 is greater than 0.6, the system classifies it as a dangerous event. Warning levels are then assigned based on the magnitude of the probability exceeding the threshold: blue for a probability within 0.1, yellow for a probability between 0.1 and 0.2, and so on. Different levels trigger different information push methods: blue and yellow are sent to the management platform via 4G network, while orange and red are additionally sent repeatedly via BeiDou short messages and fiber optic lines. Simultaneously, the system automatically links to upstream information boards displaying warning text, switching traffic lights to red, triggering audible and visual alarms, and broadcasting voice reminders. Existing technologies often provide only a single alarm method and lack a tiered linkage mechanism, which may fail to effectively notify drivers in emergencies. This solution's multi-channel push and tiered linkage ensure reliable delivery of high-level warnings while avoiding excessive traffic disruption caused by low-level warnings. The entire process, from data acquisition to warning output, forms a closed loop, achieving automated and intelligent slope safety protection. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of an embodiment of the present invention. Detailed Implementation

[0027] The following detailed description illustrates the specific implementation method: The basic implementation examples are as follows: Figure 1 As shown: A computer vision-based early warning method for dangerous rockfall on highway slopes includes the following steps: S1. Real-time monitoring data, including visible light images, thermal imaging images, laser point clouds, and environmental parameters, is collected by sensing devices deployed along the highway slopes. S2. Utilize a deep learning object detection model to detect unstable rocks and cracks in the visible light image, and extract the cumulative displacement from the detection results as the first state variable. Simultaneously, a time-series tracking algorithm is used to track the trajectory of the unstable rock in consecutive frames, and the instantaneous displacement rate is calculated as the second state variable. ; S3. Describe the first state variable using a Lotka-Volterra type differential equation system. With the second state variable The nonlinear coupling evolution relationship:

[0028] in, For the crack's own propagation rate, The displacement release inhibits crack propagation. The natural decay rate of displacement release. The coefficient by which crack propagation promotes displacement release; S4. Utilizing real-time data acquisition and Time series data are fitted and estimated online using the nonlinear least squares method. , , , The parameter value is determined and the parameter is periodically updated based on new data within the sliding time window. S5. According to the aforementioned dynamic model, in the phase plane Calculate the predefined boundary of the collapse hazard zone and the current state point. The shortest distance d to the boundary of the danger zone is used, and the warning sensitivity threshold is dynamically adjusted according to the following formula:

[0029] in Basic warning threshold, To maximize the reduction, σ is the bandwidth parameter; the lower the warning sensitivity threshold, the lower the distance d. S6. Input the current rockfall motion characteristics and environmental parameters into the risk assessment model, calculate the collapse risk probability p, and when p > When an event is identified as a dangerous incident, the system will trigger the corresponding early warning information push and emergency response measures according to the preset early warning level.

[0030] S2 includes the following steps: S21. Input the acquired visible light image into the improved YOLO deep learning model. The model adopts a training strategy without nonmaximum suppression and a dual label assignment mechanism to output the bounding box and target category of the dangerous rock target. S22. Track the image coordinates of the center point of the bounding box of the same dangerous rock target in consecutive frames, and convert the image coordinates into three-dimensional world coordinates by combining the pre-calibrated camera intrinsic and extrinsic parameters and laser point cloud ranging information. Calculate from the initial time Cumulative displacement up to the current time t As the first state variable:

[0031] S23. Use the DeepSORT algorithm to track and maintain the trajectory of the dangerous rock targets detected in consecutive frames, assign a unique tracking ID to each dangerous rock target, and record its three-dimensional world coordinates in each frame. S24. Based on the tracking trajectory, the displacement vectors between adjacent time frames are differentially analyzed to calculate the instantaneous displacement rate. As a second state variable:

[0032] For time intervals, when there are multiple targets being tracked, the rock mass with the largest displacement rate is selected as the dominant rock mass, and its rate is used as the second state variable.

[0033] Specifically, taking a 200-meter-long high slope of a highway as an example, a high-definition intelligent imaging device is installed approximately 80 meters away on the opposite side of the slope. This device is equipped with a 12-megapixel CMOS sensor and a laser ranging module. The device is installed at a height of 6 meters and a downward angle of approximately 15 degrees to ensure coverage of the entire slope. After the system is started, camera calibration is performed first: a checkerboard calibration board is placed at multiple locations on the slope to acquire 20 images, and the camera intrinsic parameter matrix and distortion coefficients are calculated. Simultaneously, the 3D coordinates of at least 4 known points on the slope are obtained using laser point clouds, and the camera extrinsic parameters (rotation matrix and translation vector) are solved to complete the conversion from image coordinates to 3D world coordinates.

[0034] During continuous monitoring, a visible light image was acquired every 0.2 seconds. The images were then fed into an improved YOLOv10 model, pre-trained on a dedicated dataset containing 5000 images of mountain slopes, capable of identifying falling rocks, loose rocks, and cracks. At one point, the model detected a dangerous rock in the image, with the center pixel coordinates of its bounding box at (640, 480). The distance to this target, determined by laser ranging, was 82.3 meters. Using a coordinate transformation formula, the three-dimensional world coordinates were calculated to be (12.5, 34.2, 8.6) meters, where X represents the direction along the road, Y represents the horizontal direction perpendicular to the road, and Z represents the elevation. At the initial time t0, the coordinates of the dangerous rock were (12.5, 34.2, 8.6) meters, and the cumulative displacement was 0. After 10 seconds, the rock mass moved to (12.8, 34.5, 8.9) meters, with a cumulative displacement x(t) = √[(0.3)² + (0.3)² + (0.3)²] = 0.52 meters. The system records the coordinates in each frame, and the DeepSORT algorithm assigns ID=001 to the rock mass. The interval between adjacent frames is Δt = 0.2 seconds, and the instantaneous displacement rate y(t) is calculated as 0.52 / 10 = 0.052 m / s. If multiple rock masses are detected simultaneously, such as ID=002 and ID=003, their instantaneous rates are calculated separately, and the maximum value is taken as the second state variable. When the rock mass is briefly obscured by trees, DeepSORT predicts its position based on Kalman filtering, and continues tracking after it reappears to ensure the trajectory is not interrupted. In this way, the system continuously outputs the time series of cumulative displacement and instantaneous rate, providing input for the subsequent dynamic model.

[0035] Taking a 200-meter-long highway embankment as an example, a high-definition intelligent imaging device is installed approximately 80 meters away on the opposite side of the embankment. This device is equipped with a 12-megapixel CMOS sensor and a laser ranging module. The device is installed at a height of 6 meters and a downward angle of approximately 15 degrees to ensure coverage of the entire slope. After the system is started, camera calibration is performed first: a checkerboard calibration board is placed at multiple locations on the slope to acquire 20 images, and the camera intrinsic parameter matrix and distortion coefficients are calculated. Simultaneously, the 3D coordinates of at least four known points on the slope are obtained using laser point clouds, and the camera extrinsic parameters (rotation matrix and translation vector) are solved to complete the conversion from image coordinates to 3D world coordinates.

[0036] During continuous monitoring, a visible light image was acquired every 0.2 seconds. The images were then fed into an improved YOLOv10 model, pre-trained on a dedicated dataset containing 5000 images of mountain slopes, capable of identifying falling rocks, loose rocks, and cracks. At one point, the model detected a dangerous rock in the image, with the center pixel coordinates of its bounding box at (640, 480). The distance to this target, determined by laser ranging, was 82.3 meters. Using a coordinate transformation formula, the three-dimensional world coordinates were calculated to be (12.5, 34.2, 8.6) meters, where X represents the direction along the road, Y represents the horizontal direction perpendicular to the road, and Z represents the elevation. At the initial time t0, the coordinates of the dangerous rock were (12.5, 34.2, 8.6) meters, and the cumulative displacement was 0. After 10 seconds, the rock mass moved to (12.8, 34.5, 8.9) meters, with a cumulative displacement x(t) = √[(0.3)² + (0.3)² + (0.3)²] = 0.52 meters. The system records the coordinates in each frame, and the DeepSORT algorithm assigns ID=001 to the rock mass. The interval between adjacent frames is Δt = 0.2 seconds, and the instantaneous displacement rate y(t) is calculated as 0.52 / 10 = 0.052 m / s. If multiple rock masses are detected simultaneously, such as ID=002 and ID=003, their instantaneous rates are calculated separately, and the maximum value is taken as the second state variable. When the rock mass is briefly obscured by trees, DeepSORT predicts its position based on Kalman filtering, and continues tracking after it reappears to ensure the trajectory is not interrupted. In this way, the system continuously outputs the time series of cumulative displacement and instantaneous rate, providing input for the subsequent dynamic model.

[0037] S3 includes the following steps: S31. Construct the standard Lotka-Volterra equations to describe the first state variables. With the second state variable The coupling evolution relationship:

[0038] S32. Rainfall is collected in real time using rain gauges deployed on the slope. Temperature is collected in real time by a temperature sensor. Using the rainfall and temperature as external driving terms, the equation is expanded as follows:

[0039] in, and The environmental coupling coefficients to be calibrated represent the accelerating effect of rainfall on crack propagation and the stimulating effect of temperature change on displacement release, respectively. S33. Based on the geological survey report of the slope, set the parameters. , , , , and The initial value; where, and The initial values ​​are set based on the degree of weathering and joint development of the slope rock mass. and The initial value is set based on the friction angle and cohesion of the slope. and The initial value is set based on the statistical relationship of historical rainfall-induced landslides.

[0040] Specifically, at a monitoring point on a highway slope, the geological survey report showed that the slope consisted of interbedded sandstone and mudstone, with moderate weathering, three sets of joints, and a structural plane connectivity rate of approximately 60%. Initial parameters were set as follows: based on the sandstone's tensile strength of approximately 8 MPa, α = 0.008 m³ / year; based on the integrity coefficient of 0.6, β = 0.025 m. - Based on the moderate weathering resistance, γ = 0.012 m³ / year is taken; based on the structural surface connectivity rate of 60%, δ = 0.5 is taken. In addition, based on the statistical relationship of historical rainfall-induced landslides in this area, k1 = 0.003 m³ / year·mm / h and k2 = 0.001 m³ / year·℃ are taken.

[0041] Suppose that continuous rainfall begins at 14:00 on a certain day, and the rain gauge measures 20 mm of rainfall within 15 minutes, with an instantaneous rainfall intensity R(t) = 80 mm / h. The temperature sensor displays the current air temperature as 28℃. The system incorporates external driving terms into the equation: =0.008x-0.025xy+0.003×80=0.008x-0.025xy+0.24.

[0042] =-0.012y+0.5xy+0.001×28=-0.012y+0.5xy+0.028.

[0043] If the current state variables are x = 30 mm (0.03 m) and y = 5 mm / s (0.005 m / s), then dx / dt ≈ 0.008 × 0.03 - 0.025 × 0.03 × 0.005 + 0.24 ≈ 0.00024 - 0.00000375 + 0.24 = 0.24023625 m / year, meaning that rainfall contributes the vast majority of the displacement growth rate; dy / dt ≈ -0.012 × 0.005 + 0.5 × 0.03 × 0.005 + 0.028 = -0.00006 + 0.000075 + 0.028 = 0.028015 m / s². It is evident that rainfall and temperature significantly alter the evolutionary trend.

[0044] When the rain stops, R(t) = 0, and dx / dt recovers to 0.008x - 0.025xy. At this point, if x and y are small, the displacement increases slowly. By comparing the outputs of models with and without rainfall-driven models, the system can predict the accelerated displacement that may be caused by heavy rainfall several hours in advance, thus issuing early warnings.

[0045] S4 includes the following steps: S41. Establish a time window of fixed length L, and store the observation data of consecutive moments in the serial port. ,in, , The sampling interval for the state variables; S42. Parameter estimation is performed on the data within the sliding window using a nonlinear least squares algorithm. The optimization objective is to minimize the sum of squared errors between the model predictions and the actual observations, and the parameter vector is solved. The estimated value; S43. Perform parameter estimation once according to the preset cycle.

[0046] Specifically, with a sliding window length L = 30 minutes and a state variable sampling interval Δt = 5 seconds, the number of data points within the window is N = 30 × 60 / 5 = 360. The system performs parameter estimation every 10 minutes. Within a certain 10-minute period, the window stores cumulative displacement x(t) and instantaneous velocity y(t) data pairs from 10:00:00 to 10:30:00. An example of one set of data is shown in Table 1 below: Table 1

[0047] A nonlinear least squares method is employed, defining the residual function as the sum of squares of the differences between the model-predicted dx / dt and dy / dt at each time step and the differences calculated from the observed values. The observed dx / dt is calculated using the central difference method: for example, at t=10:00:05, the values ​​before and after x are calculated to be dx / dt≈(12.5-12.3) / 10=0.02mm / s. Similarly, dy / dt can be obtained. Then, the parameter vector θ=(α,β,γ,δ,k1,k2) is optimized to minimize the total error. The Levenberg-Marquardt algorithm is used iteratively, typically converging after 10-20 iterations. The estimation results in this study are α=0.0075, β=0.024, γ=0.013, δ=0.51, k1=0.0031, k2=0.00098. Compared to the previous parameters, α decreased slightly while γ increased slightly, indicating that the crack propagation rate decreased while the constraint degradation rate increased during this period. The system updates the new parameters smoothly: taking the smoothing factor ρ=0.8, then α=0.8×0.008+0.2×0.0075=0.0079. If the change in α in three consecutive estimates is less than 0.0002, then convergence is determined, and active updates are stopped, but emergency updates triggered by rainfall are still retained.

[0048] If the fitting residual exceeds a preset threshold (twice the mean of the initial calibration residuals), the system immediately triggers an emergency update instead of waiting for a 10-minute period. For example, after a sudden earthquake, if the displacement data shows abnormal fluctuations and the residuals increase sharply, the system will re-estimate the parameters within 3 seconds to ensure that the model quickly adapts to the new mechanical state.

[0049] S5 includes the following steps: S51. According to the dynamic model, in the phase plane Define the collapse hazard zone ; S52. Calculate the state point at the current moment. Euclidean distance from the boundary of the danger zone:

[0050] in The set of points representing the boundary of the danger zone; if the current state point is already within the danger zone, d=0. S53. Calculate the dynamic early warning threshold based on the Euclidean distance d: .

[0051] Specifically, the collapse hazard zone is first defined based on the dynamic model. Taking a certain slope as an example, the numerical integration method is used: using the state points one hour before multiple historical collapse events as initial conditions, the differential equation is simulated in a forward direction to obtain a series of critical trajectories. The envelope of these trajectories is taken as the hazard boundary. Specifically, five small collapse events of the slope in the past three years were collected. The cumulative displacement x before each collapse was between 80 and 120 mm, and the instantaneous velocity y was between 15 and 25 mm / s. These point sets are plotted on the phase plane, and a boundary line is fitted using a spline curve, with the equation y = 0.2x - 4 (approximately linear). The hazard zone is then defined as the region where y ≥ 0.2x - 4 and x ≥ 80.

[0052] At the current time t, the system measures x = 65 mm and y = 12 mm / s. Calculate the shortest Euclidean distance d from this point to the boundary. First, calculate the distance from the point to the line y = 0.2x - 4, transforming it to the standard form 0.2xy - 4 = 0. The distance formula is d = |0.2 × 65 - 12 - 4| / √(0.2² + 1²) = |13 - 16| / √1.04 = 3 / 1.02 ≈ 2.94. Since x = 65 < 80, the actual boundary still has a vertical segment, so it is necessary to verify whether the point corresponding to the minimum distance is on the line segment. Through numerical search, the actual nearest boundary point is found to be (80, 12), with a distance d = √[(65 - 80)² + (12 - 12)²] = 15. Take the smaller value, so d = 2.94 (because the boundary is also defined when x < 80). Set the base threshold θ_base = 0.7, the maximum reduction η = 0.3, and the bandwidth parameter σ = 0.1. ,=exp(-2.94² / (2×0.01))=exp(-8.6436 / 0.02)=exp(-432.18)≈0, therefore =0.7-0.3×0=0.7, the threshold remains unchanged.

[0053] If x = 95 mm and y = 18 mm / s, then d = 0 (because we have entered the danger zone), and the exponent term = 1. =0.7-0.3=0.4, the threshold decreases. If x=85mm, y=14mm / s, calculate the distance from the point to the line d=|0.2×85-14-4| / √1.04=|17-18| / 1.02=0.98, the exponential term=exp(-0.98² / 0.02)=exp(-0.9604 / 0.02)=exp(-48.02)≈0, at this time although it is close but not entered, the threshold is still 0.7. The system updates the dynamic threshold every 0.2 seconds and performs exponential smoothing: take λ=0.3, θ(t)=0.3× (t)+0.7×θ(t-Δt).

[0054] S6 includes the following steps: S61. Input the first state variable and its rate of change, the second state vector and its rate of change, and environmental parameters into the risk assessment model to obtain its collapse risk probability. S62. Integrate collapse risk probability with dynamic early warning thresholds The system compares and determines the safety status, and issues warnings based on the safety status.

[0055] Specifically, the risk assessment model employs a random forest algorithm, pre-trained using 500 sets of historical slope monitoring data. Each set of data includes a feature vector and a label (whether it has collapsed). The feature vector includes: cumulative displacement x, cumulative displacement change rate dx / dt, instantaneous velocity y, instantaneous velocity change rate dy / dt, rainfall R, temperature T, and humidity H. The model outputs the collapse risk probability p∈[0,1].

[0056] In a certain actual monitoring, the current data is: x=72mm, dx / dt=0.5mm / s, y=11mm / s, dy / dt=0.2mm / s², R=15mm / h, T=26℃, H=85%. Input a random forest model with 100 decision trees, each tree independently outputting 0 or 1. Count the proportion of trees outputting 1. For example, if 85 trees output 1, then p=0.85.

[0057] At this time, the dynamic early warning threshold is calculated to be: =0.6, since p=0.85>0.6, it is judged as a dangerous event. Calculate the excess amplitude Δp=p- =0.25. According to the classification rules: 0.1≤Δp<0.2 is yellow, and 0.2≤Δp<0.3 is orange, thus triggering an orange alert. The system immediately sends an alert message to the highway management center via the 4G network, containing information such as the slope marker, location of the dangerous rock, and risk probability; simultaneously, it sends an encrypted data packet via Beidou short message as a backup. The actual communication delay was 2.1 seconds. The system also triggers a pulse signal: controlling the information board 1 km upstream to display "Risk of falling rocks ahead, slow down," switching the traffic light at the entrance of that section to red, and triggering the roadside audible and visual alarms to sound and flash red. The status of all linked devices is transmitted back to the cloud within 1.5 seconds, indicating successful execution. If the orange alert persists for 5 minutes without being lifted, the system automatically upgrades to a red alert, expanding the linkage range to 3 km upstream, and sending a voice call to the traffic police command center via a dedicated fiber optic line.

[0058] If p=0.55 and θ_dynamic=0.6, no warning will be triggered, and the system will continue monitoring. However, it will record any suspected events at that moment for subsequent offline analysis.

[0059] In actual deployment of the complete system, the above S1 to S6 are integrated into an edge computing device. The device uses the NVIDIA Jetson Orin NX module, runs the Ubuntu 20.04 system, and uses mixed programming with Python and C++. The multi-modal sensors are connected through Ethernet and the RS485 bus. After the system starts, it first loads the pre-trained YOLOv10 model and the random forest model, and reads the initial parameters of the geological exploration report. Subsequently, it enters the main loop: capture one frame of image every 0.2 seconds, perform object detection and tracking, and extract state variables; update the situation awareness (lighting, rain, fog, etc.) and adjust the detection frame rate every 5 seconds; perform online parameter identification every 10 minutes; calculate the dynamic threshold for each frame and compare it with the risk probability. When a warning is triggered, an independent thread is called to execute the linkage control. The system runs continuously for 30 days, and a total of 3 high-risk rock displacement acceleration events are detected, and warnings are issued 5 to 20 minutes in advance, without any missed reports, and the false alarm rate is less than 3%. The average daily CPU occupancy rate is about 45%, the memory occupancy is about 2GB, and the power consumption is about 18W, meeting the requirements of edge deployment.

[0060] The above are only embodiments of the present invention. Specific structures and characteristics and other common knowledge in the solution are not described in detail here. Those of ordinary skill in the art know all the common technical knowledge in the technical field to which the invention pertains before the filing date or the priority date, can know all the prior art in this field, and have the ability to apply conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given in this application, combine their own abilities to complete and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be noted that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicality of the patent. The protection scope required by this application should be based on the content of its claims, and the specific implementation manners and other records in the specification can be used to explain the content of the claims.

Claims

1. A computer vision-based early warning method for dangerous rockfall on highway slopes, characterized by: Includes the following steps: S1. Real-time monitoring data, including visible light images, thermal imaging images, laser point clouds, and environmental parameters, is collected by sensing devices deployed along the highway slopes. S2. Utilize a deep learning object detection model to detect unstable rocks and cracks in the visible light image, and extract the cumulative displacement from the detection results as the first state variable. Simultaneously, a time-series tracking algorithm is used to track the trajectory of the unstable rock in consecutive frames, and the instantaneous displacement rate is calculated as the second state variable. ; S3. Describe the first state variable using a Lotka-Volterra type differential equation system. With the second state variable The nonlinear coupling evolution relationship: in, For the crack's own propagation rate, The displacement release inhibits crack propagation. The natural decay rate of displacement release. The coefficient by which crack propagation promotes displacement release; S4. Utilizing real-time data acquisition and Time series data are fitted and estimated online using the nonlinear least squares method. , , , The parameter value is determined and the parameter is periodically updated based on new data within the sliding time window. S5. According to the aforementioned dynamic model, in the phase plane Calculate the predefined boundary of the collapse hazard zone and the current state point. The shortest distance d to the boundary of the danger zone is used, and the warning sensitivity threshold is dynamically adjusted according to the following formula: in Basic warning threshold, To maximize the reduction, σ is the bandwidth parameter; the lower the warning sensitivity threshold, the lower the distance d. S6. Input the current rockfall movement characteristics and environmental parameters into the risk assessment model, calculate the collapse risk probability p, and when p > When an event is identified as a dangerous incident, the system will trigger the corresponding early warning information push and emergency response measures according to the preset early warning level.

2. The method for early warning of dangerous rockfall on highway slopes based on computer vision according to claim 1, characterized in that: S2 includes the following steps: S21. Input the acquired visible light image into the improved YOLO deep learning model. The model adopts a training strategy without nonmaximum suppression and a dual label assignment mechanism to output the bounding box and target category of the dangerous rock target. S22. Track the image coordinates of the center point of the bounding box of the same dangerous rock target in consecutive frames, and convert the image coordinates into three-dimensional world coordinates by combining the pre-calibrated camera intrinsic and extrinsic parameters and laser point cloud ranging information. Calculate from the initial time Cumulative displacement up to the current time t As the first state variable: S23. Use the DeepSORT algorithm to track and maintain the trajectory of the dangerous rock targets detected in consecutive frames, assign a unique tracking ID to each dangerous rock target, and record its three-dimensional world coordinates in each frame. S24. Based on the tracking trajectory, the displacement vectors between adjacent time frames are differentially analyzed to calculate the instantaneous displacement rate. As a second state variable: For time intervals, when there are multiple targets being tracked, the rock mass with the largest displacement rate is selected as the dominant rock mass, and its rate is used as the second state variable.

3. The method for early warning of dangerous rockfall on highway slopes based on computer vision according to claim 2, characterized in that: S3 includes the following steps: S31. Construct the standard Lotka-Volterra equations to describe the first state variables. With the second state variable The coupling evolution relationship: S32. Rainfall is collected in real time using rain gauges deployed on the slope. Temperature is collected in real time by a temperature sensor. Using the rainfall and temperature as external driving terms, the equation is expanded as follows: in, and The environmental coupling coefficients to be calibrated represent the accelerating effect of rainfall on crack propagation and the stimulating effect of temperature change on displacement release, respectively. S33. Based on the geological survey report of the slope, set the parameters. , , , , and The initial value; where, and The initial values ​​are set based on the degree of weathering and joint development of the slope rock mass. and The initial value is set based on the friction angle and cohesion of the slope. and The initial value is set based on the statistical relationship of historical rainfall-induced landslides.

4. The method for early warning of dangerous rockfall on highway slopes based on computer vision according to claim 3, characterized in that: S4 includes the following steps: S41. Establish a time window of fixed length L, and store the observation data of consecutive moments in the serial port. ,in, , The sampling interval for the state variables; S42. Parameter estimation is performed on the data within the sliding window using a nonlinear least squares algorithm. The optimization objective is to minimize the sum of squared errors between the model predictions and the actual observations, and the parameter vector is solved. The estimated value; S43. Perform parameter estimation once according to the preset cycle.

5. The method for early warning of dangerous rockfall on highway slopes based on computer vision according to claim 4, characterized in that: S5 includes the following steps: S51. According to the dynamic model, in the phase plane Define the collapse hazard zone ; S52. Calculate the state point at the current moment. Euclidean distance from the boundary of the danger zone: in The set of points representing the boundary of the danger zone; if the current state point is already within the danger zone, d=0. S53. Calculate the dynamic early warning threshold based on the Euclidean distance d: 。 6. The method for early warning of dangerous rock collapse on highway slopes based on computer vision according to claim 5, characterized in that: S6 includes the following steps: S61. Input the first state variable and its rate of change, the second state vector and its rate of change, and environmental parameters into the risk assessment model to obtain its collapse risk probability. S62. Integrate collapse risk probability with dynamic early warning thresholds The system compares and determines the safety status, and issues warnings based on the safety status.

7. A computer vision-based early warning system for dangerous rockfall on highway slopes, characterized in that: The computer vision-based early warning method for dangerous rock collapse on highway slopes, as described in any one of claims 1-6, was used.

Images