A complex terrain mountain rockfall real-time monitoring and early warning method and system based on multi-modal spatio-temporal fusion and edge adaptive reasoning

Abstract

The application discloses a complex terrain mountain rockfall real-time monitoring and early warning method and system based on multi-modal space-time fusion and edge adaptive reasoning, and belongs to the technical field of intelligent geological disaster monitoring and computer vision. The method obtains multi-modal video streams under complex terrain by deploying a visible light and thermal imaging dual-light acquisition unit; runs a lightweight multi-modal fusion detection network by using an edge computing node, extracts texture features and thermal radiation features of a target, and combines a space-time sequence analysis module to construct a rockfall discrimination model based on physical motion constraints; introduces a dynamic background difference and interference suppression mechanism to effectively filter false positives caused by vegetation shaking, animal activity and light and shadow changes; and predicts the impact risk level of rockfall according to the rockfall trajectory to trigger a graded early warning strategy.

Description

Technical Field

[0001] This invention relates to the fields of geological disaster prevention and control and artificial intelligence technology application, and in particular to a method and system for real-time monitoring and early warning of rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning. Background Technology

[0002] With the rapid development of transportation networks, tourist attractions, and infrastructure in mountainous areas of my country, landslides caused by steep slopes are becoming increasingly frequent, seriously threatening people's lives and property. Traditional methods of monitoring landslides mainly rely on manual inspections, passive protective nets, and single sensors (such as vibration sensors, displacement gauges, and radar).

[0003] However, single vibration sensors are susceptible to interference from wind, rain, and vehicle vibrations, resulting in a high false alarm rate; displacement gauges can only monitor macroscopic displacements that have already occurred and cannot detect sudden small-scale rockfalls; radar equipment is expensive and easily affected by vegetation obstruction. Traditional visible light video surveillance is almost ineffective under low light or low visibility conditions such as nighttime, heavy fog, and heavy rain, making all-weather monitoring impossible. Existing video solutions mostly adopt a "front-end acquisition + cloud analysis" model. Uploading massive amounts of video data to the cloud not only consumes valuable and unstable network bandwidth in mountainous areas, but also causes transmission delays that lead to untimely warnings, often missing the best time for evacuation. Existing visual algorithms struggle to distinguish falling rocks from rolling branches, running animals, or changes in light and shadow, lacking a deep understanding of the physical laws governing object motion, leading to frequent false alarms and reducing the system's reliability.

[0004] Therefore, developing an intelligent rockfall monitoring and early warning system that can adapt to complex field environments, has all-weather high-precision detection capabilities, and features low latency and low false alarm rate has become an urgent need in the field of geological disaster prevention and control. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time monitoring and early warning method and system for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning, aiming to solve the problems of poor environmental adaptability, difficulty in nighttime detection, high false alarm rate and large response delay in the existing technology.

[0006] In a first aspect, the present invention provides a method for real-time monitoring and early warning of rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive inference, comprising the following steps:

[0007] Step S1: Acquire the visible light image sequence and thermal imaging image sequence of the monitoring area, and perform time synchronization and spatial geometric registration on the visible light image sequence and the thermal imaging image sequence to obtain a spatiotemporally aligned dual-modal image sequence;

[0008] Step S2: Based on the multimodal feature fusion detection network, each frame of the bimodal image in the spatiotemporally aligned bimodal image sequence is input into the multimodal feature fusion detection network frame by frame. The visible light features and thermal imaging features are fused at the feature level to extract deep semantic features and output the initial rockfall detection result containing the target location, target category and detection confidence.

[0009] Step S3: Based on the initial rockfall detection results of multiple consecutive frames, and based on the spatiotemporal trajectory tracking module, perform cross-frame association and trajectory generation for the same target, calculate the target's motion vector, instantaneous velocity, acceleration, and motion direction angle; combine with the pre-acquired mountain slope and terrain prior information, and use the rockfall physical motion constraint model to screen out candidate targets and their corresponding motion trajectories whose motion characteristics conform to the rockfall motion law.

[0010] Step S4: Construct a dual-modal real-time background model, remove false targets, and combine it with a pre-established interference feature library to perform secondary verification on the candidate targets to obtain confirmed real falling rock targets;

[0011] Step S5: Based on the confirmed pixel scale of the real falling rock target in the image, camera intrinsic and extrinsic parameters, and prior terrain information, estimate the volume and spatial velocity of the falling rock, and combine the terrain slope and trajectory to predict the landing point and impact range of the falling rock.

[0012] Furthermore, in step S1, the visible light image sequence and the thermal imaging image sequence are synchronized in time and registered spatially, including:

[0013] The exposure and acquisition of the visible light camera and the thermal imaging camera are controlled synchronously to ensure that the acquisition frame rates of the two cameras are the same.

[0014] Extract the timestamps of visible light frames and thermal imaging frames, use the nearest neighbor time difference matching algorithm to calculate the time difference between frames, filter out frame pairs with a time difference less than or equal to a preset threshold, and establish an initial time synchronization correspondence.

[0015] Frames with missing frames, delayed frames, or abnormal timestamps are filtered out. If there is a frame rate discrepancy between the two sequences, the timing is unified by resampling, and a time-synchronized visible light-thermal imaging frame pair sequence is output.

[0016] The homography matrix H between the visible light camera and the thermal imaging camera is obtained in advance using the calibration plate method;

[0017] The homography matrix H is used to perform perspective transformation on the thermal imaging frame to achieve preliminary spatial mapping, resulting in a pre-registered thermal imaging frame.

[0018] ORB feature points are extracted from the pre-registered thermal imaging frame and the corresponding visible light frame, feature point matching is performed, and pixel coordinate deviation is calculated.

[0019] If the pixel coordinate deviation exceeds a preset threshold, the homography matrix H is dynamically corrected based on the deviation value to obtain the corrected homography matrix H′. The corrected homography matrix H′ is then used to perform secondary perspective transformation and bilinear pixel interpolation on the thermal imaging frame to complete spatial registration, so that the pixel coordinate deviation of the same physical target in the visible light image and the thermal imaging image is less than the preset threshold.

[0020] Furthermore, the multimodal feature fusion detection network includes, in sequence, an input preprocessing module, a bimodal feature encoding module, a cross-modal feature fusion module, a multi-scale feature enhancement module, a detection prediction module, a candidate box filtering module, and a result encapsulation module;

[0021] The preprocessing module is used to: receive spatiotemporally aligned bimodal image pairs from each frame of a spatiotemporally aligned bimodal image sequence, wherein the bimodal image pair includes a visible light image Ivis(t), a thermal imaging image Ithr(t), a corresponding frame number t, and timestamp information; perform size normalization, pixel value normalization, and illumination enhancement processing on the visible light image Ivis(t), and perform size normalization, temperature value normalization, and thermal radiation gradient preservation processing on the thermal imaging image Ithr(t) to ensure that both have the same input resolution and retain the scaling parameters mapped back from the network input scale to the original image scale; if the thermal imaging is a single-channel grayscale image, the RGB three channels of the visible light image are concatenated with one channel of the thermal imaging image to form a four-channel input tensor; if the thermal imaging is a three-channel pseudo-color image, the RGB three channels of the visible light image are concatenated with the RGB three channels of the thermal imaging image to form a six-channel input tensor;

[0022] The dual-modal feature encoding module includes a visible light feature encoding branch and a thermal imaging feature encoding branch. The visible light feature encoding branch is used to extract the color, texture, edge, and contour features of the falling rock target from Ivis(t), resulting in visible light shallow features Fvis1, visible light mid-layer features Fvis2, and visible light deep features Fvis3. The thermal imaging feature encoding branch is used to extract the temperature difference, thermal radiation boundary, and thermal gradient features between the falling rock target and the mountain background from Ithr(t), resulting in thermal imaging shallow features Fthr1, thermal imaging mid-layer features Fthr2, and thermal imaging deep features Fthr3.

[0023] The cross-modal feature fusion module is used to perform feature alignment, channel concatenation, and adaptive weight allocation on Fvis1 and Fthr1, Fvis2 and Fthr2, and Fvis3 and Fthr3 at the same scale, respectively, to obtain fused shallow features Ffuse1, fused mid-level features Ffuse2, and fused deep features Ffuse3; wherein, the adaptive weight allocation is implemented through a lightweight channel attention unit;

[0024] The multi-scale feature enhancement module is used to perform top-down and bottom-up feature transfer on the fused shallow feature Ffuse1, fused mid-level feature Ffuse2, and fused deep feature Ffuse3, forming shallow multi-scale detection feature map P1, mid-level multi-scale detection feature map P2, and deep multi-scale detection feature map P3; among them, P1 is used to detect small-sized falling rock targets at a distance, P2 is used to detect medium-sized falling rock targets, and P3 is used to detect large-sized falling rock targets at close range;

[0025] The detection and prediction module is used to perform target classification, bounding box regression, and target confidence prediction on the shallow multi-scale detection feature map P1, the medium multi-scale detection feature map P2, and the deep multi-scale detection feature map P3, respectively, and outputs the predicted class probability, target confidence, and bounding box parameters for each candidate target; the bounding box parameters include the target center point coordinates, bounding box width, and bounding box height;

[0026] The candidate box filtering module is used to perform confidence threshold filtering, bounding box coordinate decoding, original image scale restoration and non-maximum suppression processing on the candidate targets output by the detection and prediction module, and remove candidate boxes with too high overlap or too low confidence to obtain the initial set of falling rock targets in each frame image.

[0027] The result encapsulation module is used to organize the initial set of falling rock targets in each frame image into an initial falling rock detection result Rt. Rt includes at least the frame number t, the number of falling rock targets Nt, and the target number, bounding box position, category label, and detection confidence of each falling rock target.

[0028] Furthermore, the spatiotemporal trajectory tracking module takes as input the initial rockfall detection results and their timestamps for multiple consecutive frames, camera calibration parameters, and terrain prior information; and outputs a set of candidate targets with unique trajectory IDs, the set of candidate targets including the trajectory sequence, bounding box position, category label, detection confidence, motion vector, instantaneous velocity, acceleration, motion direction angle, and trajectory state of each target.

[0029] The spatiotemporal trajectory tracking module is used to perform the following steps:

[0030] Invalid targets with a confidence level below 0.5 and false targets with no matching in a single frame are removed from the initial rockfall detection results;

[0031] An improved lightweight ByteTrack multi-target tracking algorithm is adopted, which combines target appearance features with Kalman filter motion prediction features to achieve optimal matching between the target in the current frame and the trajectory in the previous frame, assigning a unique identifier ID to the same target and realizing cross-frame association.

[0032] Set the trajectory buffer lifecycle to 2 to 3 frames;

[0033] An initial trajectory is generated for targets that are matched successfully for three or more consecutive frames, and the trajectory sequence is updated in real time, eliminating trajectories with insufficient matching frames or no non-continuous motion sequence.

[0034] Furthermore, based on the initial rockfall detection results of multiple consecutive frames, and using the spatiotemporal trajectory tracking module, cross-frame association and trajectory generation are performed on the same target, calculating the target's motion vector, instantaneous velocity, acceleration, and motion direction angle. Combined with pre-acquired prior information on mountain slope and terrain, a rockfall physical motion constraint model is used to screen candidate targets whose motion characteristics conform to the rockfall motion law and their corresponding motion trajectories, including:

[0035] Obtain the slope θ of the mountain slope in the monitoring area and complete the mapping transformation between the pixel coordinate system and the three-dimensional geographic coordinate system of the slope.

[0036] The target centroid pixel coordinates are converted into actual physical coordinates of the slope. For each valid trajectory, the target single-frame motion vector and instantaneous velocity (decomposed horizontal component V) are calculated sequentially. x Vertical component V y The resultant velocity V), the tangential acceleration along the direction of roll on the slope, and the angle between the motion vector and the horizontal line of the slope;

[0037] The target single-frame motion vector is decomposed into horizontal and vertical downward components of the slope; the instantaneous velocity is decomposed into a horizontal component V. x Vertical component V y Resultant velocity V;

[0038] Set up three-layer verification:

[0039] The first layer, velocity component constraint: requires the velocity component in the vertically downward direction to be constrained across multiple consecutive frames. Always greater than the horizontal component of velocity The preset multiplier is used to eliminate interfering targets whose horizontal movement is dominant.

[0040] The second layer is acceleration trend constraint: it requires that the resultant velocity V has a non-linear acceleration trend over time, and the acceleration value is within a reasonable range, eliminating non-falling rock targets that are moving at a constant speed or wandering randomly.

[0041] The third layer is terrain prior verification: invalid targets whose movement trajectories deviate from the preset legal rolling area or have anti-gravity movement are eliminated;

[0042] The targets that pass the above three-layer verification and their corresponding motion trajectories are used as candidate falling rock targets and candidate motion trajectories.

[0043] Furthermore, in step S4, a bimodal real-time background model is constructed to remove false targets, including:

[0044] We selected consecutive frames of bimodal images without abnormal disturbances, and used a lightweight Gaussian mixture model to initialize the visible light background model and the thermal imaging background model respectively. We established three Gaussian distributions for each pixel to represent the multimodal changes of the background.

[0045] Constructing pixel value probability density :

[0046] ;

[0047] in The value is the current pixel value; for visible light, it's the grayscale / color value; for thermal imaging, it's the temperature value. The weight of the i-th Gaussian component is... The mean, Let K be the variance, and K be the total number of Gaussian components.

[0048] When the current pixel value x t The probability P(x) under the Gaussian mixture background model t If P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel; when P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel. t When the pixel value is below the preset background matching threshold, it indicates that the pixel value deviates significantly from the historical background distribution, and it is determined to be a foreground pixel.

[0049] By statistically analyzing the proportion of foreground pixels within the bounding box region of candidate targets, it is determined whether the proportion of foreground pixels is within a preset range, thereby eliminating false targets with no significant foreground changes.

[0050] Furthermore, in step S4, the candidate targets are further verified using a pre-established interference feature library to obtain confirmed real rockfall targets, including:

[0051] Pre-collect dual-modal samples of typical interference sources such as vegetation swaying, running animals, cloud shadows, sudden changes in lighting, and cloud and fog obstruction in the whole scene;

[0052] For each type of interference source, the following three types of features are extracted:

[0053] Motion pattern characteristics: frequency domain distribution of optical flow field and optical flow entropy value;

[0054] Appearance features: contour stability, texture complexity, and aspect ratio;

[0055] Thermal infrared characteristics: rate of change of temperature gradient and thermal radiation stability;

[0056] Establish a categorized interference feature index library and set feature matching thresholds for each type of feature;

[0057] During the verification phase, the optical flow entropy value, contour stability, and temperature gradient change rate of the candidate target are calculated and matched with the corresponding features in the interference feature library, respectively.

[0058] A three-level fusion judgment rule is adopted:

[0059] If a candidate target is marked as a matching interference feature library in two or more dimensions of motion pattern, appearance features, and thermal infrared features, it is directly determined as a false target and removed.

[0060] If only one dimension shows an abnormal match, then a composite verification is performed using the aforementioned physical motion constraint model of falling rocks.

[0061] If all three dimensions match the characteristics of falling rocks, then the target of falling rocks is confirmed.

[0062] Furthermore, in step S5, estimating the volume of the falling rock includes:

[0063] Obtain the pixel width from the bounding box of the initial rockfall detection result. and pixel height H p ;

[0064] Based on the pinhole imaging perspective model, and combining the camera focal length f, pixel physical size s, and the actual distance D from the centroid of the falling rock to the camera, the pixel width and pixel height are converted into the actual physical width. and actual physical height H r :

[0065] ;

[0066] Where s is the physical pixel size, D is the actual distance (m) from the centroid of the falling rock to the camera, f is the camera focal length (m), and the average of the physical width and height is taken as the equivalent diameter d of the falling rock.

[0067] Calculate the equivalent diameter d of the falling rock:

[0068] ;

[0069] Based on the equivalent diameter of the falling rock, the volume of the falling rock is calculated using a spherical approximation model or an ellipsoidal approximation model.

[0070] Furthermore, in step S5, predicting the landing point and impact range of the falling rocks includes:

[0071] Starting from the current position of the falling rock, the bouncing path and rolling path are calculated iteratively through multi-frame trajectory fitting to determine the predicted landing point, and the geographic coordinates and terrain attributes of the predicted landing point are marked. The terrain attributes include buffer zones, road edges, or driving lanes.

[0072] Furthermore, the method also includes:

[0073] The disaster risk level is calculated by comprehensively considering the volume of the falling rocks, their spatial velocity, and the predicted landing point. An early warning instruction corresponding to the disaster risk level is generated and pushed to the terminal execution agency.

[0074] The comprehensive calculation of disaster risk level includes:

[0075] Using the equivalent diameter corresponding to the volume of the falling rock, its spatial velocity, and the terrain attributes of the predicted impact point as three core criteria, four risk levels are classified according to the following standards:

[0076] Level IV warning: equivalent diameter less than 20cm, spatial velocity less than or equal to 2m / s, predicted landing point located outside the buffer zone;

[0077] Level 3 warning: The equivalent diameter is 20cm to 50cm, the spatial velocity is 2m / s to 5m / s, and the predicted landing point is located at the edge of the road.

[0078] Level II warning: The equivalent diameter is 50cm to 1m, the spatial velocity is 5m / s to 10m / s, and the predicted landing point is located within the driving lane;

[0079] Level 1 Warning: Equivalent diameter greater than 1m, spatial movement speed greater than 10m / s, and expected to cause severe road blockage;

[0080] Based on the determined risk level, a standardized early warning instruction is generated. The early warning instruction includes the early warning level, unique rockfall identifier ID, equivalent diameter, spatial velocity, predicted landing point, impact range, and release time, and is accompanied by real-time trajectory screenshots and bimodal image evidence.

[0081] Push notifications will be tiered according to risk level.

[0082] Secondly, the present invention provides a real-time monitoring and early warning system for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning, including a processor and a storage medium;

[0083] The storage medium is used to store instructions;

[0084] The processor is configured to operate according to the instructions to perform the steps of the method according to the first aspect.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] 1. By fusing visible light and thermal imaging, the problem of monitoring blind spots at night and in rainy or foggy weather is solved, significantly improving the detection recall rate.

[0087] 2. It innovatively introduces a physical motion constraint model and a multi-dimensional interference suppression mechanism to effectively distinguish falling rocks from interference sources such as animals and vegetation, and significantly reduce false alarms.

[0088] 3. Based on edge computing architecture, the inference process is completed locally without uploading video streams, compressing the warning delay to within seconds, and does not rely on high-bandwidth networks.

[0089] 4. Risk assessment based on physical parameters provides more instructive tiered early warning information, assisting decision-makers in taking differentiated response measures. Attached Figure Description

[0090] Figure 1 This is a flowchart of the method of the present invention.

[0091] Figure 2 This is a schematic diagram of the monitoring system of the present invention. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0093] Example 1:

[0094] This embodiment provides a real-time monitoring and early warning method for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive inference, such as... Figure 1 As shown, it includes the following steps:

[0095] Step (1): The monitoring system used to collect information, such as Figure 2 As shown, an integrated dual-light acquisition unit is deployed at the highest point of the monitored mountain slope with an unobstructed view. The unit integrates a visible light camera and a long-wave infrared thermal imaging camera. The optical axes of the two cameras are kept parallel, and the installation distance is controlled within a preset baseline range to ensure that their field of view heights overlap, completely covering the target monitoring area. A hardware external triggering method is used to synchronously control the exposure and acquisition of both cameras with the same pulse signal, maintaining a uniform acquisition frame rate of 25fps. Visible light RGB and thermal imaging grayscale / pseudo-color image sequences are stored frame by frame, retaining acquisition parameters such as timestamps, exposure, and resolution. The original sequences are transmitted to the edge computing node. The visible light frame timestamp T is extracted from the image sequence received from the edge computing node. vis [i] and the thermal imaging frame timestamp Tthr[j] are used to calculate the inter-frame time difference using the nearest neighbor time difference matching algorithm. Frame pairs with a time difference ≤ 20ms are selected to establish an initial time synchronization correspondence.

[0096] Frame drops, delayed frames, and frames with abnormal timestamps are filtered out. If there is a slight frame rate deviation between the two sequences, timing unification is achieved through resampling, resulting in a strictly synchronized visible light-thermal imaging frame pair sequence. A metal checkerboard calibration plate capable of recognizing both visible light and thermal imaging modes is selected. Ten to twenty sets of dual-modal calibration image pairs at different angles and distances are acquired within the camera's effective field of view. Feature points of each image pair are extracted and robustly matched. After removing mismatched points, the RANSAC algorithm combined with least squares optimization is used to solve for the 3×3 homography matrix H that maps thermal imaging to the visible light image. The homography matrix H (3×3) satisfies the projection transformation:

[0097] ;

[0098] Where s is the homogeneous coordinate scale factor, which is the normalization scaling factor used to restore the homogeneous coordinates after projection transformation to ordinary two-dimensional pixel coordinates; (x,y): pixel coordinates of the thermal image; (x',y'): pixel coordinates of the visible light image; H includes rotation, translation, scaling, and perspective distortion, and is saved to the local database of the edge computing node. The pre-calibration matrix H is called to perform preliminary spatial mapping on the thermal image frame through perspective transformation; ORB feature points, which are rotation- and scale-invariant and computationally inexpensive, are extracted from the pre-registered thermal image frame and the corresponding visible light frame to complete rapid feature point matching and calculate pixel coordinate deviation. If the deviation exceeds a threshold, the matrix H is dynamically corrected based on the deviation value to obtain H'; H' is then used to perform a secondary perspective transformation and bilinear pixel interpolation on the thermal image frame to complete fine registration.

[0099] Step (2): Receive the spatiotemporally aligned visible light-thermal imaging image pair output from Step 1, complete preprocessing, normalize the size and pixel values ​​of the visible light RGB three-channel image to [0,1], and perform histogram equalization to enhance texture details, adapting to complex lighting scenes on mountains; normalize the size and pixel values ​​of the thermal imaging single-channel grayscale temperature map / three-channel pseudo-color temperature map, preserving thermal radiation gradient features, adapting to low visibility scenes such as nighttime and rain / fog. Complete channel dimension splicing at the network input layer, strictly following the early fusion strategy: if the thermal imaging is a single-channel grayscale image, splice the visible light RGB three channels with the thermal imaging one channel to form a four-channel input tensor; if the thermal imaging is a three-channel pseudo-color image, splice the visible light RGB three channels with the thermal imaging RGB three channels to form a six-channel input tensor. The spliced ​​tensor serves as the unique input to the network, realizing end-to-end learning of dual-modal features.

[0100] Spatiotemporally aligned bimodal image data of complex terrain and mountain scenes were collected and divided into training / validation / test sets in an 8:1:1 ratio. Bounding boxes and category labels of falling rock targets were labeled, covering all scenes such as day / night, sunny / rainy / foggy, and sudden changes in lighting. At the same time, interference samples such as vegetation swaying, animal activity, and changes in light and shadow were mixed in to ensure the network's generalization ability.

[0101] A multimodal feature fusion detection network is constructed. Each frame of the bimodal image in the spatiotemporally aligned bimodal image sequence is input into the multimodal feature fusion detection network. Visible light features and thermal imaging features are fused at the feature level to extract deep semantic features and output initial rockfall detection results containing target location, target category and detection confidence.

[0102] The multimodal feature fusion detection network includes, in sequence, an input preprocessing module, a bimodal feature encoding module, a cross-modal feature fusion module, a multi-scale feature enhancement module, a detection prediction module, a candidate box filtering module, and a result encapsulation module.

[0103] The preprocessing module is used to receive the spatiotemporally aligned bimodal image pair of the t-th frame output in step S1. The bimodal image pair includes a visible light image Ivis(t), a thermal imaging image Ithr(t), the corresponding frame number t, and timestamp information. The visible light image Ivis(t) is subjected to size normalization, pixel value normalization, and illumination enhancement processing. The thermal imaging image Ithr(t) is subjected to size normalization, temperature value normalization, and thermal radiation gradient preservation processing to make the two have the same input resolution and retain the scaling parameters that are mapped back from the network input scale to the original image scale.

[0104] The dual-modal feature encoding module includes a visible light feature encoding branch and a thermal imaging feature encoding branch. The visible light feature encoding branch is used to extract the color, texture, edge, and contour features of the falling rock target from Ivis(t), resulting in visible light shallow features Fvis1, visible light mid-layer features Fvis2, and visible light deep features Fvis3. The thermal imaging feature encoding branch is used to extract the temperature difference, thermal radiation boundary, and thermal gradient features between the falling rock target and the mountain background from Ithr(t), resulting in thermal imaging shallow features Fthr1, thermal imaging mid-layer features Fthr2, and thermal imaging deep features Fthr3.

[0105] The cross-modal feature fusion module is used to perform feature alignment, channel stitching, and adaptive weight allocation on Fvis1 and Fthr1, Fvis2 and Fthr2, and Fvis3 and Fthr3 at the same scale, respectively, to obtain fused shallow feature Ffuse1, fused mid-level feature Ffuse2, and fused deep feature Ffuse3. Among them, the adaptive weight allocation is implemented through a lightweight channel attention unit, which increases the weight of visible light texture features in daylight or under sufficient light conditions, increases the weight of thermal imaging temperature features in night, rain, fog, or low visibility conditions, and reduces the response of feature channels corresponding to mountain background, vegetation swaying, and light and shadow changes.

[0106] The multi-scale feature enhancement module is used to perform top-down and bottom-up feature transfer on the fused shallow feature Ffuse1, fused mid-level feature Ffuse2, and fused deep feature Ffuse3, forming shallow multi-scale detection feature map P1, mid-level multi-scale detection feature map P2, and deep multi-scale detection feature map P3; among them, P1 is used to detect small-sized falling rock targets at a distance, P2 is used to detect medium-sized falling rock targets, and P3 is used to detect large-sized falling rock targets at close range;

[0107] The detection and prediction module is used to perform target classification, bounding box regression, and target confidence prediction on multi-scale detection feature maps P1, P2, and P3 respectively, and outputs the predicted class probability, target confidence, and bounding box parameters for each candidate target; the bounding box parameters include the target center point coordinates, bounding box width, and bounding box height, or include the coordinates of the upper left corner and the lower right corner.

[0108] The candidate box filtering module is used to perform confidence threshold filtering, bounding box coordinate decoding, original image scale restoration and non-maximum suppression processing on the candidate targets output by the detection and prediction module, and remove candidate boxes with too high overlap or too low confidence to obtain the initial set of falling rock targets in the t-th frame image.

[0109] The result encapsulation module is used to organize the initial set of falling rock targets in the t-th frame image into a standardized initial falling rock detection result Rt. Rt includes at least the frame number t, the number of falling rock targets Nt, and the target number, bounding box position, category label, and detection confidence of each falling rock target.

[0110] The multimodal feature fusion detection network uses a lightweight convolutional neural network adapted for edge computing as its basic backbone. This network is specifically optimized for rockfall monitoring scenarios.

[0111] The original three-channel input convolutional kernel was modified to adapt to 4 / 6-channel input convolutional kernels, ensuring compatibility with dual-modal concatenated tensors. The shallow network's feature extraction capability for small-target falling rocks was enhanced, adapting to the detection requirements of distant, small-sized debris on mountainsides. Unnecessary deep convolutional modules were pruned to control parameter and computational loads, meeting the real-time inference requirements of edge nodes. Through convolution and pooling operations in the backbone network, the input dual-modal concatenated tensor was downsampled to generate three sets of feature maps at different scales. The shallow features retain texture / temperature details, the mid-level features extract target contours, and the deep features output semantic information, corresponding to small, medium, and large-sized falling rock targets, respectively, covering the detection needs of falling rocks at different distances and volumes in complex terrain.

[0112] In each feature output layer of the backbone network, a lightweight channel attention module is embedded to adaptively weight the bimodal features: channel weights are learned for visible light-dominated texture features and thermal imaging-dominated temperature features, and the weights are automatically adjusted according to the input scene—the weight of visible light texture features is adaptively increased during daytime when there is sufficient light, and the weight of thermal imaging temperature features is adaptively increased at night / in low visibility conditions such as rain and fog. At the same time, the weight of foreground features of falling rocks is increased, while the weight of interference features such as mountain background, vegetation, and light and shadow is reduced to reduce the probability of subsequent false alarms. A combined loss function of classification loss + bounding box regression loss + confidence loss is adopted. FocalLoss is introduced to optimize the classification loss and improve the detection accuracy of small-sized debris to address the problems of small falling rocks and imbalanced positive and negative samples. After training, INT8 quantization and model pruning are used to further compress the model size, reduce the amount of computation, and adapt to the computing power limitations of edge computing nodes. Accuracy verification is carried out on the test set to ensure that the falling rock detection rate meets the requirements of field monitoring and to control false alarms with low confidence.

[0113] Edge deployment and real-time inference involve deploying the optimized lightweight model to corresponding edge computing nodes, setting the inference frame rate to match the front-end acquisition frame rate, and receiving the spatiotemporally aligned bimodal image sequence output from step 1 in real time. This sequence is then input frame-by-frame into the trained multimodal fusion detection network. After the initial rockfall detection results are output and the network inference is complete, standardized initial rockfall detection results are output frame-by-frame, including the frame number, the number of rockfall targets, the bounding box position of each target, the category label, and the detection confidence.

[0114] Step (3): Based on the initial rockfall detection results of multiple consecutive frames, a spatiotemporal trajectory tracking module is constructed to perform cross-frame association and trajectory generation for the same target, and calculate the target's motion vector, instantaneous velocity, acceleration and motion direction angle; combined with the pre-acquired mountain slope and terrain prior information, candidate targets and corresponding motion trajectories with motion characteristics that conform to the rockfall motion law are selected through the rockfall physical motion constraint model.

[0115] The input to the spatiotemporal trajectory tracking module is the initial rockfall detection results and their timestamps for multiple consecutive frames, camera calibration parameters, and terrain prior information; the output is a set of candidate targets with unique trajectory IDs. The set of candidate targets includes the trajectory sequence, bounding box position, category label, detection confidence, motion vector, instantaneous velocity, acceleration, motion direction angle, and trajectory state of each target.

[0116] Specifically, the data processing methods of the spatiotemporal trajectory tracking module include:

[0117] The centroid coordinates, bounding box, confidence level, and time sequence information of the falling rock targets in each frame are extracted. Invalid targets with a confidence level below 0.5 and false targets isolated in a single frame without matching are removed. The detection results are time-aligned with the dual-modal image sequence according to the frame number and timestamp. The slope θ of the mountain slope in the monitoring area is obtained through on-site survey, and the mapping transformation between the pixel coordinate system and the three-dimensional geographic coordinate system of the slope is completed to eliminate the influence of perspective distortion. The effective component of gravity acceleration on the slope is calculated based on the slope. :

[0118] ,

[0119] in It is the acceleration due to gravity. The slope of the monitored mountain slope was determined to establish a reasonable range for the theoretical acceleration of falling rocks, and the legal rolling area of ​​falling rocks was preset simultaneously. A spatiotemporal trajectory tracking module was constructed to complete cross-frame target association and trajectory generation. An improved lightweight ByteTrack multi-target tracking algorithm adapted to edge computing was selected to ensure that the tracking frame rate matched the front-end acquisition frame rate (25fps). Target appearance features and Kalman filter motion prediction features were extracted to achieve optimal matching between the target in the current frame and the trajectory in the previous frame. A unique fixed ID was assigned to the same target to achieve cross-frame association. A trajectory cache lifecycle of 2-3 frames was set to solve the problem of trajectory breakage caused by brief occlusion during rockfall. An initial trajectory was generated for targets that were successfully matched for 3 or more consecutive frames, and the trajectory sequence was updated in real time, eliminating invalid trajectories with insufficient matching frames or no continuous motion sequence.

[0120] The target centroid in pixel coordinates is: (u c ,v c ), converted to slope physical coordinates (X w ,Y w )formula:

[0121] ;

[0122] Where (u0, v0) are the coordinates of the principal point of the image, and k is the actual physical length corresponding to a single pixel, which is jointly calibrated by the camera's intrinsic and extrinsic parameters and the terrain elevation. Through this transformation, perspective distortion of the image is eliminated, and the true physical position of the falling rock on the slope is obtained. The difference between the physical coordinates of the centroid of the falling rock in two adjacent frames represents the magnitude and direction of the actual displacement of the falling rock in a single frame.

[0123] Let the physical coordinates of the k-th frame be: (X w,k ,Y w,k ), Physical coordinates of frame k+1: (X w , k+1 ,Y w , k+1 )

[0124] Horizontal component (along the X-axis of the slope) ;

[0125] Vertical downward component (along the Y-axis of the slope) ;

[0126] Combined motion vector =(ΔX,ΔY);

[0127] Vector magnitude (actual displacement in a single frame) ;

[0128] The frame rate is f (25fps), and the inter-frame time interval is... :

[0129] ;

[0130] Horizontal instantaneous velocity

[0131] Vertical downward instantaneous velocity

[0132] Combined instantaneous velocity

[0133] V represents the actual physical velocity of the falling rock.

[0134] The target centroid pixel coordinates are converted into actual physical coordinates of the slope. For each valid trajectory, the target single-frame motion vector (decomposed into horizontal and vertical downward components of the slope) and instantaneous velocity (decomposed into horizontal component V) are calculated sequentially. x Vertical component V y The resultant velocity (V), tangential acceleration along the roll-off direction, and the angle between the motion vector and the horizontal line of the slope provide a quantitative basis for subsequent screening.

[0135] Candidate targets are screened using a physical motion constraint model for falling rocks, strictly adhering to a three-layer verification system: The first layer, velocity component constraint, requires that Vy always be greater than a preset multiple of Vx in consecutive frames, eliminating interference targets dominated by horizontal motion; the second layer, acceleration trend constraint, requires that the resultant velocity V exhibit a non-linear acceleration trend over time, and that the acceleration be within a preset reasonable range, eliminating non-falling rock targets with uniform velocity or random walks; the third layer, terrain prior verification, eliminates invalid targets whose trajectories deviate from the legal rolling area or exhibit anti-gravity motion. The final compliant targets are the candidate falling rock targets, and their complete trajectories and kinematic parameters are output simultaneously.

[0136] Step (4): Standardize the candidate rockfall target data, extract the target's unique ID, trajectory sequence, frame-level centroid coordinates and bounding box information, and extract the visible light and thermal imaging ROI regions of the corresponding frames based on the bounding boxes to provide a standardized data source for multi-dimensional verification; at the same time, adaptively configure the verification weights based on the monitoring scene, balance the verification weights of each dimension during the day, and increase the weights of thermal infrared features and motion patterns in low visibility scenes such as night and rain / fog, which fits the core design of edge adaptive inference.

[0137] A real-time bimodal background model based on a dynamic background update algorithm is constructed. 30 frames of bimodal images without abnormal perturbations are selected. The visible light and thermal imaging background models are initialized respectively using a lightweight Gaussian mixture model to complete the static background pixel-level modeling. K=3 Gaussian distributions are established for each pixel (lightweight, adapted to edge computing) to represent the multimodal changes of the background (illumination, shadow, texture).

[0138] Constructing pixel value probability density :

[0139] ;

[0140] in This represents the current pixel value (grayscale / color value for visible light, temperature value for thermal imaging). The weight of the i-th Gaussian component is... The mean, Let K be the variance, and K be the total number of Gaussian components.

[0141] The pixel value probability density formula is used to describe the current pixel value x. t The degree of matching between the pixel and its historical background distribution. The system establishes K Gaussian components for each pixel to characterize the normal variation state of the pixel under different lighting, shadow, thermal noise, and slight background perturbation conditions. When the current pixel value x t The probability P(x) under the Gaussian mixture background model t If P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel; when P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel. t When the pixel value is below the preset background matching threshold, it indicates that the pixel value deviates significantly from the historical background distribution, and it is identified as a foreground pixel. By statistically analyzing the proportion of foreground pixels within the candidate target bounding box area, it can be determined whether the proportion of foreground pixels is within the preset range, thereby eliminating false targets with no significant foreground changes.

[0142] Calculate the initial mean μ, variance σ, and weight ω using the pixel values ​​of the first 30 frames, and sort them by weight / standard deviation ratio: the larger ω / σ is, the more likely it is to be background; set a dual-mode learning rate update strategy, the background area is updated quickly, and the foreground rocks are not swallowed into the background, adopt foreground and background separation, and assign different learning rates.

[0143] ;

[0144] in Foreground learning rate (low learning rate) The background learning rate is set to high. Background areas without candidate targets are updated in real time with a high learning rate to adapt to dynamic changes such as sudden changes in lighting and normal shaking of vegetation. Foreground areas of candidate targets are frozen and updated with a low learning rate to avoid falling rock targets being mistakenly integrated into the background. Initial screening is completed through background subtraction to remove false targets without significant foreground features.

[0145] An interference feature library was constructed, collecting full-scene dual-modal samples of typical interference sources such as vegetation swaying, running wild animals, cloud shadows, sudden changes in illumination, and cloud and fog obstruction. Three core features of each type of interference were extracted: the frequency domain distribution and optical flow entropy value of the optical flow field in the motion mode dimension, the contour stability and texture complexity in the appearance dimension, and the temperature gradient change rate and thermal radiation stability in the thermal infrared dimension. A category-specific interference feature index library was established, feature matching degree thresholds were set, and continuous iterative optimization was performed during system operation to improve matching accuracy.

[0146] Through multi-dimensional cross-validation, the system distinguishes between genuine and counterfeit targets from three dimensions: motion pattern dimension, which calculates the target's optical flow entropy value, matches the motion patterns in the interference library, and eliminates interference from irregular motion and horizontal uniform movement; appearance feature dimension, which extracts the target's contour, aspect ratio, and texture features, matches the appearance patterns in the interference library, and eliminates interference from abrupt contour changes and abnormal aspect ratios; and thermal infrared feature dimension, which calculates the target's temperature gradient change rate, matches the thermal radiation features in the interference library, and eliminates interference from abnormal temperature differences and drastic temperature fluctuations.

[0147] The results are output through a three-level fusion judgment rule: targets marked as suspected false targets in two or more dimensions are directly eliminated; targets with anomalies in a single dimension are reviewed in conjunction with physical motion constraint results; targets that meet the characteristics of falling rocks in all dimensions are confirmed as real falling rock targets. The final output includes the target ID, complete trajectory, and motion parameters.

[0148] Step (5): Conduct preliminary preparations and basic data calibration, integrate the full data of real rockfall targets, extract the bounding box pixel scale, centroid physical coordinates, kinematic parameters, and complete trajectory sequence according to the unique target ID, and simultaneously retrieve the prior information of the monitoring area's terrain and camera intrinsic and extrinsic parameters to establish a standardized data archive. Based on the camera calibration results and the terrain DEM model, complete the precise mapping calibration of the three major coordinate systems of pixels, physical, and geographic coordinates, eliminate the calculation errors caused by camera perspective distortion and mountain terrain undulations, and lay a precise data foundation for subsequent quantitative analysis.

[0149] Accurate estimation of rockfall volume and spatial velocity: For volume estimation, based on the pixel size of the rockfall target's bounding box in the image, combined with the actual geographical distance between the target's centroid and the camera's interior / exterior, the pixel scale is converted into actual physical size using a camera imaging perspective model, and the equivalent diameter of the rockfall is calculated.

[0150] , ,

[0151] Among them W p H is the pixel width of the rockfall bounding box. p The height of the bounding box of the falling rocks in pixels. These are the pixel coordinates of the right boundary of the bounding box. These are the pixel coordinates of the left boundary of the bounding box. These are the pixel coordinates of the upper boundary of the bounding box. Given the lower boundary pixel coordinates of the bounding box, the pixel size is mapped to the actual physical width W of the falling rock based on the pinhole imaging perspective model. r Physical height H r :

[0152] ;

[0153] Where s is the physical size of a pixel, D is the actual distance (m) from the centroid of the falling rock to the camera, f is the focal length of the camera (m), and the average of the physical width and height is taken as the equivalent diameter d of the falling rock.

[0154] Take the average of the physical width and height as the equivalent diameter d of the falling rock:

[0155] ;

[0156] Based on the rigidity of the rock, a spherical / ellipsoidal approximation model is used, and the volume of the falling rock is calculated by combining the equivalent diameter:

[0157] Calculate the volume of the spherical approximation model:

[0158] ;

[0159] Calculate the volume of the ellipsoidal approximation model:

[0160] ;

[0161] , , ;

[0162] in Let a be the volume of the rock model, and a, b, c be the semi-lengths of the three axes.

[0163] When estimating the velocity of spatial motion, the resultant velocity V from step 3 can be extracted.

[0164] For predicting the impact point and impact range of falling rocks, point prediction is based on the current trajectory, actual spatial velocity, and direction angle of the falling rock, combined with topographic information such as mountain slope, elevation changes, and the direction of the roll path, to construct a dynamic model of rockfall slope motion. Starting from the current position of the target, the bounce and roll paths are calculated iteratively through multi-frame trajectory fitting to determine the accurate predicted impact point and mark its geographical coordinates and topographic attributes (buffer zone, road edge, driving lane, etc.). The impact range is delineated based on the volume and spatial velocity of the falling rock, calculating the impact energy according to the principle of energy conservation, and combining the width of roads, slope gradient, and facility distribution around the impact point to delineate the direct impact range (bounce and roll coverage area) and indirect impact range (splashed debris and potential secondary collapse area) after the rockfall impact.

[0165] Specifically, to predict the landing point and impact range of falling rocks, the following methods can be used:

[0166] Based on the current real-time trajectory, spatial velocity, and direction angle of the falling rocks, and combined with topographic information such as slope, elevation change, and roll-off path, a dynamic model of rockfall slope motion is constructed.

[0167] Starting from the current position of the falling rock, the bouncing path and rolling path are calculated iteratively through multi-frame trajectory fitting to determine the predicted landing point, and the geographic coordinates and terrain attributes of the predicted landing point are marked. The terrain attributes include buffer zones, road edges, or driving lanes.

[0168] Based on the volume and spatial velocity of the falling rocks, the impact energy is calculated according to the principle of energy conservation. Combining the road width, slope gradient, and facility distribution around the impact point, the direct and indirect impact ranges after the impact are delineated. The direct impact range includes the bounce-covered area and the roll-covered area, while the indirect impact range includes the area of ​​flying debris and the potential area for secondary collapse.

[0169] The dynamics model of rockfall slope motion includes a motion state input module, a coordinate unified transformation module, a terrain constraint analysis module, a trajectory fitting and prediction module, a bounce-roll state discrimination module, and a landing point and influence range output module.

[0170] The motion state input module is used to receive the target's unique ID, current frame number, timestamp, historical trajectory point sequence, current centroid coordinates, spatial motion velocity, acceleration, motion direction angle, equivalent diameter, and volume of the confirmed real falling rock target.

[0171] The coordinate unification and transformation module is used to convert the image pixel coordinates of the rockfall target into the physical coordinates of the slope, and combine them with the camera intrinsic and extrinsic parameters and the prior terrain information to map them into geographic coordinates; the terrain constraint parsing module is used to read the slope, elevation change, roll-off channel direction, roll-off area, road boundary, buffer zone and lane position of the monitoring area, and determine the slope direction constraint and spatial boundary constraint of the rockfall movement.

[0172] The trajectory fitting and prediction module is used to fit the trajectory based on historical trajectory points in multiple consecutive frames, and to determine the current main direction of the falling rock's movement, the trend of its speed change, and the predicted direction for the next moment.

[0173] The bouncing-rolling state discrimination module is used to determine whether the falling rock is in a rolling state, bouncing state, or stopped state based on local elevation changes, current spatial movement speed, slope gradient changes, and preset collision recovery coefficients, and updates the predicted trajectory using either the rolling path iteration method or the bouncing path iteration method respectively.

[0174] The landing point and impact range output module is used to terminate the calculation when the predicted rockfall speed is lower than the preset stop threshold, the predicted position enters the road boundary area, or the predicted trajectory reaches the maximum number of iterations. It outputs the predicted trajectory point sequence, the geographical coordinates of the predicted landing point, and the terrain attributes of the landing point. The terrain attributes of the landing point include the buffer zone, the road edge, and the driving lane.

[0175] The disaster risk level is comprehensively calculated based on three core indicators: rockfall volume (equivalent diameter), spatial movement speed, and predicted landing point. Level IV warning is for gravel with a diameter <20cm, speed ≤2m / s, and landing point outside the buffer zone; Level III warning is for rocks with a diameter of 20cm-50cm, speed 2-5m / s, and landing point at the road edge; Level II warning is for rocks with a diameter of 50cm-1m, speed 5-10m / s, and landing point within the driving lane; Level I warning is for large landslides with a diameter >1m, speed >10m / s, which are likely to cause serious blockage.

[0176] Standardized early warning instructions are generated based on risk levels, including warning level, rockfall ID, equivalent diameter, speed, predicted impact point, impact range, and release time. Real-time trajectory screenshots and bimodal image evidence are attached to ensure complete and traceable information. Based on edge computing node communication modules, warnings are pushed in tiers according to risk level: Level 4 and Level 3 warnings are pushed to the regional monitoring station backend; Level 2 and Level 1 warnings are pushed synchronously to road control terminals, traffic command centers, on-site duty points, and mobile terminals of inspection personnel, adapting to the complex communication environment of mountainous areas. Simultaneously, an instruction execution feedback mechanism is established to receive real-time reception status and handling feedback from various implementing agencies, forming a closed-loop management system of "early warning-push-execution-feedback" to improve emergency response efficiency.

[0177] Example 2:

[0178] This embodiment provides a real-time monitoring and early warning system for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive inference, including a processor and storage medium;

[0179] The storage medium is used to store instructions;

[0180] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0185] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A real-time monitoring and early warning method for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning, characterized in that, Includes the following steps: Step S1: Acquire the visible light image sequence and thermal imaging image sequence of the monitoring area, and perform time synchronization and spatial geometric registration on the visible light image sequence and the thermal imaging image sequence to obtain a spatiotemporally aligned dual-modal image sequence; Step S2: Construct a multimodal feature fusion detection network. Input each frame of the spatiotemporally aligned bimodal image sequence into the multimodal feature fusion detection network frame by frame. Perform feature-level fusion of visible light features and thermal imaging features, extract deep semantic features, and output the initial rockfall detection result containing the target location, target category, and detection confidence. Step S3: Based on the initial rockfall detection results of multiple consecutive frames, and based on the spatiotemporal trajectory tracking module, perform cross-frame association and trajectory generation for the same target, calculate the target's motion vector, instantaneous velocity, acceleration, and motion direction angle; combine with the pre-acquired mountain slope and terrain prior information, and use the rockfall physical motion constraint model to screen out candidate targets and their corresponding motion trajectories whose motion characteristics conform to the rockfall motion law. Step S4: Construct a dual-modal real-time background model, remove false targets, and combine it with a pre-established interference feature library to perform secondary verification on the candidate targets to obtain confirmed real falling rock targets; Step S5: Based on the pixel scale of the confirmed real falling rock target in the image, camera intrinsic and extrinsic parameters, and prior terrain information, estimate the volume and spatial velocity of the falling rock, and combine the terrain slope and trajectory to predict the landing point and impact range of the falling rock. The input to the spatiotemporal trajectory tracking module consists of multiple consecutive frames of initial rockfall detection results and their timestamps, camera calibration parameters, and terrain prior information; The output is a set of candidate targets with unique trajectory IDs. The set of candidate targets includes the trajectory sequence, bounding box position, class label, detection confidence, motion vector, instantaneous velocity, acceleration, motion direction angle and trajectory state of each target. The spatiotemporal trajectory tracking module is used to perform the following steps: Invalid targets with a confidence level below 0.5 and false targets with no matching in a single frame are removed from the initial rockfall detection results; An improved lightweight ByteTrack multi-target tracking algorithm is adopted, which combines target appearance features with Kalman filter motion prediction features to achieve optimal matching between the target in the current frame and the trajectory in the previous frame, assigning a unique identifier ID to the same target and realizing cross-frame association. Set the trajectory buffer lifecycle to 2 to 3 frames; For targets that are matched successfully for three or more consecutive frames, an initial trajectory is generated and the trajectory sequence is updated in real time, and trajectories with insufficient matching frames or no non-continuous motion sequence are removed. Based on the initial rockfall detection results across multiple consecutive frames, and using the spatiotemporal trajectory tracking module, cross-frame correlation and trajectory generation are performed on the same target. The target's motion vector, instantaneous velocity, acceleration, and motion direction angle are calculated. Combining pre-acquired prior information on mountain slope and terrain, a rockfall physical motion constraint model is used to screen candidate targets whose motion characteristics conform to the rockfall motion law and their corresponding motion trajectories, including: Obtain the slope of the mountain slope in the monitoring area and complete the mapping transformation between the pixel coordinate system and the three-dimensional geographic coordinate system of the slope. The target centroid pixel coordinates are converted into actual physical coordinates of the slope. For each valid trajectory, the target single-frame motion vector, instantaneous velocity, tangential acceleration along the rolling direction of the slope, and motion direction angle between the motion vector and the horizontal line of the slope are calculated in sequence. The target single-frame motion vector is decomposed into horizontal and vertical downward components of the slope; the instantaneous velocity is decomposed into a horizontal component V. x Vertical component V y Resultant velocity V; Set up three-layer verification: The first layer, velocity component constraint: requires the velocity component in the vertically downward direction to be constrained across multiple consecutive frames. Always greater than the horizontal component of velocity The preset multiplier is used to eliminate interfering targets whose horizontal movement is dominant. The second layer is acceleration trend constraint: it requires that the resultant velocity V has a non-linear acceleration trend over time, and the acceleration value is within a reasonable range, eliminating non-falling rock targets that are moving at a constant speed or wandering randomly. The third layer is terrain prior verification: invalid targets whose movement trajectories deviate from the preset legal rolling area or have anti-gravity movement are eliminated; The targets that pass the above three-layer verification and their corresponding motion trajectories are used as candidate falling rock targets and candidate motion trajectories.

2. The method for real-time monitoring and early warning of rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning as described in claim 1, characterized in that, In step S1, the visible light image sequence and the thermal imaging image sequence are synchronized in time and registered spatially, including: The exposure and acquisition of the visible light camera and the thermal imaging camera are controlled synchronously to ensure that the acquisition frame rates of the two cameras are the same. Extract the timestamps of visible light frames and thermal imaging frames, use the nearest neighbor time difference matching algorithm to calculate the time difference between frames, filter out frame pairs with a time difference less than or equal to a preset threshold, and establish an initial time synchronization correspondence. Frames with missing frames, delayed frames, or abnormal timestamps are filtered out. If there is a frame rate discrepancy between the two sequences, the timing is unified by resampling, and a time-synchronized visible light-thermal imaging frame pair sequence is output. The homography matrix H between the visible light camera and the thermal imaging camera was obtained using the calibration plate method. The homography matrix H is used to perform perspective transformation on the thermal imaging frame to achieve preliminary spatial mapping, resulting in a pre-registered thermal imaging frame. ORB feature points are extracted from the pre-registered thermal imaging frame and the corresponding visible light frame, feature point matching is performed, and pixel coordinate deviation is calculated. If the pixel coordinate deviation exceeds a preset threshold, the homography matrix H is dynamically corrected based on the deviation value to obtain the corrected homography matrix H′. The corrected homography matrix H′ is then used to perform secondary perspective transformation and bilinear pixel interpolation on the thermal imaging frame to complete spatial registration, so that the pixel coordinate deviation of the same physical target in the visible light image and the thermal imaging image is less than the preset threshold.

3. The method for real-time monitoring and early warning of rockfall in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning according to claim 2, characterized in that, The multimodal feature fusion detection network includes a preprocessing module, a bimodal feature encoding module, a cross-modal feature fusion module, a multi-scale feature enhancement module, a detection prediction module, a candidate box filtering module, and a result encapsulation module connected in sequence. The preprocessing module is used to: receive spatiotemporally aligned bimodal image pairs from each frame of a spatiotemporally aligned bimodal image sequence, wherein the bimodal image pair includes a visible light image Ivis(t), a thermal imaging image Ithr(t), a corresponding frame number t, and timestamp information; perform size normalization, pixel value normalization, and illumination enhancement processing on the visible light image Ivis(t), and perform size normalization, temperature value normalization, and thermal radiation gradient preservation processing on the thermal imaging image Ithr(t) to ensure that both have the same input resolution and retain the scaling parameters mapped back from the network input scale to the original image scale; if the thermal imaging is a single-channel grayscale image, the RGB three channels of the visible light image are concatenated with one channel of the thermal imaging image to form a four-channel input tensor; if the thermal imaging is a three-channel pseudo-color image, the RGB three channels of the visible light image are concatenated with the RGB three channels of the thermal imaging image to form a six-channel input tensor; The dual-modal feature encoding module includes a visible light feature encoding branch and a thermal imaging feature encoding branch. The visible light feature encoding branch is used to extract the color, texture, edge, and contour features of the falling rock target from Ivis(t), resulting in visible light shallow features Fvis1, visible light mid-layer features Fvis2, and visible light deep features Fvis3. The thermal imaging feature encoding branch is used to extract the temperature difference, thermal radiation boundary, and thermal gradient features between the falling rock target and the mountain background from Ithr(t), resulting in thermal imaging shallow features Fthr1, thermal imaging mid-layer features Fthr2, and thermal imaging deep features Fthr3. The cross-modal feature fusion module is used to perform feature alignment, channel concatenation, and adaptive weight allocation on Fvis1 and Fthr1, Fvis2 and Fthr2, and Fvis3 and Fthr3 at the same scale, respectively, to obtain fused shallow features Ffuse1, fused mid-level features Ffuse2, and fused deep features Ffuse3; wherein, the adaptive weight allocation is implemented through a lightweight channel attention unit; The multi-scale feature enhancement module is used to perform top-down and bottom-up feature transfer on the fused shallow feature Ffuse1, fused mid-level feature Ffuse2, and fused deep feature Ffuse3, forming shallow multi-scale detection feature map P1, mid-level multi-scale detection feature map P2, and deep multi-scale detection feature map P3; among them, P1 is used to detect small-sized falling rock targets at a distance, P2 is used to detect medium-sized falling rock targets, and P3 is used to detect large-sized falling rock targets at close range; The detection and prediction module is used to perform target classification, bounding box regression, and target confidence prediction on the shallow multi-scale detection feature map P1, the medium multi-scale detection feature map P2, and the deep multi-scale detection feature map P3, respectively, and outputs the predicted class probability, target confidence, and bounding box parameters for each candidate target; the bounding box parameters include the target center point coordinates, bounding box width, and bounding box height; The candidate box filtering module is used to perform confidence threshold filtering, bounding box coordinate decoding, original image scale restoration and non-maximum suppression processing on the candidate targets output by the detection and prediction module, and remove candidate boxes with too high overlap or too low confidence to obtain the initial set of falling rock targets in each frame image. The result encapsulation module is used to organize the initial set of falling rock targets in each frame image into an initial falling rock detection result Rt. The initial falling rock detection result Rt includes at least the frame number t, the number of falling rock targets Nt, and the target number, bounding box position, category label and detection confidence of each falling rock target.

4. The method for real-time monitoring and early warning of rockfall in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning according to claim 3, characterized in that, In step S4, a bimodal real-time background model is constructed to remove false targets, including: We selected consecutive frames of bimodal images without abnormal disturbances, and used a lightweight Gaussian mixture model to initialize the visible light background model and the thermal imaging background model respectively. We established three Gaussian distributions for each pixel to represent the multimodal changes of the background. Constructing pixel value probability density : ； in The current pixel value. The weight of the i-th Gaussian component is... The mean, Let K be the variance, and K be the total number of Gaussian components. When the current pixel value x t Pixel probability density P(x) under Gaussian mixture background model t If P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel; when P(x) is higher than the preset background matching threshold, the pixel is determined to be a background pixel. t If the pixel value is below the preset background matching threshold, it is identified as a foreground pixel. By statistically analyzing the proportion of foreground pixels within the bounding box region of candidate targets, it is determined whether the proportion of foreground pixels is within a preset range, thereby eliminating false targets.

5. The method for real-time monitoring and early warning of rockfall in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning according to claim 4, characterized in that, In step S4, the candidate targets are further verified using a pre-established interference feature library to obtain confirmed real rockfall targets, including: Pre-collect dual-modal samples of typical interference sources such as vegetation swaying, running animals, cloud shadows, sudden changes in lighting, and cloud and fog obstruction in the whole scene; For each type of interference source, the following three types of features are extracted: Motion pattern characteristics: frequency domain distribution of optical flow field and optical flow entropy value; Appearance features: contour stability, texture complexity, and aspect ratio; Thermal infrared characteristics: rate of change of temperature gradient and thermal radiation stability; Establish a categorized interference feature index library and set feature matching thresholds for each type of feature; During the verification phase, the optical flow entropy value, contour stability, and temperature gradient change rate of the candidate target are calculated and matched with the corresponding features in the interference feature library, respectively. A three-level fusion judgment rule is adopted: If a candidate target is marked as a matching interference feature library in two or more dimensions of motion pattern, appearance features, and thermal infrared features, it is directly determined as a false target and removed. If only one dimension shows an abnormal match, then a composite verification is performed using the aforementioned physical motion constraint model of falling rocks. If all three dimensions match the characteristics of falling rocks, then the target of falling rocks is confirmed.

6. The method for real-time monitoring and early warning of rockfall in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning according to claim 5, characterized in that, In step S5, predicting the landing point and impact range of the falling rocks includes: Starting from the current position of the falling rock, the bouncing path and rolling path are calculated iteratively through multi-frame trajectory fitting to determine the predicted landing point, and the geographic coordinates and terrain attributes of the predicted landing point are marked. The terrain attributes include buffer zones, road edges, or driving lanes.

7. The method for real-time monitoring and early warning of rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning as described in claim 1, characterized in that, The method further includes: The disaster risk level is calculated by comprehensively considering the volume of the falling rocks, their spatial velocity, and the predicted landing point. An early warning instruction corresponding to the disaster risk level is generated and pushed to the terminal execution agency. The comprehensive calculation of disaster risk level includes: Using the equivalent diameter corresponding to the volume of the falling rock, its spatial velocity, and the terrain attributes of the predicted impact point as three core criteria, four risk levels are classified according to the following standards: Level IV warning: equivalent diameter less than 20cm, spatial velocity less than or equal to 2m / s, predicted landing point located outside the buffer zone; Level 3 warning: The equivalent diameter is 20cm to 50cm, the spatial velocity is 2m / s to 5m / s, and the predicted landing point is located at the edge of the road. Level II warning: The equivalent diameter is 50cm to 1m, the spatial velocity is 5m / s to 10m / s, and the predicted landing point is located within the driving lane; Level 1 Warning: Equivalent diameter greater than 1m, spatial movement speed greater than 10m / s, expected to cause severe road blockage; Based on the determined risk level, a standardized early warning instruction is generated. The early warning instruction includes the early warning level, unique rockfall identifier ID, equivalent diameter, spatial velocity, predicted landing point, impact range, and release time, and is accompanied by real-time trajectory screenshots and bimodal image evidence. Push notifications will be tiered according to risk level.

8. A real-time monitoring and early warning system for rockfalls in complex terrain based on multimodal spatiotemporal fusion and edge adaptive reasoning, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-7.

Images