Slope settlement positioning method and system based on radar combined with video

Abstract

The application discloses a slope subsidence positioning method and system based on radar combined with video, relates to the technical field of geological disaster early warning, and comprises the following steps: collecting and processing point cloud data and images based on a slope radar and a camera, generating simulated SAR images based on the point cloud data, and applying a Retinex algorithm to generate enhanced video images; extracting time sequence features of the point cloud data, obtaining radar features through VAE coding based on the time sequence features, extracting multi-scale features in the enhanced video images, combining to create three-dimensional frustum features to generate a spatial feature map, generating video features based on the spatial feature map, and combining the spatial feature map and the radar features to generate fusion features; generating a two-dimensional Gaussian heat map based on the point cloud data, generating a predicted heat map using the fusion features, and combining the two-dimensional Gaussian heat map to generate an ROI feature map. Through multi-modal feature fusion and adaptive optimization, the application effectively improves the precision, robustness and real-time early warning capability of slope subsidence monitoring.

Description

Technical Field

[0001] This invention relates to the field of geological disaster early warning technology, and in particular to a method and system for locating slope settlement based on radar combined with video. Background Technology

[0002] With the continuous expansion of infrastructure construction and the frequent occurrence of extreme weather events, geological disasters, especially slope instability and settlement, have received increasing attention. Technologies such as radar remote sensing, photogrammetry, and 3D point cloud modeling are gradually maturing in slope monitoring. Synthetic Aperture Radar (SAR) has all-weather, all-day operation capabilities and can detect surface deformation at the millimeter level. Simultaneously, the development of video monitoring and optical sensors provides visualization tools for acquiring dynamic features such as slope surface crack evolution and local slippage. Current research typically employs single-modal methods; however, single sensors are limited by observation angle, environmental interference, or data resolution, often resulting in problems such as local distortion of monitoring results, incomplete features, or insufficient spatiotemporal registration accuracy. Therefore, academia and engineering are increasingly focusing on multi-source information fusion, using joint processing of point clouds, images, and deep learning models to improve monitoring robustness and early warning accuracy. However, existing technologies still have several shortcomings. Traditional SAR imagery is susceptible to geometric distortion and reduced coherence under complex terrain conditions, resulting in insufficient point cloud generation accuracy and difficulty in supporting refined settlement localization. Secondly, video monitoring performance degrades significantly at night or in low visibility conditions, and image enhancement and depth estimation suffer from noise amplification and boundary blurring. Furthermore, most multi-source data fusion methods remain at the stage of simple feature stitching or statistical filtering, lacking cross-modal depth feature alignment and spatiotemporal consistency modeling, which prevents the full utilization of the complementarity between radar data and video data. In terms of real-time performance, the computational cost of processing large-scale point cloud and image data is too high, making it difficult to meet the high-frequency monitoring requirements in slope disaster early warning. Existing methods are insufficient in the differentiated processing of high-risk areas and cannot achieve refined point cloud optimization and accurate settlement calculation in key areas such as cracks and slip surfaces, thus affecting the reliability of early warning signals. Summary of the Invention

[0003] In view of the aforementioned existing problems, the present invention is proposed.

[0004] Therefore, this invention provides a method and system for locating slope settlement based on radar combined with video, which solves the problems of insufficient multimodal information fusion, poor robustness in complex environments, limited real-time processing capabilities, and inaccurate identification of high-risk areas in existing slope settlement monitoring technologies.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] In a first aspect, the present invention provides a slope settlement location method based on radar combined with video, which includes: acquiring and processing point cloud data and images based on slope radar and camera, generating simulated SAR images based on point cloud data, and applying the Retinex algorithm to generate enhanced video images.

[0007] Temporal features are extracted from point cloud data, radar features are obtained through VAE encoding based on temporal features, multi-scale features are extracted from enhanced video images, spatial feature maps are generated by creating 3D view frustum features, video features are generated based on spatial feature maps, and fused features are generated by combining spatial feature maps and radar features.

[0008] A two-dimensional Gaussian heatmap is generated based on point cloud data, the simulated SAR image is optimized, a predicted heatmap is generated using fusion features, and a ROI feature map is generated by combining the two-dimensional Gaussian heatmap with the optimized simulated SAR image.

[0009] 3D point cloud data is generated from raw point cloud data, the 3D point cloud data is optimized by ROI feature maps, and a digital surface model grid is generated based on the optimized 3D point cloud data.

[0010] The settlement amount and digital surface model grid are updated based on the optimized point cloud data. The settlement amount is used to generate an early warning signal, and an early warning report is generated based on the early warning signal and the digital surface model grid.

[0011] As a preferred embodiment of the slope settlement location method based on radar combined with video described in this invention, the step of acquiring and processing point cloud data and images based on slope radar and camera, generating simulated SAR images based on point cloud data, and generating enhanced video images using the Retinex algorithm refers to using slope radar based on DDS and digital de-modulation technology to acquire heterogeneous SAR point cloud data. Deploy high-resolution binocular cameras to acquire RGB images ;

[0012] Adaptive resolution is used to convert radar coordinates into geographic coordinates to obtain projected coordinates, forming the initial image frame. Calculate the corresponding incident angle η and scattering coefficient. Generate grayscale values ;

[0013] For the initial image frame For empty pixel regions, use neighboring points. Values ​​are interpolated to generate a complete simulated SAR image. A video image sequence is composed of consecutive RGB image frames. For each frame of the video image... The Retinex algorithm is applied to enhance the video image, resulting in an enhanced video image. .

[0014] As a preferred embodiment of the slope settlement location method based on radar combined with video described in this invention, the following steps are taken: extracting temporal features from point cloud data; obtaining radar features through VAE encoding based on the temporal features; extracting multi-scale features from enhanced video images; generating a spatial feature map by combining and creating three-dimensional view frustum features; generating video features based on the spatial feature map; and generating a fusion feature index by combining the spatial feature map and radar features. VoxelNet is then used to process the point cloud data. Voxelization, extracting spatial features ConvLSTM is used to process multi-frame point clouds, and the data is then stitched together to obtain temporal features. VAE encoding is used to extract deep feature vectors and generate radar features. Extract using Swing Transformer and FPN Multiscale features ;

[0015] The input image is processed using a pre-trained depth estimation model to generate a depth map corresponding to the image. , depth map Multiscale features Perform fusion to construct 3D view frustum features , to 3D view cone features Through the inverse of the camera intrinsic parameter matrix Transform to the world coordinate system and then project onto the BEV space to obtain a structured spatial feature map. VAE encoding is used to generate video features. ,Will and After alignment, the parts are stitched together, and VAE features are incorporated. and A point cloud-image fusion method is used to generate fused features. .

[0016] As a preferred embodiment of the slope settlement location method based on radar combined with video described in this invention, the steps of generating a two-dimensional Gaussian heat map based on point cloud data, optimizing the simulated SAR image, generating a predicted heat map using fusion features, and generating a ROI feature map by combining the two-dimensional Gaussian heat map and the optimized simulated SAR image refer to the collection of SAR point cloud data. Generate a two-dimensional Gaussian heatmap Based on simulated SAR images, a lightweight random forest model is used to obtain the interpolation weights for each neighboring valid pixel at the missing location. The grayscale values ​​are filled using a weighted average to generate a complete simulated SAR image. The optimized image Compared with the original simulated SAR image Filtration is performed directly at pixel location to generate optimized simulated SAR images. Based on fusion features A predicted heatmap (HPF) is generated. The optimized simulated SAR image is then multiplied point-by-point with the predicted heatmap HPF to obtain an enhanced predicted heatmap. This enhanced predicted heatmap is then compared with a two-dimensional Gaussian heatmap. Multiply point by point again, and the convolution generates the ROI feature map. .

[0017] As a preferred embodiment of the slope settlement location method based on radar combined with video described in this invention, the method involves: generating 3D point cloud data from raw point cloud data; optimizing the 3D point cloud data using ROI feature maps; generating a digital surface model grid based on the optimized 3D point cloud data; and using the Range-Doppler model and the Gauss-Newton iteration method to process the collected radar point cloud data. Perform 3D coordinate calculation to obtain preliminary 3D point cloud data. Combining ROI feature maps 3D point cloud data Regional weighting is performed to retain point cloud information in high-risk areas. The same point location is matched across consecutive time frames and represented using fused features. Calculate settlement Based on optimized point cloud data The inverse distance weighted interpolation method is used to generate the digital surface model grid.

[0018] As a preferred embodiment of the slope settlement location method based on radar combined with video as described in this invention, the following steps are taken: Based on optimized point cloud data, the settlement amount and digital surface model grid are updated; the settlement amount is used to generate an early warning signal; VAE and BEV are converted to a format suitable for embedded devices and deployed to the radar's embedded system; pruning techniques are applied to the VAE and BEV; and the INT8 quantization method is used to reduce the model calculation accuracy requirements. The radar system continuously receives new frames of point cloud data from the radar and camera. Using image data, VAE and BEV models are called for feature extraction to generate a fused BEV feature map. Based on the fused BEV feature map, the settlement amount and digital surface model grid at the current moment are dynamically updated. If the settlement amount... If the value exceeds the preset threshold J, it is determined to be an abnormal state, triggering the early warning mechanism and generating an early warning signal containing timestamp and location information.

[0019] As a preferred embodiment of the slope settlement location method based on radar combined with video described in this invention, wherein: the generation of the early warning report based on the early warning signal and the digital surface model grid refers to the generation of the early warning report based on the early warning signal and the real-time settlement amount. The digital surface model grid automatically generates early warning reports with visual illustrations, and pushes the early warning information to the monitoring center's large screen, management platform, and on-site personnel's mobile terminals in real time. At the same time, it links with the emergency system and activates the preset response plan.

[0020] Secondly, the present invention provides a slope settlement positioning system based on radar combined with video, comprising,

[0021] The data acquisition and preprocessing module is used to collect point clouds and images from radar and cameras, and combine them with environmental sensor information to generate simulated SAR images and enhanced video images.

[0022] The feature extraction and cross-modal fusion module is used to extract temporal features of point clouds and multi-scale features of videos using deep learning, and achieves cross-modal fusion through 3D view frustum and VAE;

[0023] The heatmap generation and ROI detection module is used to generate a two-dimensional Gaussian heatmap and fuse it with the predicted heatmap to extract ROI feature maps of high-risk areas;

[0024] The 3D point cloud optimization and settlement calculation module is used to optimize point clouds using Range-Doppler modeling and ROI weighted optimization, align multi-time data to calculate settlement and generate DSM grids.

[0025] The real-time inference and early warning signal module is used to output updated settlement and DSM grid in real time. When the settlement exceeds the threshold, it automatically generates an early warning signal containing timestamp and location information.

[0026] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the slope settlement positioning method based on radar combined with video as described in the first aspect of the present invention.

[0027] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the slope settlement location method based on radar combined with video as described in the first aspect of the present invention.

[0028] The beneficial effects of this invention are as follows: By combining simulated SAR images with video images enhanced by the Retinex algorithm, robustness under low light, fog, and vegetation obstruction conditions is effectively improved. VoxelNet and ConvLSTM are used to extract temporal point cloud features, and VAE encoding and Swing Transformer multi-scale image features are combined to construct a three-dimensional view frustum and project it onto the BEV space, significantly enhancing cross-modal feature alignment and spatial understanding capabilities. By constructing a coupling mechanism between Gaussian heatmaps and predicted heatmaps, ROI feature maps are generated, enabling focused enhancement and accurate identification of high-risk areas such as cracks and slip surfaces. This compensates for the shortcomings of traditional methods in identifying and monitoring key areas with insufficient precision, enabling robust slope settlement monitoring in complex environments and improving cross-modal registration accuracy and real-time early warning capabilities. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the slope settlement location method based on radar combined with video in Example 1.

[0031] Figure 2 This is a schematic diagram of the slope settlement positioning system based on radar combined with video in Example 1.

[0032] Figure 3 This is a flowchart of the data fusion and early warning generation process in Example 1. Detailed Implementation

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0036] Example 1, referring to Figures 1 to 3 This is the first embodiment of the present invention, which provides a slope settlement location method based on radar combined with video, including the following steps:

[0037] S1. Based on the slope radar and camera, collect and process point cloud data and images, generate simulated SAR images based on the point cloud data, and apply the Retinex algorithm to generate enhanced video images;

[0038] Temporal features are extracted from point cloud data, radar features are obtained through VAE encoding based on temporal features, multi-scale features are extracted from enhanced video images, spatial feature maps are generated by creating 3D view frustum features, video features are generated based on spatial feature maps, and fused features are generated by combining spatial feature maps and radar features.

[0039] Specifically, the system uses slope radar and cameras to collect and process point cloud data and images, generates simulated SAR images from the point cloud data, and applies the Retinex algorithm to generate enhanced video images. This involves using a slope radar (small and lightweight, single-person installation) based on DDS and digital frequency demodulation technology to collect off-track SAR point cloud data. This represents point cloud data in the radar coordinate system. These represent the positions of the X, Y, and Z axes in the radar coordinate system, respectively, and r represents a point in the radar coordinate system. Signal strength (similar to radar cross-section);

[0040] Deploy high-resolution binocular cameras (resolution ≥ 1080p, equipped with ToF sensor) to acquire RGB images. ToF sensor collects depth information Calibrate the camera intrinsic parameter inverse matrix using a binocular camera. ;

[0041] Using an integrated environmental sensor module on a slope radar, environmental variables including light intensity, fog concentration, vegetation shading (estimated through image segmentation, range [0,1]), wind speed, air pressure, rainfall, temperature, and humidity are collected and input into a fully connected network to generate environmental regulation factors. ;

[0042] The radar coordinates are obtained using an adaptive resolution Rational Function Model (RFM). Convert to geographic coordinates , where n represents a point in the transformed geographic coordinate system. The projection resolution is dynamically adjusted based on the point cloud density to obtain the projected coordinates. (representing the pixel position of the point cloud on the SAR image plane), forming the initial image frame. (Size and actual SAR image) (Consistent), the formula is:

[0043] ,

[0044] in, , , , It is a higher-order polynomial function (third order), determined by the RPC coefficients;

[0045] Based on the same set of point cloud points ( ), calculate the corresponding incident angle η and scattering coefficient. Generate grayscale values ;

[0046] The formula for calculating the incident angle is:

[0047] ,

[0048] in, It is an inverse cosine function. The slope surface normal vector is obtained through gradient estimation based on radar point cloud data. The vector from the radar to the target point is obtained through geometric vector calculation methods based on radar point cloud coordinates and orbital parameters.

[0049] The scattering coefficient The calculation formula is:

[0050] ,

[0051] in, These are empirical constants, obtained through experimental calibration based on the Muhleman model;

[0052] grayscale value The calculation formula is:

[0053] ,

[0054] Where K is a normalization constant, set to 100 (calibrated according to radar signal strength).

[0055] For the initial image frame For empty pixel regions, use neighboring points. Values ​​are interpolated to generate a complete simulated SAR image. ;

[0056] A video image sequence consists of consecutive RGB image frames. For each frame of the video image... The Retinex algorithm is applied to enhance the video image, resulting in an enhanced video image. The formula is:

[0057] ,

[0058] in, This indicates the position in the image coordinate system. The light intensity distribution at each pixel is generated by interpolation from a radar photosensor. It is a Gaussian kernel.

[0059] Traditional radar monitoring equipment is typically bulky and complex to install, and its deployment and operating conditions in complex environments are subject to strict requirements, limiting its widespread application in mountainous areas and construction sites. Meanwhile, video surveillance suffers from limited image quality in low light or poor visibility conditions, making it difficult to establish a stable and reliable data source. This invention utilizes a small, lightweight slope radar that can be installed by a single person, combined with a high-resolution binocular camera and a ToF sensor, overcoming the limitations on deployment and usage permissions of monitoring equipment. Furthermore, by collecting environmental variables such as light, fog, and vegetation through an environmental sensor module and performing adaptive correction, the applicability and robustness of the data under complex conditions are effectively improved, thus providing a more universal and operable technical approach for slope settlement monitoring.

[0060] Furthermore, temporal features of the point cloud data are extracted, and radar features are obtained through VAE encoding based on these temporal features. Multi-scale features are extracted from the enhanced video images, and spatial feature maps are generated by creating 3D view frustum features. Video features are generated based on the spatial feature maps, and fusion feature indexes are generated by combining the spatial feature maps and radar features. VoxelNet is then used to process the point cloud data. Voxelization, extracting spatial features Using ConvLSTM to process multi-frame point clouds (cumulative) =5 frames), to alleviate timing misalignment, and spliced ​​together to obtain timing features. The formula is:

[0061] ,

[0062] in, TE is an abbreviation for Temporal Encoding. (Feature stitching operation) Fuse features from multiple SAR point cloud frames. This represents the current time t minus the cumulative number of frames δ (e.g., 5 frames), which means considering the data from the current frame to the previous 5 frames;

[0063] VAE encoding is used to extract deep feature vectors and generate radar features. The formula is:

[0064] ,

[0065]

[0066] in, This is the encoder network in the VAE model, used to map the input SAR data to the mean. and variance , From normal distribution Random variables sampled from the middle;

[0067] Extract using Swing Transformer and FPN Multiscale features The formula is:

[0068]

[0069] in, (e.g., Swin Transformer) extracts image features, while Neck (e.g., FPN) fuses multi-scale features;

[0070] The input image is processed using a pre-trained depth estimation model (such as Monodepth2 or DPT) to generate a depth map corresponding to the image. , depth map Multiscale features Perform fusion to construct 3D view frustum features The specific operation involves upsampling the generated single-channel depth map to the same spatial resolution as the multi-scale feature map, then concatenating them along the channel dimension, inputting them into a lightweight convolutional neural network (such as a 1×1 convolution or a small U-Net), and outputting the fused 3D view frustum features. , to 3D view cone features Through the inverse of the camera intrinsic parameter matrix Transform to the world coordinate system (a fixed reference coordinate system in real 3D space), then project onto the BEV space to obtain a structured spatial feature map. This transforms a 3D scene from an image perspective to a bird's-eye view, facilitating monitoring tasks such as target detection, path planning, and radar data fusion. It also eliminates occlusion effects and enhances spatial understanding. The formula is:

[0071]

[0072] in, This is a bird's-eye view projection function that projects a 3D point cloud into the BEV space.

[0073] VAE encoding is used to generate video features. This facilitates fusion with SAR features and improves cross-modal registration accuracy. The formula is:

[0074]

[0075]

[0076] in, From normal distribution Random variables sampled in the middle, The mean, For variance;

[0077] Will and After alignment, the parts are stitched together, and VAE features are incorporated. and Point-Fusion method is used to generate fused features. To resolve cross-modal geometric differences, the formula is:

[0078]

[0079] in, This refers to the convolution operation.

[0080] By using VoxelNet voxelization and ConvLSTM temporal modeling, localized extraction of point cloud temporal features was achieved, avoiding dependence on external computing platforms. Combining VAE with open-source network structures such as Swing Transformer and FPN, deep feature extraction and 3D view frustum construction can be completed in a general hardware environment, reducing the limitations caused by insufficient software or hardware permissions. Furthermore, Point-Fusion was used to align and fuse radar and video features, solving the geometric inconsistency of cross-modal data. Thus, even in real-world scenarios with limited access control and usage conditions, efficient, stable, and operable slope settlement localization can be achieved, effectively overcoming the application limitations of traditional technologies due to high algorithm complexity and limited data permissions.

[0081] S2. Generate a two-dimensional Gaussian heat map based on point cloud data, optimize the simulated SAR image, use fusion features to generate a predicted heat map, and combine the two-dimensional Gaussian heat map with the optimized simulated SAR image to generate an ROI feature map.

[0082] 3D point cloud data is generated from raw point cloud data, the 3D point cloud data is optimized by ROI feature maps, and a digital surface model grid is generated based on the optimized 3D point cloud data.

[0083] Specifically, a two-dimensional Gaussian heatmap is generated based on point cloud data, the simulated SAR image is optimized, a predicted heatmap is generated using fused features, and a Region of Interest (ROI) feature map is generated by combining the two-dimensional Gaussian heatmap and the optimized simulated SAR image. This is achieved through weighted fusion of radar features. and video features The final fusion feature representation is obtained. Based on the collected SAR point cloud data Generate a two-dimensional Gaussian heatmap , representing the spatial distribution probability of the point cloud projection points, is given by the formula:

[0084] ,

[0085] in, An index for point cloud points, used to distinguish different points. For uncertainty, f is a weighting function (such as a linear combination or a fully connected network). This is a normalization constant to ensure that the sum of the probability distributions in the heatmap is 1. , (x) represents the projection coordinates of the point cloud, and (y) represents the two-dimensional coordinates of the heatmap.

[0086] Based on simulated SAR images For each missing pixel (Including original effective pixels or filled pixels), collect local features of effective pixels in the neighborhood (including local texture features such as GLCM, elevation gradient, and intensity of neighboring pixels), input them into a pre-trained lightweight random forest model (the model learns the mapping relationship between local features and optimal interpolation weights by using a large number of known structurally complete SAR image patches as samples during the training phase), and the model outputs the optimized weights of each neighboring effective pixel at the corresponding position. The grayscale values ​​are filled using a weighted average to generate an optimized simulated SAR image. The formula is:

[0087] ,

[0088] in, Let be the gray value of the i-th neighboring valid (i.e., non-empty) pixel. For missing pixels The number of effective neighboring pixels;

[0089] The optimized image Compared with the original simulated SAR image Direct fusion based on pixel position involves the following steps: Identify all previously filled empty regions in the original image (i.e., pixel positions marked as invalid and subsequently assigned values), and perform fusion on each pixel within the region. The corresponding value in the optimized image will be The initial fill value at that location in the original image Weighted fusion is performed, while the original grayscale values ​​of valid, non-empty pixels in the original image are directly retained unchanged. Since the two images have the exact same size and spatial alignment, all pixel positions correspond one-to-one, ultimately generating an optimized simulated SAR image with a more coherent structure and richer details. ;

[0090] Based on fusion features Generate a predicted heatmap (HPF). Multiply the optimized simulated SAR image point-by-point with the predicted heatmap HPF to obtain an enhanced predicted heatmap. Finally, combine the enhanced predicted heatmap with a two-dimensional Gaussian heatmap. Multiply point by point again, and the convolution generates the ROI feature map. Specific steps: Obtain fusion features This forms a unified representation with a dimension of 256, which integrates features. Inputting a fully connected network or convolutional layer maps it to a two-dimensional space, generating a prediction heatmap (HPF) that represents the predicted probability distribution of a slope area. Higher values ​​indicate a greater likelihood that the area is the target region. This is compared to simulated SAR images. Normalization is performed to ensure pixel values ​​are within the range of [0,1]. The normalized simulated SAR image is then multiplied point-by-point with the HPF image to enhance the response intensity of the corresponding real ground structure in the predicted heatmap, resulting in an enhanced predicted heatmap. This enhanced predicted heatmap is then compared with the two-dimensional Gaussian heatmap. The data is multiplied point by point again to further integrate the radar spatial distribution probability and semantic prediction information, forming a fused heatmap. A convolution operation (such as a 3×3 convolution kernel) is then performed on the fused heatmap to extract local features and smooth noise, generating a ROI feature map. .

[0091] In the process of generating two-dimensional Gaussian heat maps and ROI feature maps based on point cloud and video features, the problem of insufficient monitoring accuracy caused by the fragmented data permissions and processing links in existing technologies is effectively alleviated. In traditional methods, different modal data often need to be processed in their own independent permission systems, making it difficult to achieve unified modeling and accurate identification of key areas, resulting in omissions or distortions in the monitoring of high-risk slope areas. By constructing a unified fusion feature representation and generating predictive heat maps, optimizing simulated SAR images and ROI feature maps under the same permission framework, not only is the sharing and collaborative processing of radar and video information realized, but the spatial positioning accuracy and key feature extraction capability of the target area are also improved, thereby ensuring the integrity and reliability of monitoring results under the condition of restricted permission management.

[0092] Furthermore, 3D point cloud data is generated using the original point cloud data, and the 3D point cloud data is optimized using ROI feature maps. Based on the optimized 3D point cloud data, a digital surface model grid is generated. The Range-Doppler model is used to apply the Gauss-Newton iterative method to the collected radar point cloud data. Perform 3D coordinate calculation to obtain preliminary 3D point cloud data. The specific operation is as follows: Based on the Range-Doppler model, initial 3D coordinates are estimated for each radar observation point using the radar echo time, Doppler frequency, and the position and motion parameters of the radar platform. These initial coordinates are then progressively optimized using a Gauss-Newton iterative method: In each iteration, the theoretical observation value is calculated based on the current coordinate estimate and compared with the actual observation value to obtain the error. The error value is then used to calculate the increment for adjusting the current coordinates, updating the coordinate estimate. This process is repeated until the error is reduced to within a set accuracy range, resulting in high-precision preliminary 3D point cloud data. ;

[0093] Combining ROI feature maps 3D point cloud data Regional weighting is performed to preserve point cloud information in high-risk areas (such as cracks and slip surfaces). Specifically, the ROI feature map and 3D point cloud data are spatially aligned using a camera projection model. Each 3D point cloud is projected onto a bird's-eye view, and its corresponding position on the ROI feature map is located. A weight is assigned to each point cloud based on its feature value, with a weight threshold Q set. Points with low weights below the threshold Q are downsampled or removed. This process preserves and enhances the point cloud density and detail information in high-risk areas, achieving enhanced processing of key regions. The same point location is matched across consecutive time frames to calculate settlement. The specific operation involves: accurately matching point clouds at the same location at two consecutive monitoring time points, and using a 1×1 convolutional layer and a sigmoid activation function to fuse the feature representation. The process involves generating a spatial weight map of high-risk areas to improve the registration accuracy between point clouds at different times. Constrained registration is prioritized for point clouds in high-risk areas, and weighted registration is performed using the ICP (Iterative Closest Point) algorithm combined with the spatial weight map. This makes the point cloud matching in high-risk areas more stable and accurate. After achieving precise spatial alignment, for each successfully matched point, the elevation value at the current time is obtained and subtracted from the corresponding elevation value at the previous time. The difference represents the elevation change of that point during that time period. A negative result indicates subsidence (settlement) at that point, while a positive result indicates uplift, thus yielding the settlement amount for each point. ;

[0094] Based on optimized point cloud data A high-resolution digital surface model (DSM) grid is generated using the inverse distance weighted interpolation method. The specific steps are as follows: Based on point cloud data, the output grid resolution and geographic coverage of the digital surface model (DSM) are determined, and a regular two-dimensional grid structure is established. For the center point of each grid cell, point cloud data points within the search range are used to calculate the horizontal distance from the center point to each neighboring point. The square of this distance is used as the weight, and the elevation values ​​of all neighboring points are weighted and averaged according to their corresponding weights to obtain the estimated elevation of the grid center point. All grid cells are traversed and the elevation values ​​are filled one by one to form a complete regular gridded digital surface model (DSM).

[0095] In the process of 3D point cloud data calculation and digital surface model grid generation, this invention effectively overcomes the limitations of existing technologies caused by the fragmented data access and processing permissions. Traditional slope monitoring systems often rely on multiple platforms and modules to complete point cloud calculation, regional weighting, and model generation separately. Due to the isolation of different permission systems, key data may be delayed or even missing during transmission and retrieval, which in turn affects the accuracy of settlement calculation and high-risk area analysis. This invention integrates Range-Doppler calculation, Gauss-Newton iterative optimization, ROI feature weighting, and DSM grid interpolation generation under a unified data processing framework, realizing integrated processing of the entire data process. This avoids redundant operations and information loss caused by cross-permission interface calls, thereby improving the continuity, reliability, and operability of slope monitoring in engineering applications.

[0096] S3. Update the settlement amount and digital surface model grid based on the optimized point cloud data, generate an early warning signal using the settlement amount, and generate an early warning report based on the early warning signal and the digital surface model grid.

[0097] Specifically, based on the optimized point cloud data, the settlement amount and digital surface model grid are updated, and the settlement amount is used to generate early warning signals. This involves converting the VAE and BEV (including RCBEV and a deep learning module with an attention mechanism for generating bird's-eye view features) into a format suitable for embedded devices and deploying them to the radar's embedded system. Pruning techniques are applied to the VAE and BEV to remove 30% redundant data, and the INT8 quantization method is used to reduce the model's computational accuracy requirements. Specifically, a representative set of data (e.g., 100 frames of point cloud-image pairs) is used for forward inference to statistically analyze the numerical distribution range of weights and activation values ​​at each layer. Appropriate quantization parameters, including scaling factors and zero-point offsets, are calculated for each layer, converting the original 32-bit floating-point data into a more accurate representation of the model. The numerical representation of (FP32) is mapped to the integer range of 0~255. All model weights are converted to 8-bit integers (INT8), and the convolution and fully connected operation modules are modified. During actual inference, the input data is also synchronously quantized to INT8 format. The entire forward propagation is executed efficiently at low precision. Finally, the network output is dequantized back to FP32 format to ensure that the model performance degradation is within an acceptable range. While ensuring inference accuracy, the model size and computational overhead are reduced, and the processing time for a single frame of data is controlled within 80 milliseconds to match the radar's 10Hz scanning frequency, ensuring the timeliness and continuity of settlement monitoring and landslide early warning. After the system starts, it continuously receives new frame point cloud data from radar and cameras. Using image data, VAE and BEV models are called for feature extraction to generate a fused BEV feature map. Based on the fused BEV feature map, the settlement amount and digital surface model grid at the current moment are dynamically updated to monitor the settlement change at each point in real time. If the value exceeds the preset threshold J, it is determined to be an abnormal state, triggering the early warning mechanism and generating an early warning signal containing timestamp and location information.

[0098] By pruning and INT8 quantization of deep learning modules such as VAE and BEV, a lightweight model adapted to embedded devices is transformed and directly deployed in the embedded system of the radar. This achieves efficient local inference under a unified permission system, avoiding the limitations caused by cross-system permission calls and external dependencies. It overcomes the problems of low computational efficiency and insufficient real-time performance caused by the dispersion of permissions in existing technologies. This solution not only significantly reduces the model size and computational overhead, thus matching the radar's 10Hz scanning frequency, but also achieves real-time updates of settlement and digital surface models and rapid generation of early warning signals while ensuring accuracy, thereby improving the continuity, reliability, and response timeliness of slope monitoring.

[0099] Furthermore, generating early warning reports based on early warning signals and digital surface model grids refers to generating early warning reports based on early warning signals and real-time settlement data. The digital surface model grid automatically generates early warning reports that include time, location, settlement amount, risk level, and visualization (such as heat map overlaid with satellite image). Through 5G or satellite communication links, the early warning information is pushed in real time to the monitoring center's large screen, management platform, and on-site personnel's mobile terminals. At the same time, it links with the emergency system to activate preset response plans, such as sealing off areas, dispatching patrols, and notifying evacuees.

[0100] By establishing a unified access control system, early warning reports are automatically generated based on early warning signals, real-time settlement data, and digital surface model grids. These reports include information such as time, location, settlement amount, risk level, and visual representations. Real-time cross-terminal push notifications are achieved via 5G or satellite communication links, eliminating the limitations of traditional multi-system access control. Early warning information can be linked to emergency response systems immediately, automatically triggering pre-set response plans. This effectively solves the problems of information transmission delays and untimely responses caused by fragmented access control in existing technologies. As a result, once a risk is detected, rapid transmission and execution can be achieved under unified access control and simplified processes, significantly improving the timeliness and reliability of early warning responses.

[0101] This embodiment also provides a slope settlement location system based on radar combined with video, including:

[0102] The data acquisition and preprocessing module is used to collect point clouds and images from radar and cameras, and combine them with environmental sensor information to generate simulated SAR images and enhanced video images.

[0103] The feature extraction and cross-modal fusion module is used to extract temporal features of point clouds and multi-scale features of videos using deep learning, and achieves cross-modal fusion through 3D view frustum and VAE;

[0104] The heatmap generation and ROI detection module is used to generate a two-dimensional Gaussian heatmap and fuse it with the predicted heatmap to extract ROI feature maps of high-risk areas;

[0105] The 3D point cloud optimization and settlement calculation module is used to optimize point clouds using Range-Doppler modeling and ROI weighted optimization, align multi-time data to calculate settlement and generate DSM grids.

[0106] The real-time inference and early warning signal module is used to output updated settlement and DSM grid in real time. When the settlement exceeds the threshold, it automatically generates an early warning signal containing timestamp and location information.

[0107] This embodiment also provides a computer device applicable to the slope settlement location method based on radar combined with video, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to realize the slope settlement location method based on radar combined with video as proposed in the above embodiment.

[0108] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0109] This embodiment also provides a storage medium storing a computer program. When executed by a processor, the program implements the slope settlement location method and system based on radar combined with video, as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0110] In summary, this invention effectively improves robustness under low light, fog, and vegetation obstruction conditions by combining simulated SAR images with video images enhanced by the Retinex algorithm. It employs VoxelNet and ConvLSTM to extract temporal point cloud features, and combines VAE encoding and Swing Transformer multi-scale image features to construct a 3D view frustum and project it onto the BEV space, significantly enhancing cross-modal feature alignment and spatial understanding capabilities. By constructing a coupling mechanism between Gaussian heatmaps and predicted heatmaps, it generates ROI feature maps, enabling focused enhancement and accurate identification of high-risk areas such as cracks and slip surfaces. This overcomes the shortcomings of traditional methods in identifying and monitoring key areas with insufficient precision, enabling robust slope settlement monitoring in complex environments and improving cross-modal registration accuracy and real-time early warning capabilities.

[0111] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for positioning slope settlement based on radar combined with video, characterized in that: The method comprises the following steps: Based on the point cloud data and image collected and processed by the slope radar and camera, simulated SAR images are generated based on the point cloud data, and enhanced video images are generated by applying the Retinex algorithm; The time sequence features of the point cloud data are extracted, the radar features are obtained by VAE coding based on the time sequence features, the multi-scale features in the enhanced video images are extracted, the spatial feature map is generated by combining the three-dimensional view cone features, the video features are generated based on the spatial feature map, and the fusion features are generated by combining the spatial feature map and the radar features; Based on the point cloud data, a two-dimensional Gaussian heat map is generated, an analog SAR image is optimized, a prediction heat map is generated using fusion features, and an ROI feature map is generated by combining the two-dimensional Gaussian heat map and the optimized analog SAR image, including generating the ROI feature map based on the collected SAR point cloud data Generate a two-dimensional Gaussian heat map Based on the analog SAR image, the interpolation weight of each adjacent valid pixel pair for the vacant position is obtained through a lightweight random forest model The gray value is filled by weighted average to generate a complete analog SAR image The optimized image is fused with the original analog SAR image according to the pixel position to generate an optimized analog SAR image based on the fusion features A prediction heat map HPF is generated, and the enhanced prediction heat map is obtained by multiplying the optimized analog SAR image and the prediction heat map HPF point by point, and the enhanced prediction heat map and the two-dimensional Gaussian heat map are multiplied again point by point to generate an ROI feature map ; 3D point cloud data is generated using the original point cloud data, the 3D point cloud data is optimized through the ROI feature map, and a digital surface model grid is generated based on the optimized 3D point cloud data; Based on the optimized point cloud data, the settlement amount and the digital surface model grid are updated, the pre-warning signal is generated using the settlement amount, and the pre-warning report is generated based on the pre-warning signal and the digital surface model grid. 2.The radar and video-based slope settlement positioning method of claim 1, wherein: The slope radar and the camera collect and process point cloud data and images, generate simulated SAR images based on the point cloud data, and apply Retinex algorithm to generate enhanced video images , deploy high-resolution binocular cameras to collect RGB images ; Adopting self-adapting resolution to convert radar coordinate into geographic coordinate to obtain projection coordinate, forming initial image frame , calculating corresponding incident angle η and scattering coefficient , generating gray value ; For the initial image frame For empty pixel regions, use neighboring points. Values ​​are interpolated to generate a complete simulated SAR image. A video image sequence is composed of consecutive RGB image frames. For each RGB image frame in the video image sequence... The Retinex algorithm is applied to enhance the video image, resulting in an enhanced video image. . 3.The radar and video-based slope settlement positioning method of claim 2, wherein: The time sequence features of the extracted point cloud data are encoded by VAE to obtain radar features, multi-scale features in the enhanced video image are extracted, a spatial feature map is generated by combining the created three-dimensional view cone features, video features are generated based on the spatial feature map, and fusion features are generated by combining the spatial feature map and the radar features Voxelization, spatial feature extraction Multi-frame point cloud processing using ConvLSTM and splicing to obtain time sequence features VAE encoding to extract deep feature vectors and generate radar features Swin Transformer and FPN are used to extract Multi-scale features ; processing the input image using a pre-trained depth estimation model to generate a depth map corresponding to the image , fusing the depth map with multi-scale features to construct a three-dimensional frustum feature , transforming the three-dimensional frustum feature into a world coordinate system through the inverse of the camera intrinsic matrix , and projecting it into a BEV space to obtain a structured spatial feature map , using a VAE to encode video features , aligning and splicing and , combining VAE features and , using a point cloud-image fusion method to generate fusion features . 4.The radar and video-based slope settlement positioning method of claim 3, wherein: The 3D point cloud data is generated by using original point cloud data, optimized by ROI feature map, and a digital surface model grid is generated based on the optimized 3D point cloud data The three-dimensional coordinate solution is performed to obtain preliminary 3D point cloud data , combined with ROI feature map The 3D point cloud data is subjected to regional weighted processing, the point cloud information of high-risk areas is retained, the same point position is matched between continuous time frames, and fusion feature representation is used The settlement amount is calculated , based on the optimized point cloud data , a digital surface model grid is generated by using inverse distance weighted interpolation method. 5.The radar and video-based slope settlement positioning method of claim 4, wherein: The optimized point cloud data is used to update the settlement amount and the digital surface model grid, the settlement amount is used to generate a warning signal, the VAE and the BEV are converted into a format suitable for embedded devices, are deployed to the embedded system of the radar, a pruning technique is applied to the VAE and the BEV, and an INT8 quantization method is used to reduce the model calculation precision requirement, and the radar system continuously receives a new frame of point cloud data from the radar and the camera The image data is called to apply the VAE and the BEV to feature extraction, a fused BEV feature map is generated, the settlement amount and the digital surface model grid at the current time are dynamically updated based on the fused BEV feature map, and if the settlement amount exceeds a preset threshold J, it is determined that an abnormal state occurs, a warning mechanism is triggered, and a warning signal containing a time stamp and position information is generated. 6.The radar and video-based slope settlement positioning method of claim 5, wherein: The generating early warning report based on early warning signal and digital surface model grid comprises the following steps: obtaining early warning signal and real-time settlement data of the monitoring area, and obtaining digital surface model grid of the monitoring area , automatically generating visualized early warning report based on early warning signal, real-time settlement data and digital surface model grid, pushing early warning information to monitoring center large screen, management platform and field personnel mobile terminal in real time, and simultaneously linking emergency system to start preset response plan.

7. The slope settlement positioning system based on radar combined with video, the slope settlement positioning method based on radar combined with video according to any one of claims 1-6, characterized in that: The method comprises the following steps: The data acquisition and preprocessing module is used for radar and camera to collect point cloud and image, and to generate simulated SAR images and enhanced video images in combination with environmental sensor information; The feature extraction and cross-modal fusion module is used for deep learning to extract point cloud time sequence features and video multi-scale features, and to realize cross-modal fusion through three-dimensional view cone and VAE; The heat map generation and ROI detection module is used for generating a two-dimensional Gaussian heat map and fusing with a predicted heat map, extracting a ROI feature map of a high-risk area, and includes a ROI feature map extraction based on collected SAR point cloud data Generating a two-dimensional Gaussian heat map Based on the simulated SAR image, an interpolation weight of each adjacent effective pixel pair for a vacant position is obtained through a lightweight random forest model The gray value is filled by weighted average to generate a complete simulated SAR image The optimized image is fused with the original simulated SAR image by pixel position to generate an optimized simulated SAR image based on the fusion features A predicted heat map HPF is generated, and the optimized simulated SAR image and the predicted heat map HPF are multiplied point by point to obtain an enhanced predicted heat map, and the enhanced predicted heat map and the two-dimensional Gaussian heat map are multiplied point by point again to generate a ROI feature map ; The three-dimensional point cloud optimization and settlement calculation module is used for Range-Doppler modeling and ROI weighting to optimize point cloud, to calculate settlement amount and generate DSM grid by aligning multi-time data; The real-time inference and pre-warning signal module is used for real-time output of updated settlement amount and DSM grid, and to automatically generate pre-warning signal containing time stamp and location information when the settlement amount exceeds the threshold.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to realize the steps of the slope settlement positioning method based on radar combined with video according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to realize the steps of the slope settlement positioning method based on radar combined with video according to any one of claims 1-6.

Images