Collapsed rockfall accurate identification and monitoring method and device based on AI and machine vision

Abstract

The invention provides a collapsed rockfall accurate identification and monitoring method and device based on AI and machine vision, and relates to the technical field of geological disaster monitoring, and the method comprises the steps: collecting the multi-modal visual data and environmental parameters of a collapse hidden danger region through a cooperative collection system; performing recognition, contour extraction and spatial parameter calculation on the multi-modal visual data through the recognition and positioning model to obtain a contour sequence, a volume sequence, a spatial coordinate sequence and a spatial attitude sequence of the collapsed rockfall; on the basis of the space coordinate sequence, the space attitude sequence and the multi-modal visual data, predicting the movement track and the speed sequence of the collapsed rockfall through a mixed time sequence AI model; inputting the environment parameters, the contour sequence, the volume sequence, the motion trail and the speed sequence into a kinetic energy calculation model, and calculating to obtain the impact energy of the collapsed rockfall; according to the contour sequence, the volume sequence, the motion trail, the speed sequence and the impact energy of the collapsed rockfall, the collapsed rockfall risk level is obtained through calculation.

Description

Technical Field

[0001] This application relates to the field of geological disaster monitoring technology, and in particular to a method and device for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision. Background Technology

[0002] Landslides and rockfalls are common and sudden geological disasters characterized by high frequency, high speed, and strong impact destructive force. Once they occur, they can easily lead to traffic disruptions, engineering damage, and even casualties. Therefore, accurate and real-time monitoring is crucial. However, current landslide and rockfall monitoring technologies suffer from insufficient accuracy in rockfall identification, significant deviations in trajectory and velocity prediction, and inaccurate impact energy calculations. These shortcomings make it difficult to meet the needs of precise monitoring and risk warning in complex scenarios. Specific deficiencies are as follows:

[0003] In the data acquisition and recognition stage, existing technologies mostly rely on single-type visual data for rockfall identification. This type of data is easily affected by changes in lighting, weather interference, and background environment, resulting in reduced differentiation between rocks and background. In complex scenarios, the false negative and false positive rates are high, making it difficult to reliably capture rockfall targets. At the same time, although some technologies introduce multi-source data, they have not formed an effective data collaborative processing mechanism, which cannot give full play to the complementary advantages of different data, further limiting the accuracy of recognition.

[0004] In terms of calculating rockfall motion parameters and predicting trajectories, traditional methods rely heavily on manual measurement or simplified model derivation for calculating spatial parameters of falling rocks. This is not only inefficient but also prone to errors due to human error and model assumption bias. In terms of trajectory and velocity prediction, existing models often use a single temporal feature extraction method, which makes it difficult to simultaneously take into account the local dynamic features of multimodal visual data and the long-sequence variation features of spatial parameters. Furthermore, feature fusion often uses a fixed weight method, which can easily lead to the dilution of key features. Ultimately, this results in large trajectory prediction deviations and inaccurate velocity sequence estimations, making it impossible to achieve dynamic control over the entire process of rockfall motion.

[0005] In terms of impact energy assessment, existing kinetic energy calculation models mostly only consider the basic kinetic energy term and ignore the actual environmental factors affecting the movement of falling rocks, resulting in a significant deviation between the calculated results and the actual impact energy of falling rocks. At the same time, the calculation of impact energy does not fully incorporate the multi-dimensional real-time parameters of falling rocks, further reducing the reliability of energy assessment and making it difficult to provide accurate quantitative basis for the key risk level determination in this field. Summary of the Invention

[0006] To address the aforementioned problems, this invention provides a method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, comprising:

[0007] Multimodal visual data and environmental parameters of the landslide hazard area are collected through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data, and three-dimensional point cloud data.

[0008] A recognition and localization model is constructed. Through the recognition and localization model, the collapse and falling rocks are identified, their contours are extracted, and their spatial parameters are calculated from the multimodal visual data. The contour sequence, volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapse and falling rocks are obtained.

[0009] A hybrid temporal AI model is constructed to predict the motion trajectory and velocity sequence of falling rocks based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data.

[0010] A kinetic energy calculation model is constructed, and environmental parameters, contour sequence, volume sequence, motion trajectory and velocity sequence are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rock.

[0011] The risk level of landslides is calculated based on the contour sequence, volume sequence, trajectory, velocity sequence, and impact energy of the landslides.

[0012] Optionally, the step of identifying, extracting contours, and calculating spatial parameters of collapsed rocks from multimodal visual data using a recognition and localization model to obtain contour sequences, volume sequences, spatial coordinate sequences, and spatial attitude sequences of collapsed rocks includes:

[0013] The identification and localization models include the lightweight YOLO-Lite model and the Faster R-CNN model;

[0014] The lightweight YOLO-Lite model is used to identify landslides and falling rocks in two-dimensional image data, and target areas containing landslides and falling rocks are selected.

[0015] Based on thermal radiation data, the target area is segmented into pixels using the Faster R-CNN model to obtain the contour sequence of the collapsed rocks.

[0016] Spatial feature matching is performed between 3D point cloud data and contour sequences to obtain effective point cloud data;

[0017] Based on effective point cloud data, the volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapsed rocks were calculated using a three-dimensional coordinate transformation algorithm.

[0018] Optionally, the step of calculating the volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapsed rocks based on effective point cloud data using a three-dimensional coordinate transformation algorithm includes:

[0019] The effective point cloud data is mapped to an effective voxel set. The volume of the collapsed rocks at each time point is calculated based on the effective voxel set. The volumes are then arranged into a volume sequence in chronological order.

[0020] The effective point cloud data is converted to the world coordinate system to obtain the real point cloud data at each time point. The geometric centroid coordinates of the collapsed rocks at each time point are calculated based on the real point cloud data. The geometric centroid coordinates are used as the spatial coordinates of the collapsed rocks. The spatial coordinates are arranged into a spatial coordinate sequence according to the time order.

[0021] Calculate the covariance matrix of the effective point cloud data at each time point, perform eigenvalue decomposition on the covariance matrix to obtain the major axis eigenvector, calculate the tilt angle based on the major axis eigenvector and the X-axis of the world coordinate system, determine the spatial attitude based on the tilt angle, and assemble the spatial attitudes into a spatial attitude sequence according to the time order.

[0022] Optionally, the hybrid temporal AI model includes a temporal convolutional network, a long short-term memory network, an attention mechanism fusion layer, and a fully connected layer.

[0023] Optionally, the prediction of the trajectory and velocity sequence of collapsed rocks using a hybrid temporal AI model based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data includes:

[0024] Local dynamic features are extracted from two-dimensional image data, thermal radiation data, and three-dimensional point cloud data using a temporal convolutional network to obtain temporal features of images, thermal radiation, and point clouds. These temporal features are then concatenated to form visual temporal features.

[0025] Long short-term memory network is used to extract long sequence change features from spatial coordinate sequence and spatial attitude sequence to obtain coordinate temporal features and attitude temporal features. The coordinate temporal features and attitude temporal features are then concatenated to form spatial temporal features.

[0026] The visual temporal features and spatial temporal features are weighted and fused through an attention mechanism fusion layer to obtain fused temporal features. The fused temporal features are then mapped to the motion trajectory and velocity sequence of the falling rocks through a fully connected layer.

[0027] Optionally, the kinetic energy calculation model includes a basic kinetic energy term, an air resistance correction term, a slope friction correction term, and a wind resistance correction term.

[0028] Optionally, the step of inputting environmental parameters, contour sequence, volume sequence, motion trajectory, and velocity sequence into the kinetic energy calculation model to calculate the impact energy of the collapsed rockfall includes:

[0029] Environmental parameters include air resistance coefficient, air density, slope friction coefficient, slope angle, rock density, wind speed and wind direction. Based on the contour sequence, volume sequence and velocity sequence, the contour, volume and velocity of the collapsed rock at the last time point are obtained respectively. Based on the motion trajectory, the time interval and straight distance between the last time point and the previous time point of the collapsed rock are obtained.

[0030] The mass is calculated based on the volume and rock density, and the initial kinetic energy is calculated by substituting the mass and velocity into the basic kinetic energy term.

[0031] The windward area is obtained by calculating the profile. The air resistance coefficient, air density, windward area, speed and time interval are substituted into the air resistance correction term to calculate the air resistance kinetic energy loss.

[0032] Substitute the slope friction coefficient, mass, straight distance and slope angle into the slope friction correction term to calculate the slope friction kinetic energy loss.

[0033] The drag velocity component is calculated based on wind speed and wind direction. The air drag coefficient, air density, windward area, speed, drag velocity component and time interval are substituted into the wind resistance correction term to calculate the environmental wind kinetic energy loss.

[0034] The total loss is the sum of the kinetic energy loss due to air resistance, the kinetic energy loss due to slope friction, and the kinetic energy loss due to ambient wind. The impact energy is obtained by subtracting the total loss from the initial kinetic energy.

[0035] This invention also provides a device for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, used to implement the aforementioned method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision. The device includes:

[0036] The data acquisition module is used to collect multimodal visual data and environmental parameters of the landslide hazard area through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data, and three-dimensional point cloud data.

[0037] The first parameter calculation module is used to construct a recognition and localization model. Through the recognition and localization model, the module performs the recognition, contour extraction and spatial parameter calculation of the multimodal visual data of the landslide and falling rocks, and obtains the contour sequence, volume sequence, spatial coordinate sequence and spatial attitude sequence of the landslide and falling rocks.

[0038] The second parameter calculation module is used to construct a hybrid temporal AI model. Based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data, the hybrid temporal AI model predicts the motion trajectory and velocity sequence of collapsed rocks.

[0039] The impact energy calculation module is used to build a kinetic energy calculation model. Environmental parameters, contour sequences, volume sequences, motion trajectories, and velocity sequences are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rocks.

[0040] The monitoring result acquisition module is used to calculate the risk level of landslides based on the contour sequence, volume sequence, trajectory, velocity sequence and impact energy of the landslides.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the aforementioned method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision.

[0042] The present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision.

[0043] The present invention has the following beneficial effects:

[0044] 1. By using an identification and localization model to recognize, extract contours, and calculate spatial parameters of multimodal visual data, this method overcomes the limitations of single data being susceptible to environmental interference and improves the accuracy of rockfall recognition in complex scenarios. Based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data, a hybrid temporal AI model accurately predicts the trajectory and velocity sequences of falling rocks, enabling dynamic control of the entire rockfall process. By constructing a kinetic energy calculation model and combining environmental parameters with multidimensional parameters of falling rocks, the impact energy can be comprehensively assessed, providing an accurate basis for risk level determination. Finally, based on the contour sequence, volume sequence, trajectory, velocity sequence, and impact energy of the falling rocks, the risk level of the falling rocks is calculated, integrating comprehensive rockfall detection data to provide comprehensive support for accurate early warning and emergency response decisions regarding falling rocks.

[0045] 2. The identification and localization models include the lightweight YOLO-Lite model and the Faster R-CNN model. The lightweight YOLO-Lite model can achieve fast inference in a single frame, efficiently filtering out target areas containing collapsed rocks, avoiding redundant calculations of invalid areas in the entire image by subsequent models, and significantly improving data processing efficiency. Combined with thermal radiation data, pixel segmentation is performed by the Faster R-CNN model. Thermal radiation data can effectively distinguish the temperature difference features between the rocks and the background. With the high-precision pixel-level segmentation capability of Faster R-CNN, the contour sequence of the rocks can be accurately extracted. Spatial feature matching is performed between the 3D point cloud data and the contour sequence, which can filter out invalid point clouds such as slope background and vegetation from the massive 3D point cloud, accurately locking the valid point cloud data corresponding to the rock contour. Finally, based on the accurate valid point cloud data, the high-precision volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapsed rocks are calculated.

[0046] 3. Local dynamic features of multimodal visual data are extracted through a temporal convolutional network. Long short-term memory network is used to extract long-period change features of spatial coordinate sequences and spatial pose sequences. Then, an attention mechanism fusion layer is used to achieve dynamic weighted fusion of visual temporal features and spatial temporal features, allowing the model to automatically focus on high-contribution features and avoid the key feature dilution problem caused by traditional fixed-weight fusion, thus improving fusion efficiency. Finally, a fully connected layer is used to accurately map the fused features into motion trajectories and velocity sequences, ensuring that the model output matches the actual physical quantities and significantly improving the prediction accuracy of trajectory and velocity.

[0047] 4. By introducing air resistance correction terms, slope friction correction terms, and environmental wind correction terms into the kinetic energy calculation model, the calculation of impact energy is made to fit the actual movement scenario and environmental conditions of falling rocks. At the same time, the calculation process is more in line with the real energy consumption mechanism of falling rocks moving along the slope, which greatly reduces the calculation error caused by traditional reliance on empirical values, improves the accuracy of impact energy calculation, and provides a reliable quantitative basis for risk level determination. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0051] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0052] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0053] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] Reference Figure 1 This invention provides a method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, including:

[0056] Multimodal visual data and environmental parameters of the landslide hazard area are collected through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data, and three-dimensional point cloud data.

[0057] In some embodiments, a collaborative acquisition system is used to collect data from areas at risk of landslides. The collected data includes multimodal visual data and environmental parameters. Specifically, the multimodal visual data encompasses two-dimensional image data, thermal radiation data, and three-dimensional point cloud data. During the acquisition process, the collaborative acquisition system must selectively cover key areas of the landslide-prone region to ensure that the collected two-dimensional image data clearly presents the appearance of objects within the area, providing basic visual information for the initial identification of potential landslides. Thermal radiation data must capture the differences in thermal radiation from different objects within the area to distinguish falling rocks from other elements in the background environment, such as vegetation and soil. Three-dimensional point cloud data must completely record the three-dimensional spatial structure of objects within the area, providing accurate spatial data support for the subsequent calculation of the volume, spatial coordinates, and spatial orientation of falling rocks. Simultaneously, the collaborative acquisition system must collect environmental parameters, including air drag coefficient, air density, slope friction coefficient, slope angle, rock density, wind speed, and wind direction. These parameters will serve as important bases for the subsequent calculation of the impact energy of falling rocks, ensuring the synchronicity and integrity of various data during the acquisition process, and laying a reliable data foundation for the subsequent identification, monitoring, and calculation of related parameters of falling rocks.

[0058] A recognition and localization model is constructed. Through the recognition and localization model, the collapse and falling rocks are identified, their contours are extracted, and their spatial parameters are calculated from the multimodal visual data. The contour sequence, volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapse and falling rocks are obtained.

[0059] In some embodiments, the step of identifying, extracting contours, and calculating spatial parameters of landslides from multimodal visual data using a recognition and localization model to obtain contour sequences, volume sequences, spatial coordinate sequences, and spatial attitude sequences of the landslides includes:

[0060] The identification and localization models include the lightweight YOLO-Lite model and the Faster R-CNN model;

[0061] The lightweight YOLO-Lite model is used to identify landslides and falling rocks in two-dimensional image data, and target areas containing landslides and falling rocks are selected.

[0062] In some embodiments, for the lightweight YOLO-Lite model, the two-dimensional image data acquired by the collaborative acquisition system is first preprocessed, and the images are uniformly adjusted to a standard size of 416×416 pixels. Data enhancement is achieved through random brightness adjustment, contrast enhancement, horizontal flipping, and Gaussian blurring. Then, median filtering is used to remove salt-and-pepper noise. Subsequently, a dataset containing different collapse scenes and falling rocks of different sizes is constructed. After labeling the falling rock categories, it is divided into training, validation, and test sets. The model is built based on the Darknet framework, with a batch size of 16, an initial learning rate of 0.001, and 100 epochs. The Adam optimizer and CIoULoss loss function are used for training. After training, the recognition accuracy must be ≥92%, the single-frame inference time ≤50ms, and the model size ≤100MB. If these requirements are not met, pruning or quantization optimization is performed. Finally, the preprocessed real-time two-dimensional images are input into the model, and detection boxes are filtered with a confidence threshold of 0.7. The coordinates of the target area are recorded and associated with the timestamp to obtain the target area containing collapsed falling rocks.

[0063] Based on thermal radiation data, the target area is segmented into pixels using the Faster R-CNN model to obtain the contour sequence of the collapsed rocks.

[0064] In some embodiments, for the Faster R-CNN model, the thermal radiation data is first preprocessed, Gaussian filtering is used to remove electronic noise, morphological opening operation is used to eliminate small areas of thermal interference, and the temperature range is mapped to gray values ​​[0, 255] by linear stretching to enhance temperature features. Then, according to the spatial calibration parameters of the collaborative acquisition system, the coordinates of the thermal radiation data are aligned with the coordinates of the two-dimensional image. Next, the model is built based on the PyTorch framework, with ResNet-50 as the backbone network. A temperature feature fusion module is added after the RoIPooling layer. A thermal radiation segmentation dataset corresponding to the two-dimensional image dataset is constructed and labeled. The batch size is set to 8, the initial learning rate is 0.0005, and the number of iterations is 80 epochs. The SGD optimizer and cross-entropy loss + Dice loss are used for training, and the segmentation accuracy IoU ≥ 88% is verified. Then, for the thermal radiation data of each time stamp, the target area selected by YOLO-Lite is extracted and input into the model to obtain the pixel-level segmentation mask of the falling rocks. The contour pixel coordinates are extracted by Canny edge detection and arranged in the order of time stamps to form the contour sequence of the collapsed rocks.

[0065] Spatial feature matching is performed between 3D point cloud data and contour sequences to obtain effective point cloud data;

[0066] In some embodiments, in the spatial feature matching stage between 3D point cloud and contour sequence, the 3D point cloud data is first preprocessed, outliers are removed by statistical filtering, the number of point clouds is reduced by downsampling with a 2cm×2cm×2cm voxel grid, and the point cloud is transformed from the local coordinate system of the LiDAR to the world coordinate system. Then, the pixel coordinates of the rockfall contour extracted by Faster R-CNN are converted into a 2D projection contour in the world coordinate system by the camera perspective projection formula. The preprocessed 3D point cloud is projected onto the XY plane of the world coordinate system, and the point clouds whose projection points fall within the contour projection range are selected. The normal vectors of these point clouds are calculated, and the point clouds with a difference of >30° from the slope normal vector are retained as valid point cloud data. At the same time, the matching IoU is verified to be ≥85%. If it is not satisfied, the projection parameters are adjusted or the noise is removed again.

[0067] Based on effective point cloud data, the volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapsed rocks were calculated using a three-dimensional coordinate transformation algorithm.

[0068] In some embodiments, the step of calculating the volume sequence, spatial coordinate sequence, and spatial attitude sequence of collapsed rocks based on effective point cloud data using a three-dimensional coordinate transformation algorithm includes:

[0069] The effective point cloud data is mapped to an effective voxel set. The volume of the collapsed rocks at each time point is calculated based on the effective voxel set. The volumes are then arranged into a volume sequence in chronological order.

[0070] In some embodiments, when calculating the volume sequence, for each time step, the effective point cloud data at time t is... Mapped to a side length of Effective voxel set The voxel size must be consistent with the voxel downsampling parameters in the 3D point cloud preprocessing stage. The voxel mesh coverage is bounded by the spatial distribution of valid point clouds within a single timestamp. If a voxel contains at least one valid point cloud data point, it is determined to be a valid voxel and its number is counted. Then, the volume of the collapsed rockfall at a single timestamp is calculated based on the set of valid voxels. , The number of effective voxels; finally, the volume of the falling rocks at each time point is arranged in order of time stamp from earliest to latest to form a volume sequence;

[0071] The effective point cloud data is converted to the world coordinate system to obtain the real point cloud data at each time point. The geometric centroid coordinates of the collapsed rocks at each time point are calculated based on the real point cloud data. The geometric centroid coordinates are used as the spatial coordinates of the collapsed rocks. The spatial coordinates are arranged into a spatial coordinate sequence according to the time order.

[0072] In some embodiments, when calculating the spatial coordinate sequence, the effective point cloud data at time t is first... Transform to world coordinate system The X-axis is defined as horizontally eastward, the Y-axis as horizontally northward, and the Z-axis as vertically upward. Based on the intrinsic and extrinsic parameters of the lidar, the true coordinates of each point in the effective point cloud in the world coordinate system are calculated using formulas to obtain the true point cloud data at each time point. Then, based on the actual point cloud data, the spatial coordinates of the collapsed rocks at a single timestamp are calculated. Using the coordinates of all points in the actual point cloud data at that timestamp as a basis, the average values ​​in the X, Y, and Z axes are calculated respectively to obtain the geometric centroid coordinates at time t. The coordinates of the center of gravity are used as the spatial coordinates of the falling rocks; finally, the spatial coordinates of each time point are arranged in order of timestamp to form a spatial coordinate sequence.

[0073] Calculate the covariance matrix of the effective point cloud data at each time point, perform eigenvalue decomposition on the covariance matrix to obtain the major axis eigenvector, calculate the tilt angle based on the major axis eigenvector and the X-axis of the world coordinate system, determine the spatial attitude based on the tilt angle, and assemble the spatial attitudes into a spatial attitude sequence according to the time order.

[0074] In some embodiments, based on the real point cloud data at time t The deviations of the X, Y, and Z coordinates of each point from the barycentric coordinates of the corresponding axes are calculated. Then, the covariance between each coordinate component is calculated using the deviations, thus constructing a 3×3 covariance matrix. Next, eigenvalue decomposition is performed on the covariance matrix to obtain three eigenvalues ​​and their corresponding eigenvectors. The eigenvector corresponding to the largest eigenvalue is taken as the major axis eigenvector at time t. Then, the tilt angle is calculated based on the major axis eigenvector and the X-axis of the world coordinate system. The angle between the major axis eigenvector and the X-axis unit vector is calculated using the vector dot product formula. This angle is the tilt angle at time t. The expression is:

[0075]

[0076] in, The unit vector representing the X-axis, passing through the tilt angle Determine the spatial attitude of the collapsed rocks at time t, and control the tilt angle accuracy to not exceed 0.5°; finally, arrange the spatial attitudes of each time point in order of timestamp to form a spatial attitude sequence.

[0077] A hybrid temporal AI model is constructed to predict the motion trajectory and velocity sequence of falling rocks based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data.

[0078] In some embodiments, the hybrid temporal AI model includes a temporal convolutional network, a long short-term memory network, an attention mechanism fusion layer, and a fully connected layer.

[0079] In some embodiments, predicting the trajectory and velocity sequence of falling rocks using a hybrid temporal AI model based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data includes:

[0080] Local dynamic features are extracted from two-dimensional image data, thermal radiation data, and three-dimensional point cloud data using a temporal convolutional network to obtain temporal features of images, thermal radiation, and point clouds. These temporal features are then concatenated to form visual temporal features.

[0081] In some embodiments, the two-dimensional image data, thermal radiation data, and three-dimensional point cloud data are first preprocessed. A time step T synchronized with the acquisition frequency is set, and the three types of data are converted into time-series frame sequences in the format [T, height, width, number of channels]: the two-dimensional image data is resized to 224×224 pixels, retaining the RGB three channels, and normalized to [0,1] by dividing the pixel value by 255, forming a two-dimensional image sequence of [T, 224, 224, 3]; the thermal radiation data is resized to 224×224 pixels with temperature grayscale value as a single channel, and normalized to [0,1] by dividing the grayscale value by 255, forming a two-dimensional image sequence of [T, 224, 224, 3]. The thermal radiation sequence is generated as [T, 224, 224, 1]. The 3D point cloud data is projected onto the XY plane to generate a 224×224 pixel density heatmap as a single channel. After normalization, a 3D point cloud sequence of [T, 224, 224, 1] is formed. Temporal flipping and random frame completion are performed on the three types of sequences to enhance the data. Then, three independent temporal convolutional network (TCN) branches are constructed. Each branch uses basic units containing causal convolution, dilated convolution, and residual connections. After four layers of basic units, global average pooling is used to compress the feature map into [T, 64] dimensional temporal features, yielding the image temporal features. Temporal characteristics of thermal radiation Point cloud temporal features Finally, the features are aligned according to the time step, and the three types of features are concatenated along the feature dimension to form the visual temporal feature [T,192]. , || represents the vector concatenation operation;

[0082] Long short-term memory network is used to extract long sequence change features from spatial coordinate sequence and spatial attitude sequence to obtain coordinate temporal features and attitude temporal features. The coordinate temporal features and attitude temporal features are then concatenated to form spatial temporal features.

[0083] In some embodiments, the spatial coordinate sequence and spatial pose sequence are first preprocessed to ensure that the time step is consistent with the TCN, and the feature values ​​are scaled to the [-1,1] interval based on the mean and standard deviation of the training set using Z-score standardization. Then, two independent Long Short-Term Memory (LSTM) branches are constructed, each with two LSTM layers, and a dropout layer is added between the two layers. The spatial coordinate branch takes a spatial coordinate sequence of [T,3] as input and outputs a coordinate temporal feature of [T,64]. The spatial attitude branch takes a spatial attitude sequence of [T,1] as input and outputs the attitude temporal features of [T,64]. Finally, align the features by time step and concatenate the two types of features along the feature dimension to form the spatial-temporal feature [T, 128]. ;

[0084] The visual temporal features and spatial temporal features are weighted and fused through an attention mechanism fusion layer to obtain fused temporal features. The fused temporal features are then mapped to the motion trajectory and velocity sequence of the falling rocks through a fully connected layer.

[0085] In some embodiments, both the visual temporal features [T,192] and the spatial temporal features [T,128] are first mapped to 128 dimensions using a 1×1 convolutional layer to obtain the visual mapped features. and spatial mapping features The attention mechanism fusion layer maps visual features using a learnable weight matrix. and spatial mapping features Weighting is performed to obtain intermediate time series features. The expression is:

[0086]

[0087] in, This represents the hyperbolic tangent activation function. and Let b represent the learnable weight matrix, and b represent the bias vector.

[0088] Intermediate temporal features are captured by a normalized attention function. Mapped to the first weight vector Second weight vector , Fusion of temporal features ; will integrate temporal features The input layer is a fully connected layer, and the output layer is the trajectory of the motion in the future T steps. and velocity sequence .

[0089] A kinetic energy calculation model is constructed, and environmental parameters, contour sequence, volume sequence, motion trajectory and velocity sequence are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rock.

[0090] In some embodiments, the kinetic energy calculation model includes a basic kinetic energy term, an air resistance correction term, a slope friction correction term, and a wind resistance correction term.

[0091] In some embodiments, the step of inputting environmental parameters, contour sequences, volume sequences, motion trajectories, and velocity sequences into a kinetic energy calculation model to calculate the impact energy of the collapsed rockfall includes:

[0092] Environmental parameters include air resistance coefficient, air density, slope friction coefficient, slope angle, rock density, wind speed and wind direction. Based on the contour sequence, volume sequence and velocity sequence, the contour, volume and velocity of the collapsed rock at the last time point are obtained respectively. Based on the motion trajectory, the time interval and straight distance between the last time point and the previous time point of the collapsed rock are obtained.

[0093] In some embodiments, environmental parameters include the air drag coefficient. air density Slope friction coefficient Slope angle Rock density Wind speed and wind direction The contours at the last time point were read from the contour sequence, volume sequence, and velocity sequence, respectively. ,volume With speed The time interval between two frames is determined from the motion trajectory. distance from the line , Indicates the last point in time. This indicates the spatial location of the center of gravity of the collapsed rocks at the last point in time.

[0094] The mass is calculated based on the volume and rock density, and the initial kinetic energy is calculated by substituting the mass and velocity into the basic kinetic energy term.

[0095] In some embodiments, mass is calculated from volume and rock density. initial kinetic energy The expression is:

[0096]

[0097] The windward area is obtained by calculating the profile. The air resistance coefficient, air density, windward area, speed and time interval are substituted into the air resistance correction term to calculate the air resistance kinetic energy loss.

[0098] In some embodiments, the outline The projected area perpendicular to the direction of motion is taken as the windward area. Air resistance and kinetic energy loss The expression is:

[0099]

[0100] Substitute the slope friction coefficient, mass, straight distance and slope angle into the slope friction correction term to calculate the slope friction kinetic energy loss.

[0101] In some embodiments, the expression for the kinetic energy loss due to slope friction is:

[0102]

[0103] in, Represents gravitational acceleration;

[0104] The drag velocity component is calculated based on wind speed and wind direction. The air drag coefficient, air density, windward area, speed, drag velocity component and time interval are substituted into the wind resistance correction term to calculate the environmental wind kinetic energy loss.

[0105] In some embodiments, the drag velocity component, which is opposite to the motion, is calculated from wind speed and wind direction. :

[0106]

[0107] Where u represents velocity The direction of the unit vector;

[0108] Then wind resistance kinetic energy loss The expression is:

[0109]

[0110] The total loss is the sum of the kinetic energy loss due to air resistance, the kinetic energy loss due to slope friction, and the kinetic energy loss due to ambient wind. The impact energy is obtained by subtracting the total loss from the initial kinetic energy.

[0111] In some embodiments, total loss Impact energy .

[0112] The risk level of landslides is calculated based on the contour sequence, volume sequence, trajectory, velocity sequence, and impact energy of the landslides.

[0113] In some embodiments, basic source tracing and credibility verification are first performed, recording the identification of the hazard area, including ID, latitude and longitude, and slope type; data collection period; information on the collection equipment, including model, deployment location, and calibration time; verification of core data, including outline and volume; and verification of the logical rationality of parameters, with any abnormalities marked.

[0114] Then, multi-dimensional time series data analysis was performed. The contour sequence was selected from three key time points: initial, middle and final, for visualization and comparison. The average volume and rate of change were calculated through the volume sequence, and a volume-time graph was drawn with abrupt change points marked. The center of gravity coordinates were extracted through the motion trajectory, and a three-dimensional trajectory and a two-dimensional projection graph were drawn. The velocity-time graph and component stacking graph were drawn through the velocity sequence.

[0115] Finally, an impact energy analysis is performed, presenting the impact energy values ​​and simultaneously displaying the initial kinetic energy and the loss values ​​and proportions of three types of losses: air resistance, slope friction, and ambient wind. An energy composition-time diagram is also drawn.

[0116] Risk levels are determined based on volume, terminal velocity, and impact energy. The extent of damage is predicted by combining this with trajectory data, and emergency response recommendations are then provided. This information is integrated into a monitoring report, ensuring timestamp alignment and unit consistency, to support disaster early warning and emergency response, as detailed below:

[0117] Obtain the equivalent diameter d and contour irregularity I at the last time node of the contour sequence. The contour sequence score is as follows: d < 0.5m and I ≤ 1.2, 3 points; 0.5m ≤ d < 1.0m and 1.2 < I ≤ 1.5, 6 points; 1.0m ≤ d < 2.0m and 1.5 < I ≤ 1.8, 10 points; d ≥ 2.0m and I > 1.8, 15 points.

[0118] The average volume V is obtained from the volume sequence. The volume sequence score is as follows: V < 1.0 m³, 4 points; 1.0 m³ ≤ V < 5.0 m³, 8 points; 5.0 m³ ≤ V < 10.0 m³, 14 points; V ≥ 10.0 m³, 20 points.

[0119] The standard deviation σ of trajectory stability is obtained through the motion trajectory. The motion trajectory score is as follows: σ < 0.5m, 4 points; 0.5m ≤ σ < 1.0m, 8 points; 1.0m ≤ σ < 2.0m, 14 points; σ ≥ 2.0m, 20 points.

[0120] The terminal velocity v and the rate of change of velocity r are obtained from the velocity sequence. The velocity sequence scores are as follows: v < 5.0 m / s and r ≤ 0, 4 points; 5.0 m / s ≤ v < 15.0 m / s and 0 < r ≤ 1.0 m / s², 8 points; 15.0 m / s ≤ v < 30.0 m / s and 1.0 m / s² < r ≤ 2.0 m / s², 14 points; v ≥ 30.0 m / s and r > 2.0 m / s², 20 points.

[0121] Impact energy scores are as follows: 5 points for impact energy E < 100kJ; 100kJ ≤ E < 500kJ; 18 points for 500kJ ≤ E < 2000kJ; and 25 points for E ≥ 2000kJ.

[0122] Overall risk score = contour sequence score × 15% + volume sequence score × 20% + motion trajectory score × 20% + velocity sequence score × 20% + impact energy score × 25%;

[0123] A comprehensive risk score of <30 indicates low risk, 30 to <60 indicates medium risk, 60 to <85 indicates high risk, and a comprehensive risk score of ≥85 indicates extremely high risk.

[0124] Reference Figure 2 This invention provides a device 20 for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, used to realize a method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision. The device includes:

[0125] Data acquisition module 21 is used to collect multimodal visual data and environmental parameters of the landslide hazard area through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data and three-dimensional point cloud data.

[0126] The first parameter calculation module 22 is used to construct an identification and localization model. Through the identification and localization model, the multimodal visual data is used to identify, extract contours and calculate spatial parameters of the collapsed rocks, and obtain the contour sequence, volume sequence, spatial coordinate sequence and spatial attitude sequence of the collapsed rocks.

[0127] The second parameter calculation module 23 is used to construct a hybrid temporal AI model, which predicts the motion trajectory and velocity sequence of collapsed rocks based on spatial coordinate sequence, spatial attitude sequence and multimodal visual data.

[0128] The impact energy calculation module 24 is used to construct a kinetic energy calculation model. Environmental parameters, contour sequence, volume sequence, motion trajectory and velocity sequence are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rock.

[0129] The monitoring result acquisition module 25 is used to calculate the risk level of landslides based on the contour sequence, volume sequence, motion trajectory, velocity sequence and impact energy of the landslides.

[0130] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements any of the above-described schemes for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision.

[0131] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0132] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0133] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the AI- and machine vision-based method for accurate identification and monitoring of landslides and falling rocks, as described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0134] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0135] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, characterized in that, include: Multimodal visual data and environmental parameters of the landslide hazard area are collected through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data, and three-dimensional point cloud data. A recognition and localization model is constructed. Through the recognition and localization model, the collapse and falling rocks are identified, their contours are extracted, and their spatial parameters are calculated from the multimodal visual data. The contour sequence, volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapse and falling rocks are obtained. A hybrid temporal AI model is constructed to predict the motion trajectory and velocity sequence of falling rocks based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data. A kinetic energy calculation model is constructed, and environmental parameters, contour sequence, volume sequence, motion trajectory and velocity sequence are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rock. The risk level of landslides is calculated based on the contour sequence, volume sequence, trajectory, velocity sequence, and impact energy of the landslides.

2. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision as described in claim 1, characterized in that, The process involves using a recognition and localization model to identify, extract contours, and calculate spatial parameters of landslides from multimodal visual data, obtaining contour sequences, volume sequences, spatial coordinate sequences, and spatial attitude sequences of the landslides, including: The identification and localization models include the lightweight YOLO-Lite model and the Faster R-CNN model; The lightweight YOLO-Lite model is used to identify landslides and falling rocks in two-dimensional image data, and target areas containing landslides and falling rocks are selected. Based on thermal radiation data, the target area is segmented into pixels using the Faster R-CNN model to obtain the contour sequence of the collapsed rocks. Spatial feature matching is performed between 3D point cloud data and contour sequences to obtain effective point cloud data; Based on effective point cloud data, the volume sequence, spatial coordinate sequence, and spatial attitude sequence of the collapsed rocks were calculated using a three-dimensional coordinate transformation algorithm.

3. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision according to claim 2, characterized in that, The process of calculating the volume sequence, spatial coordinate sequence, and spatial attitude sequence of collapsed rocks based on effective point cloud data using a three-dimensional coordinate transformation algorithm includes: The effective point cloud data is mapped to an effective voxel set. The volume of the collapsed rocks at each time point is calculated based on the effective voxel set. The volumes are then arranged into a volume sequence according to the time order. The effective point cloud data is converted to the world coordinate system to obtain the real point cloud data at each time point. The geometric centroid coordinates of the collapsed rocks at each time point are calculated based on the real point cloud data. The geometric centroid coordinates are used as the spatial coordinates of the collapsed rocks. The spatial coordinates are arranged into a spatial coordinate sequence according to the time order. Calculate the covariance matrix of the effective point cloud data at each time point, perform eigenvalue decomposition on the covariance matrix to obtain the major axis eigenvector, calculate the tilt angle based on the major axis eigenvector and the X-axis of the world coordinate system, determine the spatial attitude based on the tilt angle, and assemble the spatial attitudes into a spatial attitude sequence according to the time order.

4. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision according to claim 1, characterized in that, The hybrid temporal AI model includes a temporal convolutional network, a long short-term memory network, an attention mechanism fusion layer, and a fully connected layer.

5. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision according to claim 4, characterized in that, The method of predicting the trajectory and velocity sequence of falling rocks using a hybrid temporal AI model based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data includes: Local dynamic features are extracted from two-dimensional image data, thermal radiation data, and three-dimensional point cloud data using a temporal convolutional network to obtain temporal features of images, thermal radiation, and point clouds. These temporal features are then concatenated to form visual temporal features. Long short-term memory network is used to extract long sequence change features from spatial coordinate sequence and spatial attitude sequence to obtain coordinate temporal features and attitude temporal features. The coordinate temporal features and attitude temporal features are then concatenated to form spatial temporal features. The visual temporal features and spatial temporal features are weighted and fused through an attention mechanism fusion layer to obtain fused temporal features. The fused temporal features are then mapped to the motion trajectory and velocity sequence of the falling rocks through a fully connected layer.

6. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision according to claim 1, characterized in that, The kinetic energy calculation model includes a basic kinetic energy term, an air resistance correction term, a slope friction correction term, and a wind resistance correction term.

7. The method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision according to claim 6, characterized in that, The process of inputting environmental parameters, contour sequences, volume sequences, motion trajectories, and velocity sequences into the kinetic energy calculation model to calculate the impact energy of the collapsed rockfall includes: Environmental parameters include air resistance coefficient, air density, slope friction coefficient, slope angle, rock density, wind speed and wind direction. Based on the contour sequence, volume sequence and velocity sequence, the contour, volume and velocity of the collapsed rock at the last time point are obtained respectively. Based on the motion trajectory, the time interval and straight distance between the last time point and the previous time point of the collapsed rock are obtained. The mass is calculated based on the volume and rock density, and the initial kinetic energy is calculated by substituting the mass and velocity into the basic kinetic energy term. The windward area is obtained by calculating the profile. The air resistance coefficient, air density, windward area, speed and time interval are substituted into the air resistance correction term to calculate the air resistance kinetic energy loss. Substitute the slope friction coefficient, mass, straight distance and slope angle into the slope friction correction term to calculate the slope friction kinetic energy loss. The drag velocity component is calculated based on wind speed and wind direction. The air drag coefficient, air density, windward area, speed, drag velocity component and time interval are substituted into the wind resistance correction term to calculate the environmental wind kinetic energy loss. The total loss is the sum of the kinetic energy loss due to air resistance, the kinetic energy loss due to slope friction, and the kinetic energy loss due to ambient wind. The impact energy is obtained by subtracting the total loss from the initial kinetic energy.

8. A device for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision, used to implement the method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision as described in any one of claims 1 to 7, characterized in that, The device includes: The data acquisition module is used to collect multimodal visual data and environmental parameters of the landslide hazard area through a collaborative acquisition system. The multimodal visual data includes two-dimensional image data, thermal radiation data, and three-dimensional point cloud data. The first parameter calculation module is used to construct a recognition and localization model. Through the recognition and localization model, the module performs the recognition, contour extraction and spatial parameter calculation of the multimodal visual data of the landslide and falling rocks, and obtains the contour sequence, volume sequence, spatial coordinate sequence and spatial attitude sequence of the landslide and falling rocks. The second parameter calculation module is used to construct a hybrid temporal AI model. Based on spatial coordinate sequences, spatial attitude sequences, and multimodal visual data, the hybrid temporal AI model predicts the motion trajectory and velocity sequence of collapsed rocks. The impact energy calculation module is used to build a kinetic energy calculation model. Environmental parameters, contour sequences, volume sequences, motion trajectories, and velocity sequences are input into the kinetic energy calculation model to calculate the impact energy of the collapsed rocks. The monitoring result acquisition module is used to calculate the risk level of landslides based on the contour sequence, volume sequence, trajectory, velocity sequence and impact energy of the landslides.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for accurate identification and monitoring of landslides and falling rocks based on AI and machine vision as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI-based and machine vision-based method for accurate identification and monitoring of landslides and falling rocks as described in any one of claims 1 to 7.

Images