Clustering Method and Device for Unmanned Aerial Vehicle (UAV) Landslide Images for Emergency Monitoring

Abstract

This invention discloses a method and apparatus for clustering UAV landslide images for emergency monitoring, belonging to the field of landslide monitoring technology. The method includes: acquiring raw images captured by a camera mounted on a UAV; extracting multimodal features of landslide instances from the raw images, the multimodal features including spatial location features, global semantic embedding features, and local geometric matching features; fusing the multimodal features using an attention-based neural network model to generate instance feature vectors; constructing a graph structure representing the relationships between landslide instances based on the spatial location features, local geometric matching features, and instance feature vectors through an iterative process including a feedback mechanism; extracting connected components from the graph structure, and identifying each connected component as a landslide instance cluster. This invention can accurately associate different image instances belonging to the same landslide, reducing the error rate of cross-frame target matching in complex terrain.

Description

Technical Field

[0001] This invention belongs to the field of landslide monitoring technology, and in particular relates to a method and device for clustering UAV landslide images for emergency monitoring. Background Technology

[0002] Landslides are frequent in western my country, often causing severe damage. Post-disaster emergency investigations must prioritize speed and efficiency, providing accurate and comprehensive information in the shortest possible time. Unmanned aerial vehicles (UAVs), with their high mobility, flexibility, and lack of terrain limitations, have demonstrated significant potential in disaster emergency management. Currently, airborne real-time 3D semantic modeling provides a practical solution for the rapid extraction and representation of disaster targets, and also offers reliable data support for disaster early warning and emergency response.

[0003] However, achieving real-time airborne 3D reconstruction of disaster sites based on UAV imagery still faces significant challenges. On the one hand, UAV surveys typically generate a large amount of high-resolution imagery, while disaster targets often only appear in a small portion of these images. Traditional full-scene reconstruction methods are computationally too burdensome to meet the needs of real-time emergency applications. On the other hand, effective modeling of disaster targets requires not only geometric representation but also semantic description to support disaster identification and assessment.

[0004] In typical UAV reconnaissance, disasters usually appear only in a few images and occupy only a small portion of the surveyed area. Performing global 3D reconstruction on all images would not only incur excessive computational costs but also fail to meet real-time requirements. Therefore, it is essential to quickly detect and cluster a small number of images covering the same disaster target before performing local 3D reconstruction to achieve efficient emergency response.

[0005] Existing methods primarily rely on local feature matching, target-level feature retrieval, or fine-grained feature comparison. However, these methods either lack semantic understanding capabilities or incur high computational costs, making them unsuitable for real-time applications. More importantly, they generally neglect the spatial characteristics of UAV imagery, which limits the accurate and efficient association of disaster targets across multiple imagery images. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and device for clustering UAV landslide images for emergency monitoring.

[0007] This invention is achieved through the following technical solution:

[0008] The first aspect of this invention discloses a UAV landslide image clustering method for emergency monitoring, comprising:

[0009] Acquire raw images captured by a camera mounted on a drone;

[0010] Multimodal features of landslide instances are extracted from the original images, including spatial location features, global semantic embedding features, and local geometric matching features;

[0011] The multimodal features are fused using an attention-based neural network model to generate instance feature vectors;

[0012] Based on spatial location features, local geometric matching features, and instance feature vectors, a graph structure representing the relationship between landslide instances is constructed through an iterative process including a feedback mechanism.

[0013] Extract the connected components from the graph structure and identify each connected component as a landslide instance cluster.

[0014] Furthermore, multimodal features of landslide instances are extracted from the original images, including:

[0015] Extract spatial location features, including latitude and longitude, elevation, pitch angle, yaw angle, and roll angle extracted from the metadata of the original image;

[0016] Detect the oriented bounding boxes of landslide instances in the original image;

[0017] Based on the directional bounding box, the instance region image is cropped from the original image, and the instance region image is scaled to a preset size;

[0018] Extract global semantic embedding features, which include global semantic embedding feature vectors extracted from the instance region image;

[0019] Local geometric matching features are extracted, including key points and descriptors extracted by multi-directional rotation sampling of the instance region image.

[0020] Furthermore, the directional bounding boxes of landslide instances in the original image are detected, including:

[0021] Landslide detection is performed on the original images to identify landslide instances within them.

[0022] Mark the polygonal outline of the landslide instance;

[0023] Calculate the bounding rectangle of the polygonal outline and determine the bounding box of the landslide instance.

[0024] Furthermore, the multimodal features are fused using an attention-based neural network model to generate a fused feature vector, including:

[0025] The latitude, longitude, elevation, pitch angle, yaw angle, and roll angle are encoded to generate a geo-attitude code.

[0026] Concatenate all descriptors in the local geometric matching feature to form the local geometric embedding feature vector;

[0027] Geographic-attitude encoding, local geometric embedding feature vectors, and global semantic embedding feature vectors are concatenated to form multi-scale feature vectors;

[0028] The multi-scale feature vectors are input into the Efficient Former module for cross-modal attention fusion to generate instance feature vectors.

[0029] Furthermore, latitude, longitude, elevation, pitch angle, yaw angle, and roll angle are encoded to generate geo-attitude codes, including:

[0030] The sine-cosine position coding method is used to encode latitude and longitude, elevation, pitch angle, yaw angle and roll angle to generate static basic codes;

[0031] Calculate the changes in pitch, yaw, and roll angles in the current frame compared to those in the previous frame, and map these changes into dynamic attitude compensation codes using a fully connected layer.

[0032] Static base coding and dynamic attitude compensation coding are combined to form geo-attitude coding.

[0033] Furthermore, based on spatial location features, local geometric matching features, and instance feature vectors, an iterative process incorporating a feedback mechanism is used to construct a graph structure representing the relationships between landslide instances, including:

[0034] Create an empty undirected graph where each node represents a landslide instance.

[0035] A KD-Tree is constructed based on the spatial location features of landslide instances. Nearest neighbor queries are performed with a determined K value to generate a set of candidate instance pairs.

[0036] The Faiss algorithm is used to calculate the similarity of instance feature vectors between candidate instance pairs, and candidate instance pairs with similarity less than the first threshold are eliminated.

[0037] The LightGlue algorithm is used to perform local geometric feature matching on candidate instance pairs. The proportion of the number of matched key points to the total number of key points is counted, and candidate instance pairs with a proportion less than the second threshold are removed.

[0038] The candidate instance pairs that pass the verification in this iteration are added as edges to the undirected graph.

[0039] Determine if the maximum number of iterations has been reached; if so, terminate the iteration.

[0040] If the maximum number of iterations has not been reached, the pass rate of candidate instance pairs when using the LightGlue algorithm for matching verification in this round of iteration is counted: if the pass rate is greater than or equal to the fourth threshold, the K value is increased and the iteration is repeated; if the pass rate is less than the fourth threshold, the iteration is terminated.

[0041] Furthermore, the LightGlue algorithm is used to perform local geometric feature matching on candidate instance pairs, including:

[0042] Combine the descriptors of multiple directions from two landslide instances;

[0043] The assembled descriptor is input into the pre-trained LightGlue model to calculate the similarity confidence score for each pair of feature points.

[0044] Keypoint pairs with a similarity confidence score greater than the third threshold are identified as matching keypoint pairs.

[0045] Furthermore, the method also includes:

[0046] Obtain time-series images of landslide instance clusters;

[0047] Based on the time-series images of the landslide instance clusters and the landslide instance clusters, the active point information of the landslide instances is determined;

[0048] Based on the active point information of the landslide instance, K-means clustering algorithm was used to obtain K clusters;

[0049] Based on the K clusters, determine the high-activity area information, medium-activity area information, and low-activity area information of the landslide instance;

[0050] Based on the time-series images of the landslide instances clustered together, the information on the highly active areas of the landslide instances, the information on the moderately active areas of the landslide instances, and the information on the low-activity areas of the landslide instances, multiple types of subsequent development videos of the landslide instances are generated.

[0051] Based on multiple subsequent development videos of the landslide instance, a graph neural network is used to determine the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance.

[0052] Furthermore, the determination of the target subsequent development video of the landslide instance and the estimated time of the next landslide based on the multiple types of subsequent development videos of the landslide instance using a graph neural network includes:

[0053] A landslide instance development map is constructed, which includes multiple subsequent development video nodes and multiple edges of the multiple subsequent development video nodes. Each subsequent development video node represents a class of subsequent development videos. The node features of each subsequent development video node are time series images of a class of subsequent development videos and landslide instance clusters. The edges between nodes represent the similarity between subsequent development videos.

[0054] The development map of the landslide instance is processed using a graph neural network to obtain the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance.

[0055] A second aspect of the present invention discloses a drone-based landslide image clustering device for emergency monitoring, comprising:

[0056] The image acquisition module is used to acquire raw images taken by the camera mounted on the drone.

[0057] The feature extraction module is used to extract multimodal features of landslide instances from the original image. The multimodal features include spatial location features, global semantic embedding features, and local geometric matching features.

[0058] The feature fusion module is used to fuse the multimodal features using an attention-based neural network model to generate instance feature vectors.

[0059] The undirected graph construction module is used to construct a graph structure representing the relationships between instances based on spatial location features, local geometric matching features, and instance feature vectors through an iterative process that includes a feedback mechanism.

[0060] The clustering generation module is used to extract connected components in the graph structure and identify each connected component as a landslide instance cluster.

[0061] Furthermore, the image acquisition module, feature extraction module, feature fusion module, undirected graph construction module, and cluster generation module are deployed on an edge computing device, which is mounted on a drone.

[0062] Furthermore, the edge computing device is a Jetson NX edge computing device, which is positioned on top of the drone and connected to the drone via an adapter board.

[0063] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0064] (1) This invention can accurately associate different image instances belonging to the same landslide through spatial, semantic and geometric three-level constraints and LightGlue feedback iteration mechanism, effectively overcome the interference of complex terrain and similar ground objects, and reduce the error rate of cross-frame target matching under complex terrain.

[0065] (2) This invention extracts rotational features in four directions and enhances and fuses spatial location features, global semantic embedding features and local geometric matching features. It has strong robustness to attitude jitter during UAV flight, ensuring that the same landslide can still be correctly identified and associated under different shooting angles, which significantly improves the stability and reliability of the features.

[0066] (3) This invention uses local reconstruction instead of global reconstruction, and only performs fine processing on the clustered landslide areas, thus avoiding the huge computational overhead of processing massive invalid images.

[0067] (4) In the clustering process, the present invention proposes a dynamic adjustment mechanism for the K value of KD-Tree, which can adaptively expand the search range according to the complexity of the actual scenario, without the need for manual preset of fixed parameters, and has a high degree of intelligence and good generalization ability for different tasks.

[0068] (5) Through the iterative clustering framework, this invention can realize the dynamic life cycle tracking of landslide instances, that is, to track the evolution process of the same landslide entity across multiple images, which provides the possibility for dynamic disaster assessment.

[0069] (6) The method of the present invention has low requirements for computing power and can be realized by using edge computing devices such as Jetson NX. It can shorten the preliminary survey cycle of traditional methods, which requires several days or even a week, to output a high-precision landslide model within 3-6 hours after the disaster, thus winning valuable golden time for emergency decision-making. Attached Figure Description

[0070] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0071] Figure 1 This is a flowchart of the UAV landslide image clustering method in this invention;

[0072] Figure 2 This is a schematic diagram of the method in this invention;

[0073] Figure 3 This is a schematic diagram of directional bounding box extraction in this invention;

[0074] Figure 4 This is a schematic diagram of clustering in this invention;

[0075] Figure 5 This is a flowchart of iterative clustering in this invention;

[0076] Figure 6A flowchart illustrating a process for determining the target subsequent development video of a landslide instance and the estimated time of the next landslide of the landslide instance, provided as an embodiment of the present invention.

[0077] Figure 7 This is a schematic diagram illustrating a process for determining the target subsequent development video of a landslide instance and the estimated time of the next landslide of the landslide instance using a graph neural network based on multiple types of subsequent development videos of the landslide instance, as provided in an embodiment of the present invention. Detailed Implementation

[0078] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0079] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0080] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0081] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0082] like Figures 1 to 7 As shown in the figure, this embodiment discloses a method and device for clustering UAV landslide images for emergency monitoring.

[0083] The first aspect of this embodiment discloses a UAV landslide image clustering method for emergency monitoring, such as... Figure 1 and Figure 2As shown, the UAV landslide image clustering method includes steps S100 to S500.

[0084] Step S100. Acquire the raw images captured by the camera mounted on the drone.

[0085] For example, the camera used is a DJI L1 LiDAR camera.

[0086] In this embodiment, the rate at which the camera captures the original image is determined based on the camera's performance and shooting requirements.

[0087] In some embodiments of this example, the parameters of the drone and camera are set as follows when capturing the original images: the drone's flight mode is terrain-following flight, the flight altitude is 350-500 meters, and the heading overlap is 80%; the camera's shooting rate is 2 seconds per frame.

[0088] Step S200. Extract multimodal features of landslide instances from the original image, wherein the multimodal features include spatial location features, global semantic embedding features, and local geometric matching features.

[0089] In some embodiments of this example, multimodal features of landslide instances are extracted from the original image, including steps S210 to S250.

[0090] Step S210. Extract spatial location features, which include latitude and longitude, elevation, pitch angle, yaw angle and roll angle extracted from the metadata of the original image.

[0091] For example, the latitude, longitude, elevation, and UAV attitude angles (pitch, yaw, and roll) of each frame can be directly read from the metadata (such as EXIF ​​information) of the original image.

[0092] Step S220. Detect the directional bounding boxes of landslide instances in the original image.

[0093] In some implementations of this embodiment, such as Figure 3 As shown, detecting the oriented bounding box of landslide instances in the original image includes: performing landslide detection on the original image to identify landslide instances in the original image; marking the polygonal outline of the landslide instance; calculating the bounding rectangle of the polygonal outline, and determining the bounding rectangle as the oriented bounding box of the landslide instance.

[0094] Step S230. Based on the directional bounding box, crop the instance region image from the original image and scale the instance region image to a preset size.

[0095] The example region image is an image region surrounded by a directional bounding box.

[0096] For example, the warp_and_pad_bbox function can be used to scale the instance region image to 512x512 pixels.

[0097] In this embodiment, the instance region image is scaled to a uniform size to provide standardized input for subsequent processing, ensuring the stability and comparability of feature extraction.

[0098] Step S240. Extract global semantic embedding features, wherein the global semantic embedding features include global semantic embedding feature vectors extracted from the instance region image.

[0099] For example, input an instance region image (512*512 pixels) into the YOLOv11l-cls model, extract an original deep feature vector from the output of a specific network layer (such as the 8th layer) of the model, and standardize the deep feature vector to obtain a 512-dimensional global semantic embedding feature vector.

[0100] In these implementations, the global semantic embedding feature vector encodes the overall visual appearance and high-level semantic information (such as texture, color, and overall shape) of the landslide instance.

[0101] Step S250. Extract local geometric matching features, which include key points and descriptors extracted by multi-directional rotation sampling of the instance region image.

[0102] For example, input an instance region image (512*512 pixels) into an ONNX format local feature extractor (such as DISK or SuperPoint), perform rotation sampling on the image region in four directions (0°, 90°, 180°, 270°), and extract the key point positions and corresponding 128-dimensional descriptors for each direction of the image.

[0103] This embodiment enhances the robustness (rotation invariance) to changes in the attitude of the UAV by performing rotational sampling of the image in multiple directions.

[0104] Step S300. The multimodal features are fused using an attention-based neural network model to generate an instance feature vector.

[0105] In some embodiments of this example, the multimodal features are fused using an attention-based neural network model to generate a fused feature vector, including steps S310 to S340.

[0106] Step S310. Encode the latitude, longitude, elevation, pitch angle, yaw angle, and roll angle to generate a geo-attitude code.

[0107] In some embodiments of this example, latitude and longitude, elevation, pitch angle, yaw angle and roll angle are encoded to generate geographic-attitude codes, including steps S311 to S313.

[0108] Step S311. Use the sine-cosine position coding method to encode latitude and longitude, elevation, pitch angle, yaw angle and roll angle to generate static basic code.

[0109] For example, for each dimension of data—latitude / longitude, elevation, pitch angle, yaw angle, and roll angle—16 different frequencies are generated. The formula for calculating the frequency is: freq = 10000^(2k / 32), (k = 0, 1, ..., 15), where freq represents the frequency. The sine and cosine values ​​for each of the latitude / longitude, elevation, pitch angle, yaw angle, and roll angle at different frequencies are calculated. The formula for calculating the sine value is: sine = sin(base data × freq), and the formula for calculating the cosine value is: cos = cos(base data × freq). The calculated sine and cosine values ​​are concatenated to form a 6 * (16 + 16) = 192-dimensional encoding vector, which is the static base encoding.

[0110] Step S312. Calculate the changes in pitch angle, yaw angle and roll angle in the current frame compared with the pitch angle, yaw angle and roll angle in the previous frame, and use a fully connected layer to map the changes into dynamic attitude compensation codes.

[0111] For example, the changes in pitch angle in the current frame compared to the previous frame are calculated, as are the changes in yaw angle and roll angle in the current frame and the previous frame. These changes are then mapped to an 81-dimensional dynamic compensation vector through a fully connected layer. This dynamic compensation vector is the dynamic attitude compensation code. The fully connected layer has a 3-dimensional input and an 81-dimensional output, using the ReLU activation function.

[0112] In these implementations, dynamic attitude compensation coding is introduced, which can effectively capture the motion jitter of the UAV between consecutive frames.

[0113] Step S313. Static base coding and dynamic attitude compensation coding are concatenated to form geo-attitude coding.

[0114] For example, a 273-dimensional geo-attitude code is obtained by concatenating a 192-dimensional static base code and an 81-dimensional dynamic compensation code.

[0115] Step S320. Concatenate all descriptors in the local geometric matching features to form a local geometric embedding feature vector.

[0116] For example, assuming that 128-dimensional descriptors are extracted in each of the four directions, all the descriptors in the four directions of the local geometric matching features are concatenated to form a 512-dimensional local geometric embedding feature vector.

[0117] Step S330. Concatenate the geo-pose encoding, local geometric embedding feature vector, and global semantic embedding feature vector to form a multi-scale feature vector.

[0118] For example, a 1297-dimensional multi-scale feature vector can be formed by concatenating a 273-dimensional geo-pose encoding, a 512-dimensional local geometric embedding feature vector, and a 512-dimensional global semantic embedding feature vector.

[0119] Step S340. Input the multi-scale feature vector into the Efficient Former module for cross-modal attention fusion to generate instance feature vectors.

[0120] For example, the multi-scale feature vector is input into the Efficient Former module. The Efficient Former module learns and calculates the attention weights of three parts of features—geo-pose encoding, local geometric embedding feature vector, and global semantic embedding feature vector—through its multi-head attention mechanism, dynamically weights and fuses them, and outputs a unified, enhanced 1024-dimensional instance feature vector.

[0121] In some implementations of this embodiment, instance feature vectors, key points, descriptors, and spatial coordinates are packaged into a list of landslide_data.

[0122] Step S400. Based on spatial location features, local geometric matching features, and instance feature vectors, construct a graph structure representing the relationship between landslide instances through an iterative process including a feedback mechanism.

[0123] In some implementations of this embodiment, such as Figure 4 and Figure 5 As shown, based on spatial location features, local geometric matching features, and instance feature vectors, a graph structure representing the relationship between landslide instances is constructed through an iterative process including a feedback mechanism, including steps S410 to S470.

[0124] Step S410. Create an empty undirected graph where each node represents a landslide instance.

[0125] For example, use the NetworkX library to create an empty undirected graph.

[0126] Step S420. Construct a KD-Tree based on the spatial location features of landslide instances, perform nearest neighbor queries with a determined K value, and generate a set of candidate instance pairs.

[0127] For example, the initial value of K is set to 30.

[0128] For example, for each landslide instance i, use the KD-Tree to query its K nearest neighbor landslide instances j in space, and each (i, j) combination forms a candidate instance pair.

[0129] This step utilizes spatial proximity for spatial constraints, reducing the computational cost of subsequent fine-grained matching.

[0130] Step S430. Calculate the similarity of instance feature vectors between candidate instance pairs using the Faiss algorithm, and remove candidate instance pairs with similarity less than the first threshold.

[0131] For example, for each candidate instance pair (i, j), the cosine similarity of the feature vectors of the two landslide instances is calculated using the Faiss index. If the cosine similarity is lower than the first threshold (e.g., the first threshold is 0.7), the candidate instance pair is discarded.

[0132] In these implementations, the fused instance feature vectors are used for embedding constraints to retain high-quality candidate instance pairs that are spatially and semantically similar, thereby further simplifying the candidate set.

[0133] Step S440. Use the LightGlue algorithm to perform local geometric feature matching on candidate instance pairs, count the proportion of the number of matched key points to the total number of key points, and remove candidate instance pairs whose proportion is less than the second threshold.

[0134] In these implementations, the LightGlue algorithm is used to perform geometric verification on candidate instance pairs.

[0135] For example, the second threshold is set to 20%.

[0136] In some implementations of this embodiment, the LightGlue algorithm is used to perform local geometric feature matching on candidate instance pairs, including: combining descriptors of multiple directions of two landslide instances; inputting the assembled descriptors into a pre-trained LightGlue model to calculate the similarity confidence score of each feature point pair; and determining key point pairs with similarity confidence scores greater than a third threshold as matching key point pairs.

[0137] For example, the third threshold is set to 0.7.

[0138] In these implementations, for each landslide instance with four rotation direction descriptors, the LightGlue algorithm automatically tries all possible direction combinations (4x4=16 in total) in a batch manner, and selects the direction with the highest matching score for calculation.

[0139] Step S450. Add the verified candidate instance pairs from this iteration as edges to the undirected graph.

[0140] Specifically, candidate instance pairs that pass spatial constraints, embedding constraints, and geometric verification in this iteration are added as edges to the undirected graph.

[0141] Step S460. Determine whether the maximum number of iterations has been reached. If yes, terminate the iteration; otherwise, proceed to step S470.

[0142] For example, the maximum number of iterations is set to 3.

[0143] Step S470. Calculate the pass rate of candidate instance pairs when using the LightGlue algorithm for matching verification in this round of iteration: if the pass rate is greater than or equal to the fourth threshold, increase the K value, and then return to step S420 to start a new round of iteration; if the pass rate is less than the fourth threshold, terminate the iteration.

[0144] For example, the fourth threshold is set to 80%.

[0145] For example, when increasing the value of K, increase the value of K by 10%, adjust the value of K and perform another round of verification, and add the newly passed edges to the undirected graph.

[0146] Step S500. Extract the connected components in the graph structure and identify each connected component as a landslide instance cluster.

[0147] For example, NetworkX can be used to extract the connected components of an undirected graph. Each connected component represents a set of interconnected internal instances, i.e., an independent landslide instance cluster.

[0148] In some embodiments of this example, the UAV landslide image clustering method further includes: filtering out landslide entity clusters with a number of landslide instances less than a fifth threshold.

[0149] In these implementations, by setting a minimum cluster size, tiny clusters (e.g., clusters with fewer than 5 landslide instances) are filtered out and treated as noise or false detections.

[0150] In some embodiments of this example, the UAV landslide image clustering method further includes: creating an independent folder for each landslide instance cluster, saving the original image, mask image, etc., generating a cluster thumbnail (based on the largest instance area), and outputting a CSV file to record the cluster members. The mask region is drawn using cv2.fillPoly with an opacity of 0.2 and overlaid on the original image.

[0151] In some embodiments of this example, the UAV landslide image clustering method further includes fusing the clustering results with the DOM and 3D model. An independent folder can be generated for each landslide class, containing the DOM, masked DOM, and an identified 3D model, clearly marking the landslide location.

[0152] The first aspect of this embodiment discloses a UAV landslide image clustering method for emergency monitoring, such as... Figure 6 As shown, the method further includes steps S600 to S1100.

[0153] In some embodiments, Figure 6 This invention provides a flowchart illustrating the process of determining a target subsequent development video of a landslide instance and the estimated time of the next landslide of the landslide instance. The determination of the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance includes steps S600 to S1100:

[0154] Step S600: Obtain time-series images of landslide instance clusters.

[0155] The time-series images of landslide instance clustering are a collection of images taken at different time points by a camera mounted on a drone and arranged in ascending order of timestamps for each independent landslide instance cluster.

[0156] Time-series images of landslide instance clustering can record the spatial morphology, surface features, and deformation information of the same landslide body at different stages.

[0157] Step S700: Determine the active point information of landslide instances based on the time-series images of the landslide instance clusters and the landslide instance clusters.

[0158] In some embodiments, an active point analysis model can be used to determine active point information of landslide instances. The active point analysis model is a recurrent neural network model. The input to the active point analysis model is the time-series image of the landslide instance cluster and the landslide instance cluster; the output of the active point analysis model is the active point information of the landslide instances.

[0159] Recurrent Neural Network (RNN) models are a type of neural network model that excels at processing time-series data. RNNs contain feedback connections and can utilize internal memory states to store historical data information, thereby capturing the dependencies of data over time. RNNs process each data point in the time series stepwise, combining the current input with historical states for computation. RNNs are suitable for extracting dynamic features from continuous time-series data and performing analysis and prediction.

[0160] The active point information of landslide instances is the specific location information of points exhibiting significant movement or deformation characteristics identified through the active point analysis model. This information includes spatial location information, temporal dynamic change information, and physical morphological characteristic information of the active points.

[0161] Spatial location information includes latitude and longitude, elevation, distance from the toe of the slope, distance from the centerline of the slip zone, slope aspect, and slope gradient.

[0162] The time-series dynamic change information includes cumulative displacement, instantaneous displacement rate, displacement direction, first active time, active duration, and displacement acceleration.

[0163] Physical morphological characteristics include crack length, width, aperture variation, crack direction, as well as vegetation cover, surface roughness, and soil / rock type.

[0164] The time-series imagery of landslide instance clustering fully records the surface state changes of the same landslide body at different times, including dynamic evolution information of pixels or sub-regions on the landslide body surface, such as displacement, crack development, and vegetation changes. Landslide instance clustering then clarifies the specific landslide entity range corresponding to the time-series imagery.

[0165] Recurrent Neural Networks (RNNs) possess a core advantage in processing time-series data. The internal feedback connections of an RNN form a memory mechanism, effectively capturing the dynamic correlation features across frames in time-series images of landslide instance clusters. For the input time-series image, the RNN can extract pixel-level features within the landslide instance cluster frame by frame, and combine this with the landslide entity boundaries defined by the clusters to eliminate interference from irrelevant areas. The RNN can utilize its memory state to integrate deformation data from historical frames and compare differences in location, displacement, and physical morphology at different timestamps, thereby accurately identifying points exhibiting significant movement or deformation. By learning deformation patterns along the temporal dimension, the RNN can quantify the displacement rate, crack propagation, and other dynamic features of active points in each landslide instance, ultimately outputting information on active landslide instance points that includes spatial location, temporal dynamics, and physical morphology.

[0166] In some embodiments, determining the active point information of landslide instances based on the time-series images and landslide instance clustering includes steps S710~S730:

[0167] Step S710: Based on the time-series images of the landslide instance clustering and the landslide instance clustering, determine information on multiple significant moving points of the landslide body, a vegetation coverage decay rate distribution map, and information on multiple crack areas.

[0168] In some embodiments, recurrent neural networks can be used to determine information on multiple significant movement points of the landslide body, a distribution map of vegetation cover decay rate, and information on multiple crack zones.

[0169] Multiple significant movement points are a set of all specific points whose displacement features, output by a recurrent neural network, exceed a set baseline value. The information of significant movement points includes the location coordinates, cumulative displacement, peak displacement, and displacement increment per unit time for each point.

[0170] The vegetation coverage decay rate distribution map is a graphical representation of the rate of vegetation coverage decline over time at various locations within the landslide area, obtained through the output of a recurrent neural network.

[0171] The vegetation cover decay rate distribution map can use different values ​​and colors to indicate the differences in decay rate at different locations.

[0172] Multiple crack zone information is a dataset of independent regions composed of multiple cracks within the landslide area, output by a recurrent neural network. Crack zone information includes the boundary extent of each zone, the coordinates of the crack's origin and end points, its extension length, and its direction of cracking.

[0173] Recurrent neural networks (RNNs) can leverage their temporal memory capabilities to correlate and analyze the features of consecutive frames in time-series images of landslide instance clusters. Combined with spatial boundary constraints of landslide instance clusters, this allows for the efficient extraction of multiple salient movement points within the landslide body, vegetation cover decay rate distribution maps, and multiple crack area information. The RNN can continuously track pixel shifts at various locations within the landslide body and, after accumulating and judging displacement trends, can filter out points whose displacement characteristics exceed a set benchmark value. It then integrates the core data of these points to form multiple salient movement points within the landslide body. Simultaneously, the RNN can model the temporal changes in the vegetation spectrum of each pixel and then quantify the rate of vegetation cover decline to generate a vegetation cover decay rate distribution map. Furthermore, the RNN can capture dynamic changes in crack grayscale texture and, combined with spatial correlation, divide the cracks into independent areas, thereby extracting data such as area boundaries and crack morphology to output multiple crack area information.

[0174] Step S720: Based on the information of multiple significant movement points of the landslide body, the vegetation coverage decay rate distribution map, and the information of multiple crack areas, determine the aggregate displacement risk level of each significant movement point, the degree of influence of each significant movement point on vegetation coverage decay, the development risk level of each crack area, and the key diffusion point information of the crack area.

[0175] In some embodiments, deep neural networks can be used to determine the aggregate displacement risk level of each significant movement point, the degree of impact of each significant movement point on vegetation cover attenuation, the developmental risk level of each crack zone, and the key diffusion point information of the crack zone.

[0176] Deep neural networks (DNNs) are artificial intelligence models composed of multiple layers of neurons. They can simulate the information processing methods of the human brain by stacking hidden layers. Through the adjustment of weights and signal transmission between neurons in each layer, DNNs can efficiently uncover potential correlations between multi-source data and accurately capture complex patterns in the data, thereby enabling various tasks such as classification, regression, and prediction.

[0177] The aggregate displacement risk level of each significant movement point is a numerical indicator of the potential threat to landslide stability posed by the displacement behavior of each significant movement point, determined by a deep neural network.

[0178] The degree of influence of each significant movement point on vegetation cover reduction is a numerical index determined by a deep neural network, which measures the intensity of the effect of the displacement behavior of each significant movement point on the surrounding vegetation cover reduction process.

[0179] The development risk level of each crack zone is a numerical indicator of the potential hazard to landslide stability caused by the crack expansion trend of each crack zone, determined by a deep neural network.

[0180] The key diffusion point information of the crack area is the data of crack initiation points with the potential to further expand to the surrounding areas, determined by a deep neural network. The key diffusion point information of the crack area includes the specific location coordinates of each key diffusion point, the expected expansion direction, the expected extension length per unit time, and the radius of influence of the expansion.

[0181] Deep neural networks possess multi-level feature extraction and nonlinear correlation modeling capabilities. Through stacked hidden layers, they can capture the changing characteristics of location coordinates and displacement parameters from multiple significant moving points within a landslide body; extract the spatial distribution patterns of decay rates in different regions from vegetation cover decay rate distribution maps; and mine key features of crack morphology and extension trajectories from multiple crack areas. Deep neural networks can deeply fuse features from multiple significant moving points, vegetation cover decay rate distribution maps, and multiple crack areas through fully connected layers. Simultaneously, they autonomously learn the quantitative correlation between the displacement of significant moving points and vegetation cover decay, and the intrinsic laws governing crack development and the displacement of surrounding points. Ultimately, through parameter optimization, deep neural networks can achieve precise mapping from input data to the aggregated displacement risk level of each significant moving point, the degree of influence of each significant moving point on vegetation cover decay, the development risk level of each crack area, and key diffusion point information within the crack area.

[0182] Step S730: Based on the aggregate displacement risk level of each significant moving point, the degree of influence of each significant moving point on vegetation cover attenuation, the development risk level of each crack area, and the key diffusion point information of the crack area, determine the active point information of the landslide instance.

[0183] In some embodiments, deep neural networks can be used to determine active point information of landslide instances.

[0184] Deep neural networks can standardize input information, transforming the aggregate displacement risk level of each significant movement point, the impact of each significant movement point on vegetation cover attenuation, and the developmental risk level of each crack area into numerical features of a unified dimension. Simultaneously, deep neural networks can extract the spatial correlation data between point coordinates and crack area locations from key diffusion point information within the crack area. Through an attention mechanism, deep neural networks can assign dynamic weights to each feature, emphasizing the weights of spatial correlation features corresponding to high aggregate displacement risk levels, strong vegetation cover attenuation impacts, and high developmental risk crack areas. Then, the model can fuse all weighted features through fully connected layers, uncovering the inherent coupling patterns between features. Based on the judgment logic learned during training, deep neural networks can filter out core points whose comprehensive feature scores exceed a set threshold. These points collectively represent the core characteristics of landslide activity, ultimately forming information on active landslide points.

[0185] Step S800: Based on the active point information of the landslide instance, K-means clustering algorithm is used to obtain K clusters.

[0186] K-means clustering is a commonly used unsupervised clustering algorithm. Its core idea is to divide the dataset into K predetermined clusters, ensuring high similarity among data points within each cluster and low similarity between data points in different clusters. The value of K can be pre-set manually. K-means clustering iteratively updates the cluster centers until they no longer change significantly or the predetermined number of iterations is reached, ultimately achieving data clustering.

[0187] The K clusters are K independent sets of active points obtained by clustering the active point information of landslide instances using the K-means clustering algorithm. Each cluster can represent a type of active point of landslide instances with similar active characteristics, such as similarity in spatial location, temporal dynamic changes, physical morphology, etc., while active points in different clusters show significant differences in dynamic change characteristics.

[0188] For example, clustering the active points information of a landslide instance yields three clusters. Cluster 1 contains all active points located at the leading edge of the landslide body, with fast displacement rates and significant changes in crack opening. Cluster 2 contains active points located in the middle of the landslide body, with moderate displacement rates and slow vegetation cover degradation. Cluster 3 contains active points located at the trailing edge of the landslide body, with slow displacement rates and no obvious crack development.

[0189] The process of clustering active points of landslide instances using the K-means clustering algorithm is as follows: First, K points are randomly selected from the active point data set of landslide instances as initial cluster centers. Next, for each active point in the dataset, Euclidean distance is used to measure and calculate its distance to these K initial cluster centers, and the active point is assigned to the corresponding cluster according to the principle of closest proximity. After all active points have been partitioned, the average values ​​of various features of the active points within each cluster are recalculated to update the cluster centers of each cluster. This process of partitioning and updating cluster centers is repeated until the changes in cluster centers are minimal. At this point, the clustering process is considered to have converged, thus completing K-means clustering.

[0190] K-means clustering can effectively integrate complex information on active landslide points. The sheer number and diverse characteristics of these active landslide points make direct analysis highly complex. K-means clustering groups active points with similar characteristics into K clusters, significantly simplifying the data structure and facilitating the extraction of valuable information on active landslide areas. Dividing the active landslide point information into K clusters provides a clear visual representation of the distribution of areas with varying levels of activity within the landslide body.

[0191] Step S900: Based on the K clusters, determine the high-activity area information, medium-activity area information, and low-activity area information of the landslide instance.

[0192] In some embodiments, an active region analysis model can be used to determine the highly active region information, moderately active region information, and inactive region information of a landslide instance. The active region analysis model is a Transformer model. The input to the active region analysis model is the K clusters, and the output of the active region analysis model is the highly active region information, moderately active region information, and inactive region information of the landslide instance.

[0193] The Transformer model is a neural network architecture based on self-attention. It consists of two core components: an encoder and a decoder. Through self-attention, the Transformer model can compute the relationships between elements in the input sequence in parallel, regardless of their distance within the sequence. It can capture global contextual information and focus on different features of the data from different subspaces through multi-head attention.

[0194] The highly active area information of landslide instances is the information of the most active areas on the landslide body determined by analyzing K clusters using an active area analysis model.

[0195] The information on the moderately active areas in the landslide instance is determined by analyzing K clusters using an active area analysis model, identifying areas on the landslide body with moderate activity.

[0196] The low-activity area information of the landslide instance is the area information with a low activity level on the landslide body determined by analyzing K clusters using the active area analysis model.

[0197] Regional information includes spatial boundary information, quantitative feature information, spatial distribution information, and temporal evolution information of the corresponding region.

[0198] Spatial boundary information includes the region's latitude and longitude range, elevation interval, and complete regional outline coordinates.

[0199] The quantitative feature information includes the average displacement rate of active points in the region, the average rate of change of crack opening, and the average vegetation cover decay.

[0200] Spatial distribution information includes the range of distances from the toe of the slope and the center line of the slip zone, as well as the slope aspect and slope range of the location.

[0201] The temporal evolution information includes the duration of regional activity and the mean displacement acceleration.

[0202] The K clusters are sets of active points from landslide instances with similar activity levels. The active points within each cluster share consistency in core characteristics such as displacement rate and crack propagation rate, directly reflecting the differences in activity levels across different areas of the landslide body. These clusters have completed the initial grouping of active points, providing accurate foundational data for further delineating active areas of different levels. This allows the active area analysis model to clearly distinguish areas of different activity levels based on the characteristic differences of the clusters.

[0203] The Transformer model can extract features from K input clusters in parallel. Through the encoder's self-attention mechanism, the model can simultaneously focus on various feature data of active points in all landslide instances within each cluster, calculating statistical values ​​for each cluster on core quantitative features such as average displacement rate, average crack aperture change rate, and average vegetation cover decay. It also integrates the spatial distribution patterns and temporal evolution trends of active points within the cluster to form a comprehensive characterization of the overall activity state of each cluster. Next, the model uses a multi-head attention mechanism to perform cross-cluster comparative analysis of features from all clusters, thereby mining the differences in activity levels between clusters from different feature subspaces and establishing a unified activity level assessment dimension. Subsequently, the Transformer model can match the feature statistics of each cluster one by one according to preset activity level judgment criteria, defining the region corresponding to the cluster that meets the high-activity level feature criteria as a high-activity region. The spatial coordinates of all active points within this region are extracted and integrated into spatial boundary information. Combined with the cluster's quantitative features, spatial distribution, and temporal evolution statistics, complete information on the high-activity region of the landslide instance is formed. Similarly, the model can classify clusters that meet the criteria for medium-activity level and low-activity level as medium-active and low-activity regions, respectively, and extract and integrate the medium-active and low-activity region information to form landslide instances. If multiple clusters meet the same activity level criterion, the model will spatially merge the regions corresponding to these clusters.

[0204] Step S1000: Generate multiple types of subsequent development videos of landslide instances based on the time-series images of the landslide instance clusters, the high-activity area information of the landslide instance, the medium-activity area information of the landslide instance, and the low-activity area information of the landslide instance.

[0205] In some embodiments, a landslide evolution inference model can be used to generate multiple types of subsequent development videos of landslide instances. The landslide evolution inference model is a variational autoencoder. The inputs to the landslide evolution inference model are time-series images of the landslide instance clusters, information on highly active areas of the landslide instance, information on moderately active areas of the landslide instance, and information on low-activity areas of the landslide instance. The output of the landslide evolution inference model is multiple types of subsequent development videos of the landslide instance.

[0206] Variational autoencoders (VAEs) are generative neural network models based on probabilistic models. A VAE consists of an encoder and a decoder. The encoder maps input data to the probability distribution space of latent variables and then encodes the data by learning the latent feature distribution. The decoder generates new data similar to the input data based on the probability distribution of the latent variables. VAEs can capture the inherent patterns and features of input data and generate new data with reasonable logic and feature consistency from existing data.

[0207] The multiple subsequent development videos of landslide instances are a collection of videos generated by a landslide evolution simulation model, reflecting the future development state of the landslide body under different preset scenarios.

[0208] The preset scenarios include no rainfall scenario, moderate rainfall scenario, heavy rainfall scenario, earthquake impact scenario, and human engineering activity impact scenario.

[0209] The various follow-up development videos of landslide examples are arranged in chronological order and can dynamically show the surface morphology changes, active area evolution, and characteristic changes of the landslide body in the corresponding scenarios.

[0210] The time-series imagery of landslide instance clusters provides dynamic change data on the history of landslides, including the temporal patterns and characteristic trends of landslide evolution. Information on highly active, moderately active, and inactive areas of landslide instances clarifies the activity levels and change characteristics of different regions within the landslide body. This information collectively constitutes a complete data foundation for landslide evolution.

[0211] Variational autoencoders can extract features frame-by-frame from time-series images of landslide instance clusters, acquiring surface morphology, displacement changes, and crack development characteristics of the landslide body at different historical periods. By combining information from highly active, moderately active, and inactive regions, the evolutionary patterns of different active regions can be clarified, such as the acceleration trend of displacement in highly active regions and the stable change patterns in moderately active regions. The encoder of the model maps the input data to the probability distribution space of latent variables to learn the probability distribution patterns of landslide evolution, including the correlation of change rates in different active regions and the weight of environmental factors on landslide evolution. Then, based on different preset scenarios, the model can probabilistically sample and predict the future evolutionary state of the landslide body in the latent variable space to generate latent feature sequences for the corresponding scenarios. The decoder decodes these latent feature sequences into continuous video frames. Each video frame contains the surface morphology of the landslide body at the corresponding time point, and the annotation of active regions. Highly active regions are marked in red, moderately active regions in yellow, and inactive regions in green, along with parameters such as displacement and crack length for each region. Finally, the model can combine these video frames in chronological order to form multiple types of subsequent development videos with a fixed frame rate, resolution consistent with the original image, and duration covering a certain period in the future. Each type of video corresponds to a landslide evolution trend under a preset scenario.

[0212] Step S1100: Based on the multiple subsequent development videos of the landslide instance, a graph neural network is used to determine the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance.

[0213] In some embodiments, Figure 7This invention provides a schematic flowchart illustrating how a graph neural network is used to determine the target subsequent development video of a landslide instance and the estimated time of its next landslide based on multiple subsequent development videos of the landslide instance. The steps of determining the target subsequent development video and the estimated time of the next landslide based on multiple subsequent development videos of the landslide instance include steps S1110 to S1120:

[0214] Step S1110: Construct a landslide instance development map. The landslide instance development map includes multiple subsequent development video nodes and multiple edges of the multiple subsequent development video nodes. Each subsequent development video node represents a class of subsequent development videos. The node features of each subsequent development video node are time series images of a class of subsequent development videos and landslide instance clusters. The edges between nodes represent the similarity between subsequent development videos.

[0215] The landslide instance development map is a structured graph dataset used to represent different landslide evolution paths and their interrelationships. It consists of nodes and edges and can transform unstructured video data into a form suitable for graph computation.

[0216] The subsequent development video node is the basic unit in the landslide instance development map. Each subsequent development video node represents a possible future evolution scenario of the landslide.

[0217] The node features of subsequent development video nodes include a type of subsequent development video in the corresponding scenario and time series images of landslide instance clusters. The node features can represent the historical evolution pattern and future development trend of this evolution path.

[0218] The edges between nodes represent the degree of similarity in visual appearance and physical features between different evolutionary scenes, and the weight of the edges reflects the degree of similarity between two subsequent development videos.

[0219] In some embodiments, deep neural networks can be used to determine the similarity between subsequent development videos.

[0220] Step S1120: Process the development map of the landslide instance based on the graph neural network to obtain the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance.

[0221] Graph Neural Networks (GNNs) are deep learning models that operate directly on graph data. GNNs operate through a message-passing mechanism, allowing nodes in the graph to aggregate information from their neighbors. In each layer of the network, nodes can collect features from connected nodes and update their data by combining these features with their own. This mechanism enables GNNs to capture the complex topology and dependencies between nodes in the graph. GNNs can evaluate the importance and likelihood of each node in the graph from a global perspective. The input to the GNN is the landslide instance development graph, and the output is a video showing the subsequent development of the landslide instance and the estimated time of the next landslide.

[0222] The target follow-up development video of the landslide instance is a landslide evolution video that is output by deep analysis and pattern mining of the landslide instance development map through graph neural network. It is the video with the highest probability of occurrence and best conforms to the current geological laws, environmental conditions and landslide evolution trend.

[0223] The estimated time of the next landslide in the landslide instance is determined by graph neural network analysis, which identifies the specific time point at which the landslide is expected to cause catastrophic damage or significant displacement.

[0224] By constructing a landslide instance development map, the correlations and evolutionary differences between different subsequent development videos can be clearly reflected. This correlation information is crucial for accurately determining the target subsequent development video of a landslide instance and predicting the next landslide time, because the future evolution trend of a landslide is often influenced by multiple scene factors and has potential correlation patterns. Using the time-series images of each type of subsequent development video and landslide instance cluster as node features of subsequent development video nodes, and using the similarity between subsequent development videos as edge features, can more fully utilize multi-source monitoring and extrapolation data. This helps graph neural networks to understand more deeply the internal logic and mutual influence of different evolution paths, thereby improving the accuracy of target evolution trend judgment and time prediction. Processing landslide instance development map data based on graph neural network models can effectively learn the complex correlations and information transmission between subsequent development video nodes, thereby more accurately uncovering the core evolutionary laws and key triggering conditions of landslides. Compared with traditional time-series prediction methods, graph neural networks have stronger feature representation and pattern learning capabilities when processing correlation-type data such as landslide instance development maps, and can better adapt to the complexity and uncertainty of landslide evolution.

[0225] Graph Neural Networks (GNNs) initialize features for each subsequent development video node in the landslide instance development graph, and deeply fuse the subsequent development video features in the node features with the time-series image features of landslide instance clustering to form an initial node embedding vector. Then, the GNN iteratively updates the node embedding vector through graph convolutional layers. In each convolutional operation, each node aggregates the embedding vectors of its neighboring nodes and adjusts the contribution ratio of neighboring node features based on edge similarity weights. The higher the similarity weight of a neighboring node, the more significantly its node features contribute to the update of the current node. Through multiple layers of graph convolutional operations, the node embedding vector continuously integrates its core features with the correlation information of the global graph, and accurately captures the deep evolutionary correlations and pattern differences between subsequent development videos under different preset scenarios. Subsequently, the GNN uses fully connected layers to map the updated node embedding vector into two core input types. The model outputs a set of target subsequent development videos, including the probability value of occurrence for each node's corresponding scenario-based subsequent development video, as well as the predicted time of landslide occurrence for the corresponding scenario. Based on the actual geological conditions and environmental realities monitored, such as recent rainfall trends, soil and rock moisture content, and the real-time evolution status of high, medium, and low-activity areas of the landslide body, the model cross-validates the probability values ​​and evolutionary logic of all scenarios. The model considers both the objective probability of each scenario and the fit between the actual activity characteristics of the current landslide body and the scenario, ultimately integrating them into a multi-scenario fusion of target subsequent development videos. This target subsequent development video can cover the most likely evolutionary segments under each preset scenario. The model uses regression analysis to analyze the temporal characteristics of the target subsequent development video and, based on the displacement acceleration and critical failure points recorded in the target subsequent development video frames, calculates the time required to reach the failure threshold, thereby determining the estimated time of the next landslide for the landslide instance.

[0226] The second aspect of this embodiment discloses a UAV landslide image clustering device for emergency monitoring, which includes an image acquisition module, a feature extraction module, a feature fusion module, an undirected graph construction module, and a cluster generation module.

[0227] The image acquisition module is used to acquire raw images taken by a camera mounted on a drone.

[0228] The feature extraction module is used to extract multimodal features of landslide instances from the original image. The multimodal features include spatial location features, global semantic embedding features, and local geometric matching features.

[0229] The feature fusion module is used to fuse the multimodal features using an attention-based neural network model to generate instance feature vectors.

[0230] The undirected graph construction module is used to construct a graph structure representing the relationship between landslide instances based on spatial location features, local geometric matching features, and instance feature vectors, through an iterative process including a feedback mechanism.

[0231] The clustering generation module is used to extract connected components in the graph structure and determine each connected component as a landslide instance cluster.

[0232] In some embodiments of this example, the image acquisition module, feature extraction module, feature fusion module, undirected graph construction module, and cluster generation module are deployed on an edge computing device, which is mounted on a drone.

[0233] In some embodiments of this example, the edge computing device is a Jetson NX edge computing device. The average image processing speed of the Jetson NX edge computing device is 200ms / image.

[0234] The Jetson NX edge computing device has a power consumption of 10-15W and can provide 21 TOPS of AI computing power. The main driver of the Jetson NX edge computing device is based on the ROS2 framework. Through ROS2 lifecycle nodes, it achieves multi-module collaborative control and real-time data flow. A lifecycle state management mechanism is adopted to orderly transition the configuration and activation states of the core positioning and detection nodes, ensuring that nodes complete resource initialization according to a preset process and avoiding real-time processing interruptions caused by disordered startup order. A global parameter coordination mechanism is designed, defining project directory names containing sequence numbers and timestamps in the startup file and synchronizing them to all nodes. This ensures that real-time collected positioning data, images, etc., are uniformly stored in a specified path on an external device (USB flash drive / SD card), solving the problem of inconsistent data paths across multiple nodes.

[0235] In some embodiments of this example, the edge computing device is positioned on the top of the drone and is connected to the drone via an adapter board (such as the OSDK interface on the top of the DJI M300) to ensure the stability of data transmission, thereby avoiding additional packaging and improving heat dissipation and maintainability.

[0236] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0237] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A UAV landslide image clustering method for emergency monitoring, characterized in that, The method comprises the following steps: acquiring original images captured by a camera mounted on a UAV; extracting multi-modal features of landslide instances from the original images, the multi-modal features including spatial position features, global semantic embedding features, and local geometric matching features; fusing the multi-modal features by using a neural network model based on an attention mechanism to generate instance feature vectors; constructing a graph structure representing the correlation between landslide instances through an iterative process with a feedback mechanism based on the spatial position features, the local geometric matching features, and the instance feature vectors; extracting connected components in the graph structure, and determining each connected component as a landslide instance cluster; constructing a graph structure representing the correlation between landslide instances through an iterative process with a feedback mechanism based on the spatial position features, the local geometric matching features, and the instance feature vectors, comprising: creating an empty undirected graph, and each node in the graph representing a landslide instance; constructing a KD-Tree based on the spatial position features of the landslide instances, performing a nearest neighbor query with a determined K value, and generating a candidate instance pair set; calculating the similarity between the instance feature vectors of the candidate instance pairs by using a Faiss algorithm, and removing the candidate instance pairs with a similarity less than a first threshold value; performing local geometric feature matching on the candidate instance pairs by using a LightGlue algorithm, counting the proportion of matching key points in the total number of key points, and removing the candidate instance pairs with a proportion less than a second threshold value; adding the candidate instance pairs that pass the verification in the current iteration to the undirected graph as edges; determining whether the maximum number of iterations is reached, and if so, terminating the iteration; if the maximum number of iterations is not reached, counting the pass rate of the candidate instance pairs when the matching verification is performed by using the LightGlue algorithm in the current iteration, and if the pass rate is greater than or equal to a fourth threshold value, increasing the K value and iterating again; if the pass rate is less than the fourth threshold value, terminating the iteration. 2.The UAV landslide image clustering method for emergency monitoring of claim 1, wherein, extracting multi-modal features of landslide instances from the original images, comprising: extracting spatial position features, including longitude, latitude, elevation, pitch angle, yaw angle, and roll angle extracted from the metadata of the original images; detecting the oriented bounding box of the landslide instance in the original image; cropping the instance region image from the original image according to the oriented bounding box, and scaling the instance region image to a preset size; extracting global semantic embedding features, including global semantic embedding feature vectors extracted from the instance region image; extracting local geometric matching features, including key points and descriptors extracted by multi-directional rotation sampling of the instance region image. 3.The UAV landslide image clustering method for emergency monitoring of claim 2, wherein, detecting the oriented bounding box of the landslide instance in the original image, comprising: performing landslide detection on the original image to identify the landslide instance in the original image; labeling the polygon outline of the landslide instance; calculating the circumscribed rectangle of the polygon outline, and determining the circumscribed rectangle as the oriented bounding box of the landslide instance. 4.The emergency monitoring oriented UAV landslide image clustering method of claim 1, wherein, fusing the multi-modal features by using a neural network model based on an attention mechanism to generate a fused feature vector, comprising: encoding the longitude, latitude, elevation, pitch angle, yaw angle, and roll angle to generate a geographic-pose encoding; concatenate all descriptors in the local geometric matching feature to form a local geometric embedding feature vector; concatenate the geo-pose encoding, the local geometric embedding feature vector and the global semantic embedding feature vector to form a multi-scale feature vector; input the multi-scale feature vector into the Efficient Former module for cross-modal attention fusion to generate an instance feature vector. 5.The emergency monitoring oriented UAV landslide image clustering method of claim 4, wherein, encode the longitude, latitude, height, pitch angle, yaw angle and roll angle to generate a geo-pose encoding, including: encode the longitude, latitude, height, pitch angle, yaw angle and roll angle by using the sine-cosine position encoding method to generate a static basic encoding; respectively calculate the change amount of the pitch angle, yaw angle and roll angle in the current frame and the pitch angle, yaw angle and roll angle in the previous frame, and map the change amount to a dynamic pose compensation encoding by using a fully connected layer; concatenate the static basic encoding and the dynamic pose compensation encoding to form the geo-pose encoding. 6.The emergency monitoring oriented UAV landslide image clustering method of claim 1, wherein, perform local geometric feature matching on the candidate instance pair by using the LightGlue algorithm, including: combine the descriptors in multiple directions of the two landslide instances; input the assembled descriptors into the pre-trained LightGlue model to calculate the similarity confidence score of each feature point pair; determine the matching key point pair as the key point pair with a similarity confidence score greater than a third threshold. 7.The emergency monitoring oriented UAV landslide image clustering method of claim 1, wherein, The method further includes: obtain time series images of the landslide instance cluster; determine landslide instance active point information based on the time series images of the landslide instance cluster and the landslide instance cluster; cluster the landslide instance active point information using a K-means clustering algorithm to obtain K clusters; determine high active area information of the landslide instance, medium active area information of the landslide instance and low active area information of the landslide instance based on the K clusters; generate multi-class subsequent development videos of the landslide instance based on the time series images of the landslide instance cluster, the high active area information of the landslide instance, the medium active area information of the landslide instance and the low active area information of the landslide instance; determine the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance based on the multi-class subsequent development videos of the landslide instance by using a graph neural network. 8.The emergency monitoring oriented UAV landslide image clustering method of claim 7, wherein, The method of determining the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance based on the multi-class subsequent development videos of the landslide instance by using a graph neural network includes: construct a landslide instance development graph, the landslide instance development graph including a plurality of subsequent development video nodes and a plurality of edges between the subsequent development video nodes, each subsequent development video node representing a class of subsequent development videos, and the node features of each subsequent development video node being a class of subsequent development videos and time series images of the landslide instance cluster, and the edges between the nodes being the similarity between the subsequent development videos; process the landslide instance development graph based on a graph neural network to obtain the target subsequent development video of the landslide instance and the estimated time of the next landslide of the landslide instance.

9. The UAV landslide image clustering apparatus for emergency monitoring for performing the UAV landslide image clustering method for emergency monitoring according to claim 1, characterized in that, including: an image acquisition module configured to acquire original images captured by a camera mounted on a UAV; The feature extraction module is configured to extract multi-modal features of the landslide instances from the original images, the multi-modal features including spatial position features, global semantic embedding features, and local geometric matching features; The feature fusion module is configured to fuse the multi-modal features by using a neural network model based on an attention mechanism to generate instance feature vectors; The undirected graph construction module is configured to construct a graph structure representing the association relationship between the landslide instances based on the spatial position features, the local geometric matching features, and the instance feature vectors through an iterative process including a feedback mechanism; The clustering generation module is configured to extract connected components in the graph structure and determine each connected component as a landslide instance cluster. The undirected graph construction module is configured to construct a graph structure representing the association relationship between the landslide instances based on the spatial position features, the local geometric matching features, and the instance feature vectors through an iterative process including a feedback mechanism, including: An empty undirected graph is created, and each node in the graph represents a landslide instance; A KD-Tree is constructed based on the spatial position features of the landslide instances, and a neighbor query is performed with a determined K value to generate a candidate instance pair set; The similarity between the instance feature vectors of the candidate instance pairs is calculated by using the Faiss algorithm, and the candidate instance pairs with a similarity less than a first threshold value are removed; The local geometric feature matching of the candidate instance pairs is performed by using the LightGlue algorithm, the proportion of the number of matching key points to the total number of key points is counted, and the candidate instance pairs with a proportion less than a second threshold value are removed; The candidate instance pairs that pass the verification in the current iteration are added as edges to the undirected graph; It is determined whether the maximum number of iterations is reached, and if so, the iteration is terminated; If the maximum number of iterations is not reached, the pass proportion of the candidate instance pairs in the current iteration when the matching verification is performed by using the LightGlue algorithm is counted: if the pass proportion is greater than or equal to a fourth threshold value, the K value is increased and the iteration is performed again; if the pass proportion is less than the fourth threshold value, the iteration is terminated.

Images