Landslide modeling method and system based on unmanned aerial vehicle image and multi-modal feature fusion

Abstract

The invention discloses a landslide modeling method and system based on unmanned aerial vehicle image and multi-modal feature fusion, and belongs to the technical field of landslide monitoring. The landslide modeling method comprises the following steps: acquiring an original image; detecting a bounding box of a landslide in the image, and extracting a feature vector of a landslide area in the bounding box and a central point of the bounding box; a sliding window is used for sliding on the image sequence, and center points contained in the window at each position form a point pool; collinear points in the point pool are detected, and center points corresponding to straight lines containing the most collinear points are determined as a point set; forming a candidate class by the center points with the similarity meeting a first preset condition; combining the candidate classes of which the similarity meets a second preset condition to form a final class; and drawing a mask for each final class and performing three-dimensional modeling. The problem that in the prior art, the landslide identification process needs to depend on manual intervention to define the specific landslide object of the geographic space corresponding to the landslide body in the image is solved.

Description

Technical Field

[0001] This invention belongs to the field of landslide monitoring technology, and in particular relates to a landslide modeling method and system based on the fusion of UAV imagery and multimodal features. Background Technology

[0002] In rugged mountainous areas with complex geological structures, active neotectonic movements, and fractured rock masses, the dramatic topographic relief, well-developed active faults, and the influence of climatic factors such as heavy rainfall create a unique environment prone to frequent "earthquake-flash flood-mudslide-landslide" disaster chains, posing a serious threat to people's lives and property. Therefore, timely and accurate detection and early warning of geological disasters such as landslides are crucial.

[0003] In the field of landslide detection, traditional methods rely on static GNSS point deployment. This method requires manual intervention, has limited accuracy, and is time-consuming. Due to regional geographical conditions, it takes 3-7 days to obtain preliminary survey results. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a landslide modeling method and system based on the fusion of UAV imagery and multimodal features. This solves the technical problem in the existing technology that landslide identification requires manual intervention to define the specific landslide object in the geographic space corresponding to the landslide body in the image.

[0005] This invention is achieved through the following technical solution:

[0006] The first aspect of this invention discloses a landslide modeling method based on the fusion of UAV imagery and multimodal features, comprising:

[0007] Acquire raw images, which include at least 10 consecutive images taken by a camera mounted on the UAV along the same flight path;

[0008] Detect the bounding box of a landslide in an image, extract the feature vector of the landslide area within the bounding box, and calculate the center point of the bounding box;

[0009] A sliding window is used to slide across the image sequence, and at each position, all the center points contained within the window constitute a point pool;

[0010] Collinear points in the detection point pool are identified by defining the center point of the line containing the most collinear points as a point set.

[0011] Based on the feature vector corresponding to the center point and the area of ​​the bounding box, calculate the similarity between any two center points in the point set, and form a candidate class for center points whose similarity meets the first preset condition.

[0012] The similarity between any two candidate classes is calculated based on the feature vectors corresponding to the center points contained in the candidate classes and the area of ​​the bounding boxes. Candidate classes whose similarity satisfies the second preset condition are merged to form the final class.

[0013] Each final class has a mask drawn and a 3D model created.

[0014] Furthermore, the bounding boxes of landslides in the image are detected, including:

[0015] Detect the polygonal outline of a landslide in an image;

[0016] Calculate the bounding rectangle of the polygonal outline and determine the bounding box of the landslide using this bounding rectangle.

[0017] Furthermore, the feature vectors of the landslide area within the bounding box are extracted, including:

[0018] Feature vectors of landslide regions within bounding boxes are extracted using a pre-trained residual network.

[0019] Furthermore, for collinear points in the detection point pool, the center points corresponding to the lines containing the most collinear points are defined as a point set, including:

[0020] Get all point pairs in the point pool;

[0021] For each pair of points, calculate the line defined by that pair;

[0022] Calculate the perpendicular distance from all other center points in the point pool to the line;

[0023] All center points whose vertical distance is less than the first threshold are identified as a set of collinear points;

[0024] A point set is defined as the pair of collinear points with the largest number of points and the corresponding straight lines.

[0025] Furthermore, the similarity between any two center points in the point set is calculated based on the feature vector corresponding to the center point and the bounding box area, including:

[0026] For any two center points in the point set, calculate the similarity between the feature vectors corresponding to the two center points, and calculate the similarity between the bounding box areas corresponding to the two center points;

[0027] If the similarity between the feature vectors corresponding to two center points is greater than the third threshold, and the similarity between the bounding box areas corresponding to the two center points is greater than the fourth threshold, then the similarity between the two center points satisfies the first preset condition.

[0028] Furthermore, the similarity between feature vectors is calculated using the Faiss library.

[0029] Furthermore, the similarity between any two candidate classes is calculated based on the feature vectors corresponding to the center points contained in the candidate classes and the bounding box areas, including:

[0030] For each candidate class, the average of the feature vectors corresponding to all center points in the candidate class is calculated as the feature vector of the candidate class, and the average of the bounding box areas corresponding to all center points in the candidate class is calculated as the bounding box area of ​​the candidate class.

[0031] For any two candidate classes, calculate the similarity between the feature vectors corresponding to the two candidate classes, and the similarity between the bounding box areas corresponding to the two candidate classes;

[0032] If the similarity between the feature vectors corresponding to two candidate classes is greater than the fifth threshold, and the similarity between the bounding box areas corresponding to the two candidate classes is greater than the sixth threshold, then the similarity between the two candidate classes satisfies the second preset condition.

[0033] Furthermore, the landslide modeling method also includes: extracting GPS information from the image, wherein the GPS information of the image includes longitude, latitude, and altitude.

[0034] Furthermore, the sliding window has a size of 3 or more images, and the sliding step of the sliding window is 1 image.

[0035] A second aspect of the present invention discloses a landslide modeling system based on the fusion of UAV imagery and multimodal features, comprising:

[0036] An image acquisition module is used to acquire raw images, which include at least 10 consecutive images taken by a camera mounted on the UAV along the same flight path;

[0037] The data preprocessing module is used to detect the bounding box of landslides in the image, extract the feature vector of the landslide area within the bounding box, and calculate the center point of the bounding box.

[0038] A point pool generation module is used to slide a sliding window over an image sequence, wherein all the center points contained within the window at each position constitute a point pool.

[0039] The point set generation module is used to detect collinear points in the point pool and determine the center point corresponding to the line containing the most collinear points as a point set;

[0040] The candidate class generation module is used to calculate the similarity between any two center points in the point set based on the feature vector corresponding to the center point and the area of ​​the bounding box, and to form a candidate class by center points whose similarity meets the first preset condition.

[0041] The final class generation module is used to calculate the similarity between any two candidate classes based on the feature vectors corresponding to the center points contained in the candidate classes and the area of ​​the bounding boxes, and merge the candidate classes whose similarity meets the second preset condition to form the final class;

[0042] The 3D modeling module is used to draw masks for each final class and perform 3D modeling.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) The present invention constructs a point-to-line distance model (e.g., with a threshold of 10 pixels) by using the image-side coordinates of the center point of landslide instances in continuous images, identifies the linear motion trajectory of the same landslide, solves the problem of cross-frame target association, and solves the problem of target loss in image fault zones;

[0045] (2) Compared with using single sensor data, this invention uses residual network to extract deep visual features of instances, combines the FAISS library to calculate feature similarity (threshold adjustable), introduces size ratio constraints (based on the size of the bounding box), and reduces the false matching rate.

[0046] (3) Compared with the method of manually delineating the landslide range, the present invention achieves the aggregation of landslide entities in geospatial space by iteratively merging similar landslide classes (requiring all instances in the class to meet the similarity and size thresholds in pairs), and the mask and DOM fusion error is less than 0.5m. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, do not constitute a limitation thereof. In the drawings:

[0048] Figure 1 This is a flowchart of a landslide modeling method in this invention. Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0051] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the accompanying drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0052] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0053] like Figure 1 As shown in the figure, this embodiment discloses a landslide modeling method and system based on the fusion of UAV imagery and multimodal features.

[0054] The first aspect of this embodiment discloses a landslide modeling method based on the fusion of UAV imagery and multimodal features, such as... Figure 1 As shown, the landslide modeling method includes steps S100 to S700.

[0055] Step S100. Acquire raw images, which include at least 10 consecutive images taken by a camera mounted on the UAV along the same flight path.

[0056] In some embodiments of this example, the UAV's flight mode is terrain-following flight, with a terrain-following altitude of 350-500 meters, a heading overlap of 80%, a flight path overlap of 80%, and a camera shooting frequency of 3 seconds per frame.

[0057] The selection of the camera needs to comprehensively consider multiple factors such as resolution, lens attributes, focal length, and field of view. Furthermore, for airborne platform shooting, data transmission interface compatibility must also be considered. During flight, the drone will sway and shift in the air, requiring the camera to be able to set appropriate exposure times to obtain clear images. For example, an Insta360 camera is selected. The sensor chosen is the Sony IMX586 stacked CMOS image sensor, which offers high dynamic range and high sensitivity, enabling clear and complete imaging during drone flight operations. It supports the Open Spherical Camera (OSC) protocol.

[0058] Step S200. Detect the bounding box of the landslide in the image, extract the feature vector of the landslide area within the bounding box, and calculate the center point of the bounding box.

[0059] In some embodiments of this example, detecting the bounding box of a landslide in an image includes: detecting the polygonal outline of the landslide in the image; calculating the circumscribed rectangle of the polygonal outline, and determining the circumscribed rectangle as the bounding box (BBox) of the landslide.

[0060] In some implementations of this embodiment, the feature vector of the landslide area within the bounding box is extracted, including: using a pre-trained residual network (ResNet18) to extract the feature vector of the landslide area within the bounding box.

[0061] Step S300. A sliding window is used to slide on the image sequence, wherein all the center points contained in the window at each position constitute a point pool.

[0062] For example, suppose the sliding window is 6 images in size and the sliding step is 1 image, with each image having a sliding bounding box; when the sliding window is in its initial position, the first to sixth images are contained within the sliding window, and the center points corresponding to the first to sixth images form a point pool; when the sliding window slides one step from its initial position, the second to seventh images are contained within the sliding window, and the center points corresponding to the second to seventh images form a point pool.

[0063] In some embodiments of this example, the size of the sliding window is 3 or more images (for example, the size of the sliding window is 6 images), and the sliding step of the sliding window is 1 image.

[0064] In some implementations of this embodiment, if the number of center points contained in the sliding window is less than 2 when the sliding window is in a certain position, then the center points contained in the window do not constitute a point pool.

[0065] Step S400. Detect collinear points in the point pool, and determine the center point corresponding to the line containing the most collinear points as a point set.

[0066] In some embodiments of this example, detecting collinear points in the point pool and determining the center points corresponding to the line containing the most collinear points as a point set includes: acquiring all point pairs in the point pool; for each point pair, calculating the line determined by the point pair, calculating the perpendicular distance from all other center points in the point pool to the line, and determining all center points whose perpendicular distance is less than a first threshold (e.g., the first threshold is 10 pixels or 20 pixels) as a group of collinear points; and determining the group of collinear points with the most points and the point pairs that determine the corresponding lines as a point set. That is, the point set contains the group of collinear points with the most points and the two center points that determine the lines corresponding to the group of collinear points.

[0067] The center point in each point pool generates only one point set.

[0068] If the number of center points in the collinear point pool with the most points is less than the second threshold (for example, the second threshold is 3), then the point pool is skipped and no point set is generated from that point pool.

[0069] In some embodiments of this example, the formula for calculating the perpendicular distance from the center point to the line is:

[0070] In the formula, Distance represents the perpendicular distance from the center point to the line; x and y represent the pixel coordinates of the center point; A, B, and C represent the coefficients of the general equation of the line, Ax + By + C = 0. It represents the length of the normal vector of the line and serves as a normalization function.

[0071] In this embodiment, the initial landslide class is extracted by detecting collinear points and using trajectory constraints.

[0072] Step S500. Calculate the similarity between any two center points in the point set based on the feature vector corresponding to the center point and the bounding box area, and form a candidate class for center points whose similarity meets the first preset condition.

[0073] For example, the similarity between two center points includes the similarity between the feature vectors corresponding to the two center points and the similarity between the bounding box areas corresponding to the two center points, where the bounding box area corresponding to the center point refers to the area of ​​the bounding box corresponding to the center point.

[0074] For any center point in the point set, if the similarity between the center point and another center point in the point set meets the first preset condition, then the center point is added to the candidate class corresponding to the point set.

[0075] In some embodiments of this example, calculating the similarity between any two center points in the point set based on the feature vectors corresponding to the center points and the bounding box area includes: for any two center points in the point set, calculating the similarity between the feature vectors corresponding to the two center points and calculating the similarity between the bounding box areas corresponding to the two center points; if the similarity between the feature vectors corresponding to the two center points is greater than a third threshold (for example, the third threshold is 0.9), and the similarity between the bounding box areas corresponding to the two center points is greater than a fourth threshold (for example, the fourth threshold is 0.4), then the similarity between the two center points satisfies the first preset condition.

[0076] For example, the similarity between the areas of two bounding boxes is calculated as follows: obtain the areas of the two bounding boxes, divide the smaller area by the larger area to get the area similarity between the two bounding boxes, and the closer the ratio is to 1, the more similar they are.

[0077] In some implementations of this embodiment, the similarity between feature vectors is calculated using the Faiss library.

[0078] Step S600. Calculate the similarity between any two candidate classes based on the feature vectors corresponding to the center points contained in the candidate classes and the bounding box area. Merge the candidate classes whose similarity satisfies the second preset condition to form the final class.

[0079] In this embodiment, after merging two candidate classes whose similarity meets the second preset condition, the similarity between the newly obtained candidate class and other candidate classes is calculated again. When there are no two candidate classes whose similarity meets the second preset condition, each candidate class constitutes a final class.

[0080] In some embodiments of this example, the similarity between any two candidate classes is calculated based on the feature vectors corresponding to the center points of the candidate classes and the bounding box areas. This includes: for each candidate class, calculating the average of the feature vectors corresponding to all center points in the candidate class as the feature vector of the candidate class, and calculating the average of the bounding box areas corresponding to all center points in the candidate class as the bounding box area of ​​the candidate class; for any two candidate classes, calculating the similarity between the feature vectors corresponding to the two candidate classes and the similarity between the bounding box areas corresponding to the two candidate classes; if the similarity between the feature vectors corresponding to the two candidate classes is greater than a fifth threshold (for example, the fifth threshold is 0.9), and the similarity between the bounding box areas corresponding to the two candidate classes is greater than a sixth threshold (for example, the sixth threshold is 0.4), then the similarity between the two candidate classes satisfies the second preset condition.

[0081] Step S700: Draw a mask for each final class and perform 3D modeling.

[0082] For example, create a separate folder for each final class, draw a mask on the original image for landslides belonging to that final class, and save it to the corresponding folder; use the mask image to build a 3D model and a digital orthophoto map (DOM), and the mask area clearly marks the landslide location in the 3D model.

[0083] In some embodiments of this example, the landslide modeling method further includes: extracting GPS information from an image, wherein the GPS information of the image includes information such as longitude, latitude, and altitude.

[0084] The method in this embodiment solves the technical problem in the prior art that the landslide identification process requires manual intervention to define the specific landslide object in the geographic space corresponding to the landslide body in the image, thereby effectively improving the efficiency of landslide information extraction and reducing the interpretation cost.

[0085] The pseudocode corresponding to the method in this embodiment is as follows:

[0086] # Pseudocode for the main process of landslide clustering

[0087] function main(image_json_filepath):

[0088] # Initialize timers and data structures

[0089] start_time = get_current_time()

[0090] boxes_dict = {} # Store image IDs and bounding box coordinates

[0091] feature_vecs = {} # Store feature vectors

[0092] img_info = {} # Store image GPS information

[0093] # Data Preprocessing

[0094] For each JSON annotation file in image_json_filepath:

[0095] Parse JSON file to obtain landslide labeling information

[0096] For each landslide polygon, use the annotation:

[0097] Calculate the bounding box (BBox) coordinates

[0098] Calculate the coordinates of the center point of the BBox

[0099] Extract GPS information (longitude / latitude / altitude) from the image.

[0100] Extracting feature vectors of the bounding box region using ResNet18

[0101] Stored in boxes_dict, feature_vecs, and img_info

[0102] # Collinear point detection in sliding window

[0103] baseline_candidates = [] # Set of candidate baselines

[0104] sorted_img_ids = A list of image IDs sorted by time

[0105] for window start position in image sequence:

[0106] Window end position = min(start position + 6, total number of images)

[0107] Current window image = image sequence [start position: end position + 1]

[0108] point_pool = [] # All center points of the current window

[0109] point_index = [] # Point corresponds to (image ID, bounding box ID)

[0110] for image ID in the current window image:

[0111] For each landslide frame in the image:

[0112] Add center point to point_pool

[0113] Add index to point_index

[0114] if points < 2: Skip the current window

[0115] # Find collinear points (geometric constraints)

[0116] Candidate baselines = find_collinear_points(point_pool, point_index, distance_threshold=20 pixels)

[0117] If the number of candidate baseline points is less than the preset threshold (3): skip.

[0118] # Intra-class feature fusion

[0119] valid_baseline = []

[0120] for candidate points in candidate baselines:

[0121] Obtain the corresponding feature vector and BBox information

[0122] # Feature Similarity Validation (FAISS)

[0123] valid_baseline = faiss_verify_similarity(

[0124] candidates = candidate baseline,

[0125] feature_threshold=0.9,

[0126] size_threshold=0.4 )

[0128] If valid_baseline is not empty:

[0129] Add to baseline_candidates

[0130] # Inter-class clustering optimization (disjoint-set data structure)

[0131] uf = Initialize Disjoint-set Dataset()

[0132] # Phase 1: Building the Cluster Index

[0133] for i in range(len(baseline_candidates)):

[0134] uf.add(i)

[0135] # Phase Two: Clustering Fusion

[0136] for i in all candidate classes:

[0137] for j in all candidate classes and j>i:

[0138] # Calculate the average eigenvector and average bounding box size of cluster i

[0139] mean_feature_i = Calculate the mean of features in class i ()

[0140] mean_size_i = Calculate the average size of the bounding box in class i ()

[0141] # Calculate the average eigenvector and average bounding box size of cluster j

[0142] mean_feature_j = Calculate the mean of features in class j()

[0143] mean_size_j = Calculate the average size of the bounding box in class j()

[0144] # Calculate feature similarity and size similarity

[0145] feature_sim = faiss_compare(mean_feature_i, mean_feature_j)

[0146] size_sim = min(mean_size_i, mean_size_j) / max(mean_size_i, mean_size_j)

[0147] # Dual threshold fusion judgment

[0148] if feature_sim > 0.9 and size_sim > 0.4:

[0149] uf.union(i, j) # Merge similar clusters

[0150] # Generate the final cluster

[0151] clusters = {}

[0152] for candidate class index in baseline_candidates index:

[0153] Root node = uf.find(candidate class index)

[0154] clusters[root node].append(candidate index)

[0155] # Results Output and Visualization

[0156] result_data = []

[0157] for cluster ID, class member in clusters.items():

[0158] for image ID, frame ID in class member:

[0159] Image filename = img_info[image ID][frame ID].filename

[0160] Latitude and longitude = img_info[image ID][frame ID].GPS information

[0161] Add record to result_data

[0162] # Draw landslide mask

[0163] draw_landslide_masks(

[0164] Cluster ID,

[0165] Image file name,

[0166] Frame ID,

[0167] Output directory )

[0169] end_time = get_current_time()

[0170] Print ("Clustering time: {end_time-start_time} seconds")

[0171] return result_data

[0172] # Collinearity Detection Algorithm

[0173] function find_collinear_points(points, point_index, threshold):

[0174] if points < 2: return an empty list

[0175] best_points = [] # Best set of collinear points

[0176] For each pair of points (p1, p2), consider all possible combinations of points:

[0177] # Calculate the equation of the line: Ax + By + C = 0

[0178] A = p2.y - p1.y

[0179] B = p1.x - p2.x

[0180] C = p2.x*p1.y - p1.x*p2.y

[0181] Denominator = sqrt(A² + B²)

[0182] If the denominator is close to 0: skip.

[0183] # Calculate the distance from all points to the line

[0184] distances = []

[0185] For points:

[0186] dist = |A*pt.x + B*pt.y + C| / denominator

[0187] distances.append(dist)

[0188] # Filter points within the distance threshold

[0189] current_points = indices of points whose distance is less than the threshold

[0190] If current points > best_points:

[0191] best_points = current_points

[0192] # Convert to raw index format

[0193] return [point_index[idx] for idx in best_points]

[0194] # FAISS Feature Validation (Within Class)

[0195] function faiss_verify_similarity(candidates, feature_threshold, size_threshold):

[0196] valid_pairs = []

[0197] for point pairs (cand_i, cand_j) in all candidate point combinations:

[0198] # Calculate feature similarity

[0199] sim = cosine_similarity(cand_i.feature, cand_j.feature)

[0200] # Calculate BBox size similarity

[0201] size_i = (cand_i.bbox.w * cand_i.bbox.h)

[0202] size_j = (cand_j.bbox.w * cand_j.bbox.h)

[0203] size_sim = min(size_i, size_j) / max(size_i, size_j)

[0204] if sim >= feature_threshold and size_sim >= size_threshold:

[0205] valid_pairs.append((cand_i, cand_j))

[0206] return the set of valid point pairs.

[0207] The second aspect of this embodiment discloses a landslide modeling system based on the fusion of UAV imagery and multimodal features. The landslide modeling system includes an image acquisition module, a data preprocessing module, a point pool generation module, a point set generation module, a candidate class generation module, a final class generation module, and a 3D modeling module.

[0208] The image acquisition module is used to acquire raw images, which include at least 10 consecutive images taken by a camera mounted on the UAV along the same flight path.

[0209] The data preprocessing module is used to detect the bounding box of the landslide in the image, extract the feature vector of the landslide area within the bounding box, and calculate the center point of the bounding box.

[0210] The point pool generation module is used to slide a sliding window on an image sequence, wherein all the center points contained within the window at each position constitute a point pool.

[0211] The point set generation module is used to detect collinear points in the point pool and determine the center point corresponding to the line containing the most collinear points as a point set.

[0212] The candidate class generation module is used to calculate the similarity between any two center points in the point set based on the feature vector corresponding to the center point and the area of ​​the bounding box, and to form a candidate class by center points whose similarity meets the first preset condition.

[0213] The final class generation module is used to calculate the similarity between any two candidate classes based on the feature vectors corresponding to the center points contained in the candidate classes and the area of ​​the bounding boxes, and merge the candidate classes whose similarity satisfies the second preset condition to form the final class.

[0214] The 3D modeling module is used to draw masks for each final class and to perform 3D modeling.

[0215] In some embodiments of this example, the landslide modeling system further includes a landslide instance segmentation model. This model is used to segment images into instances and output a JSON file containing landslide contours with pixel coordinates. The landslide instance segmentation model is deployed on an airborne edge computing device. This device acts as the master node, responsible for real-time data processing, storage, distribution, and time synchronization and data communication among various hardware devices. It can monitor newly added images in real time and extract landslide instances during UAV flight. For example, the airborne edge device uses Jetson NX, which can provide 100 TOPS (Tera Operations Per Second) of AI performance under harsh conditions with a power input of 10 to 25 watts, meeting the needs of real-time processing. This allows for the generation of landslide mask JSON files while in flight, and clustering operations can be performed on-board during the return flight, improving response speed by over 50%.

[0216] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0217] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion, characterized in that, The method comprises the following steps: obtaining an original image, the original image comprising at least 10 continuous images taken by a camera mounted on a UAV on the same flight line; detecting a bounding box of a landslide in the image, extracting a feature vector of a landslide region in the bounding box, and calculating a center point of the bounding box; sliding a sliding window on the image sequence, wherein all center points contained in the sliding window at each position form a point pool; detecting collinear points in the point pool, and determining the center points corresponding to the straight line containing the most collinear points as a point set; calculating the similarity between any two center points in the point set based on the feature vectors and the bounding box areas corresponding to the center points, and grouping the center points satisfying a first preset condition in similarity as a candidate class; calculating the similarity between any two candidate classes based on the feature vectors and the bounding box areas corresponding to the center points contained in the candidate classes, and merging the candidate classes satisfying a second preset condition in similarity to form a final class; drawing a mask for each final class respectively, and performing three-dimensional modeling. 2.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The method for detecting a bounding box of a landslide in an image comprises the following steps: detecting a polygon contour of a landslide in the image; calculating a circumscribed rectangle of the polygon contour, and determining the circumscribed rectangle as the bounding box of the landslide. 3.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The method for extracting a feature vector of a landslide region in the bounding box comprises the following steps: extracting a feature vector of a landslide region in the bounding box by using a pre-trained residual network. 4.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The method for detecting collinear points in the point pool and determining the center points corresponding to the straight line containing the most collinear points as a point set comprises the following steps: obtaining all point pairs in the point pool; calculating a straight line determined by each point pair; calculating the perpendicular distance of all other center points in the point pool to the straight line; determining a group of collinear points by center points with a perpendicular distance less than a first threshold value; determining a point set by the point pair corresponding to the straight line containing the most collinear points. 5.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The method for calculating the similarity between any two center points in the point set based on the feature vectors and the bounding box areas corresponding to the center points comprises the following steps: for any two center points in the point set, calculating the similarity between the feature vectors corresponding to the two center points, and calculating the similarity between the bounding box areas corresponding to the two center points; if the similarity between the feature vectors corresponding to the two center points is greater than a third threshold value, and the similarity between the bounding box areas corresponding to the two center points is greater than a fourth threshold value, then the similarity between the two center points satisfies a first preset condition. 6.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 5, characterized in that, The similarity between the feature vectors is calculated by using a Faiss library. 7.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The method for calculating the similarity between any two candidate classes based on the feature vectors and the bounding box areas corresponding to the center points contained in the candidate classes comprises the following steps: for each candidate class, calculating the average value of the feature vectors corresponding to all center points in the candidate class as the feature vector corresponding to the candidate class, and calculating the average value of the bounding box areas corresponding to all center points in the candidate class as the bounding box area corresponding to the candidate class; for any two candidate classes, calculating the similarity between the feature vectors corresponding to the two candidate classes, and calculating the similarity between the bounding box areas corresponding to the two candidate classes. If the similarity between the feature vectors corresponding to the two candidate classes is greater than a fifth threshold value, and the similarity between the areas of the bounding boxes corresponding to the two candidate classes is greater than a sixth threshold value, the similarity between the two candidate classes satisfies a second preset condition. 8.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The landslide modeling method further includes extracting GPS information of the image, the GPS information of the image including longitude, latitude, and height. 9.The landslide modeling method based on unmanned aerial vehicle image and multi-modal feature fusion according to claim 1, characterized in that, The size of the sliding window is 3 or more images, and the sliding step of the sliding window is 1 image.

10. A landslide modeling system based on unmanned aerial vehicle image and multi-modal feature fusion, characterized in that, Comprise: An image acquisition module is configured to acquire an original image, the original image including at least 10 continuous images taken by a camera mounted on a UAV on the same flight line; A data preprocessing module is configured to detect a bounding box of a landslide in the image, extract a feature vector of a landslide region in the bounding box, and calculate a center point of the bounding box; A point pool generation module is configured to slide a sliding window on the image sequence, all center points included in the sliding window at each position forming a point pool; A point set generation module is configured to detect collinear points in the point pool, and determine center points corresponding to a straight line containing the most collinear points as a point set; A candidate class generation module is configured to calculate the similarity between any two center points in the point set based on the feature vectors and the areas of the bounding boxes corresponding to the center points, and group center points whose similarity satisfies a first preset condition into a candidate class; A final class generation module is configured to calculate the similarity between any two candidate classes based on the feature vectors and the areas of the bounding boxes corresponding to the center points included in the candidate classes, merge candidate classes whose similarity satisfies a second preset condition, and form a final class; A three-dimensional modeling module is configured to draw a mask for each final class respectively, and perform three-dimensional modeling.

Images