Satellite and low-altitude unmanned aerial vehicle remote sensing data fusion method

Abstract

The invention belongs to the field of image processing, and provides a satellite and low-altitude unmanned aerial vehicle remote sensing data fusion method, which comprises the following steps: extracting linear ground feature contour information from a satellite remote sensing image of a target area, and dividing a linear ground feature contour into a plurality of contour segments; extracting real-time linear ground feature contour information from the unmanned aerial vehicle remote sensing image of the target area, and dividing the real-time linear ground feature contour into a plurality of contour segments with the same structural form as the satellite remote sensing image; performing corresponding matching on linear ground feature contour segments in the satellite remote sensing image and the unmanned aerial vehicle remote sensing image, and constructing a stable contour segment pair; carrying out geometric registration processing on stable regions in the satellite remote sensing image and the unmanned aerial vehicle remote sensing image; and adjusting the observation angle of the unmanned aerial vehicle image in the changed area based on the relative spatial relationship between the flight path of the unmanned aerial vehicle and the linear ground feature, and fusing the image after angle adjustment and the satellite remote sensing image.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing data fusion, and particularly relates to a method for fusing satellite and low-altitude unmanned aerial vehicle remote sensing data. Background Art

[0002] With the continuous development of remote sensing technology, remote sensing data based on different platforms have been widely used in fields such as geographic information acquisition, environmental monitoring, disaster warning, and urban management. As the main means of current wide-area observation, satellite remote sensing has the characteristics of wide coverage, stable time series, and strong multi-spectral imaging ability, and is suitable for performing continuous monitoring tasks of large-scale geographical phenomena. However, limited by the orbital altitude and sensor characteristics, satellite remote sensing images still have limitations in terms of spatial resolution and time update frequency, and it is difficult to meet the high-precision and rapid response requirements for local detail changes.

[0003] To make up for the deficiencies of satellite remote sensing data, low-altitude unmanned aerial vehicle remote sensing technology has developed rapidly in recent years. By carrying high-resolution visible light, multi-spectral or thermal infrared sensors, unmanned aerial vehicles can flexibly obtain surface information of designated areas, and have the advantages of high spatial resolution, rapid deployment, and low cost, especially suitable for scenarios such as disaster response, key area monitoring, and fine recognition of ground objects. Unmanned aerial vehicle images have significant image details and real-time performance, but limited by the flight range, meteorological conditions, and flight altitude, their observation range is limited, and it is difficult to independently undertake large-area or multi-temporal remote sensing tasks.

[0004] In order to give full play to the wide coverage advantage of satellite images and the high-precision advantage of unmanned aerial vehicle images, the technical idea of multi-source remote sensing image fusion has been gradually proposed in recent years. Existing remote sensing image fusion methods mainly include image registration methods based on feature points, feature fusion methods based on deep learning, and projection superposition methods based on orthorectification. Among them, registration methods are mainly based on local features such as SIFT, SURF, and ORB, and use matching points to construct a geometric transformation model to achieve image alignment; deep learning methods extract image features through an end-to-end network model and complete image enhancement or detail compensation under the constraint of a specific loss function; orthorectification methods usually rely on high-precision DEM and sensor parameter models to project remote sensing images into a standard geospatial reference system for fusion.

[0005] However, existing methods still have problems such as insufficient matching robustness, structural distortion, and semantic inconsistency in processing multi-source remote sensing image fusion tasks with significant differences in spatial resolution, inconsistent imaging perspectives, and severe ground object mutations. Especially in sudden disaster scenarios, such as floods and debris flows, due to the satellite map not being updated in time due to delay, while unmanned aerial vehicle images are collected in real time, it is often impossible to establish an effective feature point matching relationship in areas where ground objects have changed violently, resulting in a decrease in fusion accuracy and even incorrect fusion results. Summary of the Invention

[0006] To solve the problems in the prior art, the present invention provides a method for fusing satellite and low-altitude unmanned aerial vehicle (UAV) remote sensing data, including the following steps: Step S10: Extract linear feature contour information from the satellite remote sensing image of the target area. The linear feature is a linear structure with spatial extensibility, and its length is greater than a first preset threshold; divide the linear feature contour into multiple contour segments; Step S20: Extract real-time linear feature contour information from the UAV remote sensing image of the target area, and based on the geographical location information and flight trajectory data of the UAV, divide the real-time linear feature contour into multiple contour segments with the same structural form as those in the satellite remote sensing image; Step S30: Based on the geometric shape similarity index and geographical location matching relationship of the contour segments, perform corresponding matching on the linear feature contour segments in the satellite remote sensing image and the UAV remote sensing image to construct stable contour segment pairs; Step S40: Using the stable contour segment pairs as the anchoring benchmark, construct a local area grid registration model, and perform geometric registration processing on the stable areas in the satellite remote sensing image and the UAV remote sensing image based on the registration model; Step S50: For the image areas where stable contour segment pairs cannot be established, determine them as areas where the ground object state has changed; in the changed areas, adopt a non-aligned fusion strategy mainly based on the UAV remote sensing image, and based on the relative spatial relationship between the UAV flight trajectory and the linear feature, adjust the observation angle of the UAV image in the changed areas, and perform fusion processing on the image with the adjusted angle and the satellite remote sensing image.

[0007] Further, step S10 includes the following sub-steps: Step S101: Perform image enhancement processing on the original satellite remote sensing image, including using histogram equalization to enhance the image contrast; Step S102: Use the Canny edge detection algorithm to extract the edge area, and use the Hough transform to detect the linear feature contour with spatial continuity; Step S103: Based on the geometric skeleton or boundary length, filter out short line segments smaller than the first preset threshold; Step S104: Adopt a combination of equidistant division and curvature change division to divide the remaining linear features into multiple contour segments; Step S105: Generate spatial attributes for each contour segment, including start and end point coordinates, length, direction, curvature characteristics, and coding identifiers.

[0008] Further, the sampling interval for equidistant division in step S104 is set to be 10 meters to 50 meters, and the identification of curvature mutation points is determined by the local curvature difference being greater than a set threshold, ensuring the continuity and comparability of the divided segments in terms of spatial form.

[0009] Further, step S20 includes the following sub-steps: Step S201, perform image distortion correction, attitude normalization, and GNSS-aided spatial positioning enhancement on the UAV images; Step S202, extract directional linear feature contours by combining Canny edge detection and morphological closing operation; Step S203, divide the contours into multiple segments with consistent structures using a combination rule of equidistant length and curvature threshold; Step S204, attach spatial attribute descriptions to each contour segment, including image number, heading angle, position code, and geometric feature vector.

[0010] Further, the spatial attribute of each segment in step S204 includes the timestamp of the track point, and the timestamp is used to establish a consistency judgment in the time series for subsequent registration and change recognition.

[0011] Further, step S30 includes the following sub-steps: Step S301, extract the satellite image segment set and the UAV image segment set, and establish a segment spatial index; Step S302, perform candidate matching screening based on the geographical center point within a set radius; Step S303, construct a comprehensive similarity scoring function using direction difference, curvature matching, length ratio, and shape similarity; Step S304, eliminate pseudo-matching segment pairs based on segment adjacency consistency; Step S305, record the high-scoring segment pairs as stable contour segment pairs and write them into the anchor point matching data table.

[0012] Further, the weight coefficients of the comprehensive scoring function in step S303 can be optimized and set through a training set to improve the matching accuracy and avoid segment mis-matching in complex terrain areas.

[0013] Further, step S40 includes the following sub-steps: Step S401, extract the geometric midpoints from the stable contour segment pairs as control anchor points and record their corresponding image coordinate pairs; Step S402, construct a Delaunay triangulation grid based on the control anchor points and establish a mapping relationship between the source map and the target map; Step S403: For each grid cell, use the affine or thin plate spline interpolation method to generate a local geometric transformation function, and perform pixel remapping on the local block of the satellite image; Step S404: Use the image stitching and edge fusion method to generate a consistent and stable regional image for output.

[0014] Furthermore, step S50 includes the following sub-steps: Step S501: Identify all regions where stable fragment pairs have not been established, and construct their two-dimensional boundary masks; Step S502: Match the changed region with the UAV image through the spatial coverage relationship, and extract the corresponding flight trajectory data and shooting poses; Step S503: Based on the angle between the flight trajectory direction and the ground object direction, determine whether to perform image perspective correction, and use the perspective transformation or affine transformation method to adjust the angle of the image; Step S504: Use the dominant substitution, weighted fusion or semantic-guided fusion method to cover the adjusted image block to the changed region of the satellite image to complete the fusion operation.

[0015] Furthermore, during the image fusion in step S504, the Laplacian pyramid fusion method is used for multi-scale transition at the fusion boundary to enhance the naturalness and visual consistency of the fusion region boundary.

[0016] The satellite and low-altitude UAV remote sensing data fusion method provided by the present invention can, based on the spatial extensibility and structural stability of linear ground objects, construct a set of contour fragments with registrability before image fusion, and achieve highly reliable contour fragment matching through geometric shape similarity and geographical location relationship, effectively improving the accuracy and robustness of multi-source image registration. By constructing a local area grid registration model with stable contour fragment pairs, high-precision geometric alignment of the satellite image and the UAV image is achieved within the region where the structure has not changed, and the fusion result shows excellent performance in terms of spatial consistency and image detail retention.

[0017] After the present invention identifies the regions where stable contour pairing relationships cannot be established, it determines them as regions where the ground object state has changed, and actively adopts a non-aligned fusion strategy mainly based on the UAV image within these regions. By adjusting the image observation angle through the spatial relationship between the UAV flight trajectory and the ground object trend, the problems of fusion distortion and structural misalignment caused by forced registration are effectively avoided, making the fusion image more real, clear and time-sensitive when expressing change information.

[0018] Through the differential design of structure differentiation and fusion strategies, the present invention realizes the alignment and fusion of stable structure regions and the independent expression of variable regions, improving the practical capabilities of the fusion system in disaster response, anomaly monitoring, and emergency response, and having broad engineering application prospects and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is a comparison diagram of satellite and low-altitude UAV remote sensing data; Figure 2 is a flowchart of the satellite and low-altitude UAV remote sensing data fusion method of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] Next, in combination with the drawings and specific embodiments, a preferred description of the invention will be made.

[0022] This embodiment solves the above problems through the following steps: Satellite remote sensing data refers to the surface observation data obtained by remote sensing sensors carried on orbital satellite platforms. It usually has characteristics such as wide coverage, stable acquisition cycle, and rich multi-spectral channels, and is widely used in global-scale and regional-scale geographical information monitoring. Satellite remote sensing images have strong integrity and trend expression capabilities. However, due to factors such as orbital altitude, sensor resolution, and revisit cycle, their spatial resolution is usually at a medium-low level (generally 10 meters to the hundreds of meters level), and the time update frequency is limited. The update time is usually several days or even weeks, making it difficult to meet the information acquisition requirements for high-precision and rapid response in local areas.

[0023] Low-altitude UAV remote sensing data refers to the remote sensing image data close to the surface obtained by carrying remote sensing equipment (such as high-resolution visible light cameras, multi-spectral sensors, etc.) on low-altitude flight platforms (such as multi-rotor UAVs, fixed-wing UAVs). Compared with satellite remote sensing, UAV remote sensing data has advantages such as high spatial resolution, flexible acquisition, and short operation cycle, and is particularly suitable for emergency monitoring after emergencies, refined analysis of target areas, and contour detection of edge structures. However, its observation range is limited, the flight altitude is low, and it is greatly affected by factors such as the environment, flight route, and meteorology, making it difficult to complete large-scale data coverage and continuous time-series monitoring tasks alone.

[0024] In order to simultaneously leverage the respective advantages of satellite remote sensing and low-altitude UAV remote sensing, it is proposed to fuse the information of the two, that is, to register and jointly process the data from different platforms in the spatial, temporal or semantic dimensions through algorithms or models, and finally generate a remote sensing information product with both macroscopic vision and local accuracy. The fused result can not only inherit the global consistency and time series of satellite remote sensing data, but also overlay the high-precision and high-contrast image details of UAV data, thus significantly improving the perception ability and decision-making support ability of the remote sensing system in multi-source scenarios.

[0025] However, as Figure 1 shown, there are many differences between satellite remote sensing images and UAV images. Due to the characteristics of long update cycle and relatively low spatial resolution of satellite remote sensing images, when facing sudden changes in ground object states (such as debris flows, floods, landslides, forest fires, etc.), they cannot timely reflect the current true state of the ground surface; while low-altitude UAV remote sensing images, although having the advantages of real-time and high resolution, are limited by the small imaging range and large perspective changes, and it is difficult to establish an accurate corresponding relationship with the existing satellite images at the geometric structure and semantic levels.

[0026] In the above background, the present invention proposes a method for fusing satellite and low-altitude UAV remote sensing data. By extracting features, matching, registering and reconstructing the satellite remote sensing image and the real-time remote sensing image obtained by the low-altitude UAV of the same target area, on the basis of maintaining wide-area coverage and time continuity, the detail enhancement and spatio-temporal consistency expression of the local high-resolution change area are realized, which is applicable to various geospatial intelligent perception scenarios such as disaster monitoring, urban fine management, and agricultural situation dynamic assessment.

[0027] To achieve the above object, the present invention realizes data fusion through the following steps: As Figure 2 shown, in step S10, linear ground object contour information is extracted from the satellite remote sensing image of the target area. The linear ground object is a linear structure with spatial extensibility, and its length is greater than the first preset threshold; the linear ground object contour is divided into multiple contour segments.

[0028] To achieve the precise alignment and local change recognition between the low-altitude UAV remote sensing image and the satellite remote sensing image, a set of stable reference structures need to be constructed in space first as the basis for fusion anchoring. Since linear ground objects have characteristics such as strong spatial extensibility, good structural continuity, and strong anti-local disturbance ability, they can be used as the basis for constructing a regional registration model between satellite images and UAV images. By extracting the contour information of linear ground objects from satellite remote sensing images and dividing them into multiple contour segments with spatial order and morphological characteristics, a preliminary spatial anchoring framework can be formed, which can further support subsequent segment matching and local registration operations.

[0029] Linear features refer to geographical entities with significant directionality, continuity, and linear extension characteristics in the geographical space. Typical linear features include roads, rivers, coastlines, boundary zones, transmission lines, irrigation ditches, and railway tracks. In remote sensing images, linear features usually appear as strip or band-shaped areas with certain gray-scale difference boundaries and strong extensibility. Their length is usually much greater than the width, and they have a clear orientation attribute in the geographical space.

[0030] Specifically, the process of extracting and dividing the contour information of linear features in satellite remote sensing images includes the following sub-steps: Step S101: Perform preprocessing operations on the original satellite remote sensing image. The preprocessing includes image enhancement, edge enhancement, and multi-scale filtering. Image enhancement can use histogram equalization to enhance the contrast. Edge enhancement can use the Canny edge detection algorithm to extract preliminary edge candidate regions. Multi-scale filtering can be achieved through a Gaussian pyramid to retain structures at different scales.

[0031] Step S102: On the basis of the preprocessed image, perform linear structure extraction operations. The extraction operations can be obtained through one of the following implementation methods: Optional implementation schemes include using a line detection algorithm based on the Hough transform to extract a continuous edge set with significant direction features, or using a deep learning model for linear feature segmentation. For example, using a trained semantic segmentation network (such as DeepLabv3+ or UNet) to perform pixel-by-pixel classification on targets such as roads and rivers and extract the corresponding boundary contours.

[0032] Step S103: Perform length screening on the extracted linear feature contour information, and only retain linear features with a length greater than the first preset threshold. The length can be obtained by calculating the spatial distance or geometric skeleton length of the boundary contour. The setting of the first preset threshold is determined according to the target area scale and the expected accuracy of the UAV image, usually between dozens of meters and hundreds of meters.

[0033] Step S104: Divide the linear feature contours that meet the length requirements into multiple contour segments. The division method is based on equidistant sampling or curvature change segmentation. Equidistant sampling means setting a segmentation point at a fixed distance along the contour curve to form segments. Curvature change segmentation means setting segmentation boundaries at the mutation points of the contour curve curvature change to ensure that each segment has relatively uniform geometric features and stable direction attributes.

[0034] Step S105: Generate a unique segment identifier and geometric description information for each contour segment. The geometric description information includes the start and end point coordinates of the segment, the midline direction, the length value, the local curvature feature, and the position coding information, which are used for subsequent cross-image matching and registration operations.

[0035] The above processing flow can construct a set of linear feature contour segments with clear structure and semantics in satellite remote sensing images, providing a spatial anchoring basis for identifying stable and changing regions in subsequent fusion steps, and at the same time improving the robustness and accuracy of geometric registration between different remote sensing data.

[0036] In a specific example, linear feature extraction and segmentation operations are performed on high-resolution satellite remote sensing images of a mountainous area, obtaining a set of contour segments containing dozens of mountain roads and rivers. Among them, main road A is divided into 8 segments with balanced lengths, and river B is divided into 10 segments with balanced lengths.

[0037] Step S20: Extract real-time linear feature contour information from the UAV remote sensing image of the target area, and based on the geographical location information and flight trajectory data of the UAV, divide the real-time linear feature contour into multiple contour segments with the same structural form as those in the satellite remote sensing image.

[0038] To ensure that the low-altitude UAV remote sensing image can establish an accurate correspondence with the satellite remote sensing image in terms of structure and semantics, it is necessary to perform contour extraction and structured segmentation operations on the linear features contained in the UAV image, and perform spatial calibration and segment division on the extraction results based on the geographical location information and flight trajectory data of the UAV. Since there are significant differences in the viewing angle, resolution, and imaging time of the images collected by the UAV and the satellite image, if no unified spatial segment structure constraint is imposed on the linear features, it will be difficult to establish a stable mapping relationship for subsequent structure matching and regional fusion. By extracting linear feature contour segments with the same structural form as those in the satellite image from the UAV image and maintaining the consistency of the geometric division strategy, it helps to improve the success rate of matching between segments and form an effective basis for change recognition in disaster areas.

[0039] The geographical location information of the UAV refers to the geographical coordinates of the camera center point recorded when the UAV takes pictures, generally including longitude, latitude, and flight altitude. The flight trajectory data refers to the sequence of path points formed during the continuous flight of the UAV, which is used to identify the shooting angles, position sequences, and spatial coverage ranges of each region in the image.

[0040] Specifically, the process of extracting linear feature contour information from the UAV remote sensing image and performing segment division includes the following steps: Step S201: Perform preprocessing operations on the UAV image to improve the geometric consistency and spatial usability of the image in subsequent contour extraction and structure division. The preprocessing includes image distortion correction, viewing angle normalization, and spatial positioning enhancement.

[0041] Image distortion correction includes correction processing for the radial distortion and tangential distortion generated by the camera lens. Specifically, it can be achieved by obtaining the internal parameter matrix and distortion parameter set of the camera, and using the inverse mapping function in the pinhole imaging model for pixel remapping to eliminate non-linear distortions such as image edge stretching and compression.

[0042] Viewpoint normalization refers to correcting the geometric distortions caused by the different pitch angles and yaw angles of the images due to the tilting flight or attitude change of the UAV, making it as consistent as possible with the vertical viewing angle of satellite images. Optional implementation methods include calculating the heading angle, pitch angle, and roll angle based on the attitude data collected by the IMU, and using the perspective transformation function of a single image for affine or perspective correction.

[0043] Spatial positioning enhancement means using the GNSS coordinates and timestamp data recorded during the flight of the UAV, combined with the shutter trigger time when the image is taken, projecting the four corners of the image onto the geographic coordinate system, and constructing an image spatial reference system for subsequent contour spatial coding and matching.

[0044] Step S202, extract the linear feature contour information from the preprocessed image. The linear feature contour information is a set of boundary lines of strip-shaped structures with significant direction continuity in the image.

[0045] The basic operations of contour extraction include edge detection and continuity enhancement processing. For edge detection, the Canny edge detection algorithm can be selected. This algorithm includes four stages: Gaussian filtering, gradient calculation, non-maximum suppression, and double-threshold connection, and can effectively extract boundary lines with strong edge responses and clear structures.

[0046] Continuity enhancement can adopt closing operation and thinning processing in image morphology operations to connect the broken edges and shrink the wide edges into single-pixel lines, improving the topological usability of the contour.

[0047] Optional implementation solutions include performing semantic segmentation processing on UAV images using a deep neural network, outputting categories such as roads, rivers, and canal belts in the image in the form of a mask, and performing boundary extraction operations on the mask to obtain a set of contour lines. The network model can include structures such as UNet, DeepLabv3+, and SegFormer that have been trained and completed in remote sensing tasks.

[0048] Step S203, divide the linear feature contours based on the set of contour lines to achieve unity with the contour segment structure in the satellite image.

[0049] The core of segment division is to divide the continuous curve into sub-curves with balanced structures, and each sub-curve has independent geometric description capabilities.

[0050] The division method includes two basic strategies, namely equidistant division and curvature-driven division: Equidistant division means inserting division points on the contour curve at fixed geographical intervals, with each segment having the same length. The interval value can be set according to the average length of the satellite image contour segment to ensure that the two source image segments are consistent in scale; Curvature-driven division means setting segment boundaries at the contour curvature mutation points (i.e., inflection points or bending points). This method calculates the local curvature value at each pixel point. When the change amplitude exceeds the preset threshold, it is set as the segmentation point. This method can preserve the polygon geometric features in complex structural areas.

[0051] To avoid too short segment lengths or too many segments, a minimum segment length threshold and a maximum allowed number of segments can be set.

[0052] In step S204, generate a spatial attribute description for each contour segment for structural comparison and spatial alignment in subsequent matching and fusion operations.

[0053] Spatial attributes include but are not limited to the following: The geographical coordinates of the starting and ending points (calculated from the image projection and GNSS data); The central axis direction vector (used to describe the structural direction consistency); The segment length (used to measure the structural scale); The average gray value and texture gradient (used for image comparison to assist in matching); The shooting angle and flight attitude of the image to which the segment belongs (used to compensate for the perspective difference); Spatial coding tags, including image serial number, flight trajectory timestamp, segment number, etc.

[0054] The above information is stored in a structured data format, such as GeoJSON or Shapefile format, with a coordinate reference system encoding to support direct spatial matching operations between images.

[0055] After being processed by the above sub-steps, the linear features in the UAV images are restored to a set of segment collections with spatial positions, geometric features, and structural tags, and are in a segmented form that is consistent with the satellite image structure in the geographical space, providing a data basis for establishing stable anchor points and identifying change areas in subsequent steps. This processing strategy not only significantly improves the expression consistency of multi-source remote sensing images at the structural level but also enhances the registration and fusion robustness in non-rigid terrain and local change areas, and is applicable to complex scenarios such as disaster response, road fracture monitoring, and river overflow analysis.

[0056] In a specific example, continuing with the example of the previous step, after the structural processing of the drone images, the system identifies a set of contour segments of the main road A, numbered from A1 to A8, where the geometric shapes of A1 to A3 and A6 to A8 are normal, while the segments A4 and A5 are severely deformed due to being covered by debris flow.

[0057] Step S30: Based on the geometric shape similarity index and geographical location matching relationship of the contour segments, perform corresponding matching on the linear feature contour segments in the satellite remote sensing image and the drone remote sensing image, and construct stable contour segment pairs.

[0058] To achieve the precise fusion of the spatial structures between the satellite remote sensing image and the drone remote sensing image, it is necessary to establish the corresponding relationship between the linear feature contour segments with the same geographical entity meaning in the two images. Due to the differences in resolution, viewing angle, and time between the two types of images, without explicit structural segment-level matching, it may lead to misusage of registration anchor points, misjudgment of changed areas, or distortion of the fusion result. By performing one-to-one structure matching based on the geometric shape similarity and geographical location matching relationship of the contour segments, stable segment pairs with spatial consistency and structural stability can be effectively constructed, serving as the basis for subsequent regional grid registration and separation of changed areas.

[0059] Specifically, the matching process for constructing stable contour segment pairs includes the following detailed steps: Step S301: Extract and organize all the divided contour segment sets in the satellite image and the drone image, and denote them as the first segment set and the second segment set respectively.

[0060] In this step, call the contour segment data with spatial attributes generated in the previous step, and establish a segment attribute index structure, including the unique identifier, start and end point coordinates, direction vector, center point, length, curvature sequence, and timestamp of each segment, etc.

[0061] The segment data is loaded into the matching module in a structured format (such as GeoJSON or vector layer).

[0062] Step S302: Conduct a preliminary geographical location pre-screening on the two sets of segment sets to narrow down the range of candidate matching segments.

[0063] In this step, taking the center point of each satellite image contour segment as a reference, check whether the center points of all drone image segments are within the set geographical search radius.

[0064] The search radius is set according to the resolution of the satellite image, the satellite imaging time, and the time interval between the actual acquisitions of the drone, and is generally set between 50 meters and 200 meters.

[0065] Optional implementation solutions include: using a spatial index structure (such as a quadtree or R-tree) to accelerate spatial queries.

[0066] In step S303, perform geometric shape similarity analysis on the initially screened fragment pairs, calculate the similarity index scores, and determine whether they are matching fragment pairs according to a set threshold.

[0067] Specific indicators include the following items: The difference in the included angle of the starting and ending point directions, used to determine whether the trends are consistent; The comparison of the mean curvature difference and the change trend, used to analyze the bending shape of the fragment; The length ratio, used to exclude non-matching items with large scale differences; The shape similarity score can be obtained by cumulatively calculating the Euclidean distance for an equi-length sample point sequence.

[0068] Set corresponding weights for each indicator to form a comprehensive matching scoring system.

[0069] In step S304, perform geometric consistency verification on the fragment pairs with a high comprehensive score to further eliminate false matches caused by local deformation or damage.

[0070] In this step, introduce local grid consistency constraints, that is, among the adjacent fragments around a certain fragment pair, if more than a set proportion of adjacent fragments cannot be successfully matched, the current fragment pair will be marked as an unstable pair.

[0071] The adjacent relationship here can be calculated through a topological adjacency graph or a connection graph between the starting and ending points.

[0072] At the same time, for the areas near some curvature mutation points, a more stringent matching tolerance can be set to avoid misjudging a broken line as a continuous structure.

[0073] In step S305, mark the fragment pairs that pass the geometric verification as stable contour fragment pairs and write them into the fusion anchor point table for subsequent steps to call.

[0074] The fusion anchor point table includes information such as the ID of each pair of fragments, the spatial coordinate pair, the direction pair, the length value, the confidence score, and the image number to which they belong.

[0075] The anchor point pairs provide structural support for grid registration and are used as geometric transformation benchmarks within the matching area.

[0076] Through the above steps, it is possible to screen out pairs of segments with clear structural correspondence, similar geometric shapes, and close spatial positions from a large number of redundant contour segments, serving as registration anchor points between satellite images and UAV images, ensuring that subsequent fusion operations are only performed within structurally stable regions. At the same time, inconsistent regions are separately stripped, enhancing the geometric consistency and temporal expression ability of the overall fused image. Compared with traditional full-image feature point matching methods, this method has stronger structural constraints and better robustness, and is suitable for processing remote sensing multi-source data in scenarios with sudden changes.

[0077] In a specific example, continuing from the previous example steps, when performing contour segment matching on UAV images and satellite images, the system finds that segments A1 to A3 and A6 to A8 of the main road A can be matched one by one with the segments in the satellite image in terms of direction, position, and length, and the matching confidence is greater than the set threshold. However, due to severe road interruption and coverage, segments A4 and A5 fail to match successfully. The system thus defines A1A3 and A6A8 as stable contour segment pairs and writes them into the fusion anchor table, providing a structural basis for subsequent registration grid generation. At the same time, the areas of A4 and A5 are marked as unstable regions to be processed using non-alignment strategies during subsequent fusion.

[0078] Step S40: Using the stable contour segment pairs as the anchoring benchmark, construct a local area grid registration model, and perform geometric registration processing on the stable regions in the satellite remote sensing image and the UAV remote sensing image based on the registration model.

[0079] To achieve high-precision fusion of satellite remote sensing images and UAV remote sensing images within structurally stable regions, it is necessary to establish a geometric registration model in space that can describe the local area transformation relationship. Since a set of stable contour segment pairs with spatial correspondence and structural consistency have been obtained in the previous steps, these segment pairs can be used as the anchoring benchmark to construct a local area grid, and geometric transformation of the satellite image and the UAV image can be performed within this grid structure to achieve registration alignment. This registration model can effectively adapt to geometric differences at the local scale, especially suitable for compensating for local non-linear deformations caused by oblique imaging and attitude disturbances in low-altitude UAV images.

[0080] In this step, the anchoring reference refers to a pair of spatial control points with a clear corresponding relationship formed by matching stable contour segments. These point pairs can serve as input constraint conditions for the transformation model in the geometric space. The local area grid refers to a grid structure formed by dividing the image to be registered with the spatial positions of multiple pairs of anchoring segments as nodes. This structure supports deformation modeling based on segmented regions. Geometric registration refers to aligning the positions of the same geographical entity in two images under different image coordinate systems through a spatial transformation function, so that the image content has consistency at the pixel level. Common geometric transformations include affine transformation, projective transformation, thin plate spline deformation (TPS), etc.

[0081] Specifically, the process of constructing a grid registration model based on pairs of anchoring segments and completing the registration of stable regions includes the following sub-steps: Step S401, extract the set of control points for registration anchoring from the pairs of stable contour segments as the spatial reference for the subsequent local grid registration model.

[0082] First, traverse all the determined pairs of stable contour segments. For each pair of segments, extract their geometric center points as anchoring control points; The geometric center point can be obtained by calculating the midpoint between the starting point and the ending point of the segment, or the actual geometric center point position can be obtained through the curve length integration method; If the segment length is relatively long, to increase the grid density, multiple equally spaced sampling points can be selected at fixed intervals on the segment, and each point serves as a local anchor point; At the same time, record the pixel coordinates and geographical coordinates of each control point in the satellite image and the UAV image respectively, and establish a one-to-one correspondence; Each pair of control points is stored in the form of (satellite image coordinates, corresponding UAV image coordinates) as an anchor point pair structure to form a set of control point pairs; The set is stored in the form of a structured array and can be optionally written into the GeoJSON format or the Shapefile vector data format for subsequent spatial modeling.

[0083] Step S402, construct a local area grid structure based on the set of control point pairs to support region division and local registration model fitting.

[0084] Take the satellite image coordinates of all control points as node inputs and construct a spatial grid structure based on their two-dimensional plane positions; Optional grid construction methods include but are not limited to: Use the Delaunay triangulation method to triangulate the control points to obtain a topologically stable triangular grid without generating acute angles; Use the regular grid division method to construct a rectangular grid with a fixed side length within the distribution range of the control points and map the control points to the grid nodes; Build a local index for each grid cell (whether triangular or rectangular), record the control points it contains, the spatial boundaries, and the region numbers it belongs to; At the same time, use the corresponding control point pairs in the drone image as target nodes to establish a transformed target grid in the target image space; Establish the "source grid - target grid" mapping relationship between the starting image and the target image, and each cell has a unique registration correspondence; The grid structure is stored in the form of a topological graph structure, supporting node update, edge weight assignment, and local interpolation.

[0085] Step S403: Establish a local geometric transformation model within each grid cell and perform geometric registration processing on the stable region image patches.

[0086] For each grid cell, take all the control point pairs it contains, and construct a local geometric transformation function based on the spatial coordinate differences between the source grid and the target grid; The optional implementation methods of the transformation function include: Affine transformation model: Suitable for small - scale linear deformation regions, by calculating the affine matrix between control points; Perspective transformation model: Suitable for the projection distortion caused by low - altitude oblique imaging; Thin - plate spline interpolation model: Suitable for slightly non - linear distortion regions, with strong local deformation expression ability; Before performing geometric transformation, first crop the corresponding original image patches in the satellite image according to the grid cell range; Perform pixel - level remapping on the cropped image patches and output new image patches with the same spatial structure as the drone image space; During the remapping process, use bilinear interpolation method to maintain image details, or use Lanczos interpolation method to improve the edge accuracy of the image; After each image patch is transformed, record its transformation parameters and image position index for full - map reconstruction.

[0087] Step S404: Perform fusion reconstruction operations on all registered image patches to form a geometrically consistent image output result for the stable region.

[0088] Rearrange all the registered image patches to their corresponding positions in the output image according to their original grid numbers; Perform edge smoothing and stitching operations between the grid cell boundaries to avoid image breaks caused by transformation differences; Smoothing methods include Gaussian mixture, edge blur processing, weighted average fusion of transition regions, etc.; Perform coordinate verification on the reconstructed region to ensure the consistency of the output image and the original image in the georeference system; The output image is exported in GeoTIFF format or other geocoded image formats, along with coordinate reference information (such as EPSG code and affine matrix). The stable region fusion result will serve as the background layer for the subsequent steps of change region expression and overall fusion map generation.

[0089] Through the above steps, using the control points formed by the stable contour segments as registration anchor points, a highly consistent image grid transformation model is constructed within the structurally stable region, achieving high-precision geometric alignment between the UAV remote sensing image and the satellite image. This method has the advantages of strong scalability, strong local deformation expression ability, and high fusion accuracy. It is particularly suitable for multi-source remote sensing image fusion tasks in non-rigid and non-globally symmetric scenarios, effectively avoiding the problem of full-map registration failure caused by sudden disaster changes.

[0090] Specifically, by way of example, continuing with the aforementioned mountain landslide scenario, based on the anchor points generated in the stable sections A1 to A3 and A6 to A8, the system constructs a Delaunay local grid structure containing 18 triangular elements. Subsequently, the system fits an affine transformation matrix within each element to align the local area of the satellite image with the corresponding area of the UAV image. After all elements are completed, an satellite reference base map consistent with the UAV structure is reconstructed through edge fusion, providing a comparison benchmark for the change expression of the landslide sections A4 and A5.

[0091] Step S50, for the image regions where stable contour segment pairs cannot be established, they are determined as regions where the ground object state has changed; within the changed regions, a non-aligned fusion strategy mainly based on the UAV remote sensing image is adopted. Based on the relative spatial relationship between the UAV flight trajectory and the linear ground object, the observation angle of the UAV image in the changed region is adjusted, and the image after angle adjustment is fused with the satellite remote sensing image.

[0092] During the fusion process of satellite remote sensing images and low-altitude UAV remote sensing images, there are some regions where due to significant changes in the ground object morphology, an effective contour segment matching relationship cannot be established between the two-source images. Such regions are usually areas affected by disasters or emergencies, such as roads washed away by debris flows, rivers covered by landslides, etc. In this case, if a registration or alignment strategy is still used for fusion, it may instead introduce spatial distortion or semantic errors. Therefore, a non-aligned fusion strategy mainly based on the UAV image needs to be adopted, and combined with the spatial relationship between the UAV flight trajectory information and the ground object structure, the image angle is adjusted to improve the authenticity and semantic continuity of the fusion effect, so that the fusion result can accurately express the actual state of the ground object change region.

[0093] In this step, the changed area refers to the image area in the satellite image and the UAV image where there is no effective pairing relationship for linear features at the same geographical location. This area may have significantly different image contents due to landform changes, occlusion, or the generation of new structures. The non-aligned fusion strategy means that in the case of drastic changes in image content and invalid registration relationships, the geometric registration operation is abandoned, and the UAV image is directly used as the main image and fused into the full image in the form of an optimized observation angle. The flight trajectory refers to the continuous sequence of spatial coordinates recorded by the UAV during the shooting mission, usually generated by the GNSS positioning system, which can be used to determine the shooting position and attitude direction corresponding to each frame of the image. The observation angle adjustment refers to correcting the projection angle of the UAV image based on the relative spatial relationship between the trajectory points and the ground object objects to improve the structural expression ability and visual consistency of the fused image.

[0094] Specifically, the non-aligned fusion operation for processing the changed area includes the following sub-steps: Step S501, identify the image areas where stable contour segment pairs cannot be established and mark them as areas where the ground object state has changed.

[0095] Traverse all UAV contour segment records to find the segments that have not successfully established a pairing relationship with any satellite segments in the contour segment matching step.

[0096] For each unmatched segment, obtain its boundary coordinate information and construct a minimum bounding rectangle or polygon contour to describe the range of the image area covered by the segment.

[0097] Perform spatial merging processing on multiple adjacent or continuous unmatched segments, and use topological merging rules to combine their contour boundaries to form one or more complete changed area masks.

[0098] The changed area mask is a two-dimensional spatial mask layer, saved in bitmap format or vector format, and is used to identify the target areas in the image that need to perform non-aligned fusion processing.

[0099] At the same time, assign a unique identification code and attribute information to each changed area, including the associated unmatched segment numbers, estimated change degree (based on gray difference or texture mutation), area of the area, center coordinates, etc.

[0100] Step S502, locate the corresponding UAV image source within the changed area and extract the flight trajectory data when shooting this area.

[0101] According to the spatial boundary coordinates of the changed area, retrieve the image numbers containing this area in the UAV image database. The retrieval methods include spatial index query and image coverage relationship matching; The geographic coverage of a drone image can be defined by a polygon composed of the geographic coordinates of the four corners of the image; After confirming the image containing the changed area, extract the flight trajectory data at the moment when the image was taken from the flight log; The trajectory data includes content such as longitude and latitude coordinates, altitude value, heading angle, pitch angle, roll angle, flight speed, image frame number, etc., and is usually jointly recorded by GNSS and IMU modules, with the format being standard CSV or dedicated binary flight control log; At the same time, calculate the distance and direction between the center point of the image and the center point of the changed area for subsequent perspective adjustment calculation; All trajectory and image matching relationships are recorded in the image metadata table and bound to the changed area mask.

[0102] Step S503, based on the spatial relationship between the flight trajectory and the direction of the linear feature, calculate the viewing angle of the changed area and adjust the angle of the drone image.

[0103] First, extract the expected trend direction of the features within the changed area, which can be estimated by the direction vectors of adjacent registered segments or obtained through the region boundary fitting algorithm; Calculate the angle between the shooting direction (i.e., the heading angle) of the drone image and the trend of the feature. If the angle is greater than the set threshold (such as 30 degrees), then perform perspective adjustment; Optional implementation methods for image perspective adjustment include: Perspective transformation method: Based on the geographic coordinates of the four corners of the image and the expected top-down transformation matrix, calculate the projection mapping of the image in the two-dimensional space, which is suitable for correcting slightly tilted shooting; Affine transformation method: Remap the image to the target direction coordinate system through linear translation, scaling, and rotation; 3D modeling and orthographic projection method: When the flight attitude is complex or the regional height difference is large, use 3D surface reconstruction based on DSM or point cloud data to generate a standard orthographic image; During the adjustment process, the spatial resolution of the image should be kept unchanged, and the edge area should be filled with zeros or using an edge extrapolation algorithm; The adjusted image regenerates the image index and spatial reference information and is saved as an independent image data block.

[0104] Step S504, perform non-aligned image fusion operation within the changed area to generate the local update result of the fused image.

[0105] For each changed area, according to its spatial mask, intercept the image block corresponding to the area in the adjusted drone image; The corresponding satellite image block is extracted from the original fused background map as the fusion reference; The fusion method can select different strategies according to business requirements, and the optional implementation solutions include: Dominant substitution fusion: Completely cover the original satellite image area with UAV image patches; Weighted fusion: The UAV image and the satellite image are weighted and averaged at the pixel level, and the weights are dynamically adjusted according to texture complexity or grayscale mean; Semantic fusion: Under the guidance of the semantic segmentation result, only replace the target category areas (such as roads, water bodies), and keep the rest; At the fusion boundary, perform image patch fusion stitching processing, and use techniques such as edge gradient, convolution blur, and Laplacian pyramid to smooth the transition area; Embed the fusion result into the full map to ensure coordinate consistency and pixel continuity; Finally, generate a fused image and output additional data such as the location layer of the changed area and the fusion quality assessment report.

[0106] Through the above steps, the system realizes a non-aligned fusion method dominated by UAV images in areas with drastic changes in ground features, completes image attitude correction using the spatial relationship between the flight trajectory and the direction of ground features, and performs a replacement expression of the changed area based on high-resolution and low-latency images. This method effectively solves the problem that the conventional registration model fails in disaster areas, and improves the adaptability, expression ability and engineering practicability of the image fusion system in complex changing scenarios.

[0107] In a specific example, continuing the previous example of mountain landslide, the areas where segments A4 and A5 are located are identified and defined as changed areas. The system retrieves and finds that the corresponding UAV image number is IMG_0785, extracts its flight heading angle as the southeast direction, and analyzes that the included angle between this direction and the road alignment exceeds 45 degrees. Therefore, the system calls the perspective transformation module to perform projection correction on this image. Subsequently, the system performs dominant substitution fusion on the changed area, covers the section covered by the landslide with clear and high-resolution UAV image patches to the fused image, and uses the Laplacian pyramid fusion method for boundary transition at the edge. Finally, the generated image completely expresses the spatial change details such as road fracture and landslide coverage, meeting the needs of disaster emergency analysis and subsequent patch identification.

[0108] For the part of the module structure not specifically defined in the present invention, it shall be subject to the content recorded in the prior art. The prior art mentioned in the foregoing background art part and specific embodiment part of the present invention can be used as a part of the present invention to understand the meaning of some technical features or parameters.

Claims

1. A method for fusing satellite and low-altitude UAV remote sensing data, characterized in that, The method includes the following steps: Step S10, extract the linear feature contour information from the satellite remote sensing image of the target area. The linear feature is a linear structure with spatial extensibility, and its length is greater than the first preset threshold; Divide the linear feature contour into multiple contour segments; Step S20, extract the real-time linear feature contour information from the UAV remote sensing image of the target area, and based on the geographical location information and flight trajectory data of the UAV, divide the real-time linear feature contour into multiple contour segments with the same structural form as that in the satellite remote sensing image; Step S30, based on the geometric shape similarity index and geographical location matching relationship of the contour segments, perform corresponding matching on the linear feature contour segments in the satellite remote sensing image and the UAV remote sensing image to construct stable contour segment pairs; Step S40, use the stable contour segment pairs as the anchoring benchmark, construct a local area grid registration model, and perform geometric registration processing on the stable areas in the satellite remote sensing image and the UAV remote sensing image based on the registration model; Step S50, for the image area where stable contour segment pairs cannot be established, determine it as the area where the feature state has changed; in the changed area, adopt a non-aligned fusion strategy mainly based on the UAV remote sensing image, and based on the relative spatial relationship between the UAV flight trajectory and the linear feature, adjust the viewing angle of the UAV image in the changed area, and perform fusion processing on the image with the adjusted angle and the satellite remote sensing image.

2. The method according to claim 1, wherein The step S10 includes the following sub-steps: Step S101, perform image enhancement processing on the original satellite remote sensing image, including using histogram equalization to enhance the image contrast; Step S102, use the Canny edge detection algorithm to extract the edge area, and use the Hough transform to detect the linear feature contour with spatial continuity; Step S103, based on the geometric skeleton or boundary length, screen out the short line segments smaller than the first preset threshold; Step S104, adopt a combination of equidistant division and curvature change division to divide the remaining linear features into multiple contour segments; Step S105, generate spatial attributes for each contour segment, including start and end point coordinates, length, direction, curvature characteristics, and coding identifiers.

3. The method according to claim 2, wherein In the step S104, the sampling interval of the equidistant division is set to 10 meters to 50 meters, and the identification of the curvature mutation point is determined by the local curvature difference being greater than the set threshold, ensuring the continuity and comparability of the divided segments in terms of spatial form.

4. The method according to claim 1, wherein The step S20 includes the following sub-steps: Step S201, perform image distortion correction, attitude normalization, and GNSS-assisted spatial positioning enhancement on the UAV image; Step S202, use a combination of Canny edge detection and morphological closing operation to extract the directional linear feature contour; Step S203, use the combination rule of equidistant length and curvature threshold to divide the contour into multiple segments with the same structure; Step S204, attach spatial attribute descriptions to each contour segment, including image number, heading angle, position code, and geometric feature vector.

5. The method according to claim 4, characterized in that, The spatial attributes of each segment in step S204 include waypoint timestamps, which are used to establish consistency judgments in the subsequent registration and change recognition in the time series.

6. The method according to claim 1, wherein Step S30 includes the following sub-steps: Step S301: Extract the satellite image segment set and the UAV image segment set, and establish a segment spatial index. Step S302: Perform candidate matching screening based on the geographic center point within a set radius. Step S303: Construct a comprehensive similarity scoring function using direction difference, curvature matching, length ratio, and shape similarity. Step S304: Eliminate false matching segment pairs based on segment adjacency consistency. Step S305: Record the high-scoring segment pairs as stable contour segment pairs and write them into the anchor point matching data table.

7. The method according to claim 6, characterized in that, The weight coefficients of the comprehensive scoring function in step S303 can be optimized and set through the training set to improve the matching accuracy and avoid segment misregistration in complex terrain areas.

8. The method according to claim 1, wherein Step S40 includes the following sub-steps: Step S401: Extract the geometric midpoints from the stable contour segment pairs as control anchor points and record their corresponding image coordinate pairs. Step S402: Construct a Delaunay triangulation grid based on the control anchor points and establish the mapping relationship between the source image and the target image. Step S403: For each grid cell, use the affine or thin plate spline interpolation method to generate a local geometric transformation function and perform pixel remapping on the local block of the satellite image. Step S404: Use the image stitching and edge fusion methods to generate a consistent stable region image output.

9. The method according to claim 1, wherein Step S50 includes the following sub-steps: Step S501: Identify all regions where stable segment pairs have not been established and construct their two-dimensional boundary masks. Step S502: Match the change region with the UAV image through the spatial coverage relationship, and extract the corresponding flight trajectory data and shooting postures. Step S503: Based on the angle between the flight trajectory direction and the ground object direction, determine whether to perform image perspective correction, and use the perspective transformation or affine transformation method to adjust the angle of the image. Step S504: Use the dominant substitution, weighted fusion, or semantic-guided fusion method to cover the adjusted image block to the satellite image change region to complete the fusion operation.

10. The method according to claim 9, wherein When performing image fusion in step S504, the Laplacian pyramid fusion method is used for multi-scale transition at the fusion boundary to enhance the naturalness and visual consistency of the fusion region boundary.

Images