Maritime remote sensing surveying and mapping method and surveying and mapping system
Through feature point matching and stitching algorithms, the problem of image resolution and field of view balance in drone maritime remote sensing surveying and mapping is solved, and the image stitching accuracy and reliability of maritime remote sensing surveying and mapping is improved, and it is suitable for complex ocean field of view data.
Patent Information
- Application Number
- CN202411222356.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-12-06
AI Technical Summary
In drone maritime remote sensing surveying and mapping, traditional image stitching technology has limitations, and it is difficult to effectively balance image resolution and field of view, resulting in low efficiency in sea target recognition and analysis.
Multi-step feature point matching and stitching methods are adopted, including preprocessing, feature point extraction, initial matching, quadratic matching and stitching, and SIFT, FLANN, RANSAC and DLT algorithms are used to ensure high-precision image stitching.
It improves the reliability and accuracy of the final stitching image of marine remote sensing mapping, is suitable for complex and changing ocean field of view data, and provides more efficient ocean mapping tools.
Smart Images

Figure CN119090716A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of surveying and mapping technology, and in particular to a marine remote sensing surveying and mapping method and a surveying and mapping system. Background Art
[0002] With the rapid development of UAV aerial survey technology, UAVs (Unmanned Aerial Vehicle) have the advantages of strong maneuverability, high flexibility and no obstacles at high altitudes, making them more suitable for marine surveying and mapping operations. They are increasingly used in areas such as updating land elements of nautical charts and dynamic supervision of sea areas.
[0003] In the application of UAV remote sensing mapping at sea, the balance between image resolution and field of view is a key technical challenge. UAV platforms are usually equipped with high-resolution cameras that can provide clear and detailed images, which are essential for the identification and analysis of maritime targets. However, high-resolution imaging is usually accompanied by a small field of view, which means that in order to cover a larger area of the sea, UAVs need to take more images and merge these images into a complete large-scale map through image stitching technology.
[0004] Image stitching is a technology that solves the limitation of image field of view. It stitches multiple overlapping images into an image with a wide field of view. However, due to the particularity of marine surveying and mapping, the image stitching technology used in traditional remote sensing surveying and mapping has limitations when applied to marine remote sensing surveying and mapping. Summary of the invention
[0005] In order to overcome the above technical defects, the present invention provides a marine remote sensing surveying and mapping method and a surveying and mapping system.
[0006] In order to solve the above problems, the present invention is implemented according to the following technical solutions:
[0007] In a first aspect, the present invention provides a method for remote sensing mapping at sea, comprising the following steps:
[0008] Acquire multiple remote sensing images, where the remote sensing images are obtained by photographing a target ocean area by a drone;
[0009] Preprocessing the remote sensing image to obtain a preprocessed remote sensing image;
[0010] Extracting a plurality of feature points from the preprocessed remote sensing image;
[0011] Performing initial matching on the feature points of two pre-processed remote sensing images to be spliced with overlapping areas to obtain multiple initial feature point pairs;
[0012] Perform secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs;
[0013] Based on the precise feature point pairs, two pre-processed remote sensing images to be stitched with overlapping areas are stitched to output a stitched remote sensing image.
[0014] In combination with the first aspect, the present invention also provides a first preferred implementation scheme of the first aspect, specifically, multiple feature points are extracted from the preprocessed remote sensing image, specifically, multiple feature points are extracted from the preprocessed remote sensing image based on the SIFT algorithm.
[0015] In combination with the first aspect, the present invention also provides a second preferred implementation manner of the first aspect, specifically, performing an initial matching to obtain multiple initial feature point pairs, specifically, performing an initial matching of multiple feature points based on the FLANN algorithm to obtain multiple initial feature point pairs.
[0016] In combination with the first aspect, the present invention also provides a third preferred implementation manner of the first aspect, specifically, performing secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs, specifically, performing secondary matching on multiple initial feature point pairs based on the RANSAC algorithm to obtain accurate feature point pairs.
[0017] In combination with the first aspect, the present invention further provides a fourth preferred implementation of the first aspect, specifically, based on the precise feature point pairs, two pre-processed remote sensing images to be stitched having overlapping areas are stitched, and a stitched remote sensing image is output, specifically comprising the following steps:
[0018] Using the pixel coordinates of the feature points of the precise feature point pairs, the global projection transformation matrix is solved based on the DLT algorithm;
[0019] Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model;
[0020] The global projection transformation model is used to stitch two pre-processed remote sensing images with overlapping areas to output a stitched remote sensing image.
[0021] In a second aspect, the present invention provides a marine remote sensing mapping system, comprising:
[0022] An acquisition module, which is used to acquire multiple remote sensing images, wherein the remote sensing images are obtained by photographing the target ocean area by a drone;
[0023] A preprocessing module, which is used to preprocess the remote sensing image to obtain a preprocessed remote sensing image;
[0024] An extraction module, which is used to extract multiple feature points from the pre-processed remote sensing image;
[0025] A first matching module is used to perform initial matching on feature points of two pre-processed remote sensing images to be spliced with overlapping areas to obtain a plurality of initial feature point pairs;
[0026] A second matching module is used to perform secondary matching on the multiple initial feature point pairs to obtain accurate feature point pairs;
[0027] The stitching module is used to stitch two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and output a stitched remote sensing image.
[0028] In combination with the second aspect, the present invention also provides a first preferred implementation scheme of the second aspect, specifically, the extraction module extracts multiple feature points from the preprocessed remote sensing image, specifically, extracts multiple feature points from the preprocessed remote sensing image based on the SIFT algorithm.
[0029] In combination with the second aspect, the present invention also provides a second preferred implementation scheme of the second aspect. Specifically, the first matching module performs an initial matching to obtain multiple initial feature point pairs. Specifically, based on the FLANN algorithm, multiple feature points perform an initial matching to obtain multiple initial feature point pairs.
[0030] In combination with the second aspect, the present invention also provides a third preferred implementation scheme of the second aspect. Specifically, the second matching module performs secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs. Specifically, based on the RANSAC algorithm, secondary matching is performed on multiple initial feature point pairs to obtain accurate feature point pairs.
[0031] In combination with the second aspect, the present invention further provides a fourth preferred implementation of the second aspect, specifically, the stitching module stitches two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and outputs a stitched remote sensing image, specifically comprising the following steps:
[0032] Using the pixel coordinates of the feature points of the precise feature point pairs, the global projection transformation matrix is solved based on the DLT algorithm;
[0033] Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model;
[0034] The global projection transformation model is used to stitch two pre-processed remote sensing images with overlapping areas to output a stitched remote sensing image.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] The invention provides a marine remote sensing mapping method, comprising the following steps: acquiring a plurality of remote sensing images, wherein the remote sensing images are obtained by photographing a target ocean area by an unmanned aerial vehicle; preprocessing the remote sensing images to obtain a preprocessed remote sensing image; extracting a plurality of feature points from the preprocessed remote sensing image; performing a primary matching on feature points of two preprocessed remote sensing images to be spliced with overlapping areas to obtain a plurality of initial feature point pairs; performing a secondary matching on the plurality of initial feature point pairs to obtain precise feature point pairs; splicing the two preprocessed remote sensing images to be spliced with overlapping areas based on the precise feature point pairs, and outputting a spliced remote sensing image.
[0037] The main innovation of the present invention is to use the primary matching and secondary matching algorithms to accurately identify the feature point pairs of two remote sensing images to be spliced, and even the remote sensing image data of the complex and changing ocean field can maintain a high matching rate. Based on the marine remote sensing mapping method of the present invention, the reliability and accuracy of the final spliced image are greatly improved, providing a new tool for marine mapping. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The specific embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings, wherein:
[0039] Figure 1 It is a flow chart of a method for offshore remote sensing mapping of the present invention. DETAILED DESCRIPTION
[0040] The following specifically illustrates the implementation mode of the present invention in conjunction with the accompanying drawings. The embodiments are provided for illustrative purposes only and cannot be understood as limiting the present invention. The accompanying drawings are provided for reference and illustration only and do not constitute a limitation on the scope of patent protection of the present invention, because many changes can be made to the present invention without departing from the spirit and scope of the present invention.
[0041] refer to Figure 1 The embodiment of the present invention provides a flow chart of a method for remote sensing surveying and mapping at sea. The present invention is mainly based on remote sensing data obtained by shooting a target ocean area by a drone. After preprocessing the remote sensing data, extracting feature points, matching feature points and stitching images, the remote sensing data is formed at sea, providing a new tool for marine surveying and mapping.
[0042] The present invention is used for remote sensing mapping at sea. The method can be performed by a remote sensing mapping system at sea. The system can be implemented in the form of hardware and / or software. The system can be configured in an electronic device. Figure 1 As shown, the method includes:
[0043] S100: Acquire multiple remote sensing images, where the remote sensing images are obtained by photographing a target ocean area using a drone.
[0044] S200: Preprocessing the remote sensing image to obtain a preprocessed remote sensing image.
[0045] S300: extracting a plurality of feature points from the pre-processed remote sensing image.
[0046] S400: performing initial matching on feature points of two pre-processed remote sensing images to be spliced that have overlapping areas to obtain a plurality of initial feature point pairs.
[0047] S500: Perform secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs.
[0048] S600: Based on the precise feature point pairs, two pre-processed remote sensing images to be stitched having overlapping areas are stitched together to output a stitched remote sensing image.
[0049] The present invention provides a tool for marine surveying and mapping. The main innovation of the present invention is to use the primary matching and secondary matching algorithms to accurately identify the feature point pairs of two remote sensing images to be spliced, and even the remote sensing image data of the complex and changing ocean field can maintain a high matching rate. Based on the marine remote sensing surveying and mapping method of the present invention, the reliability and accuracy of the final spliced image obtained are greatly improved. Specifically, this embodiment provides a detailed description of each step.
[0050] S100: Acquire multiple remote sensing images, where the remote sensing images are obtained by photographing a target ocean area using a drone.
[0051] In a specific implementation, the Dapeng CW-10C 1:500 image-free aerial survey system can be used to obtain remote sensing images. The Dapeng CW-10C 1:500 image-free aerial survey system is a complete solution that integrates the CW-10C vertical take-off and landing fixed-wing drone of Zongheng Dapeng UAV Technology Co., Ltd., post-differential GPS, and the Tiangong image-free aerial survey data processing software of Xuntu Technology Co., Ltd. It has the functions of automatic route generation, autonomous take-off and landing and flight control, real-time monitoring of flight status, high-precision POS data generation, on-site data quality inspection, one-button image-free aerial three-solution, etc. It can be applied to the basic geographic information data collection of large areas of islands and coastal zones, dynamic monitoring of coastal engineering, digital city construction, land and resources survey, geological environment survey, disaster emergency response and other fields.
[0052] In the specific implementation, the general workflow of the drone to collect remote sensing images is as follows: first, turn on the ground workstation, set the longitude and latitude of the flight endpoint, flight speed, flight altitude and other parameters, and plan the route; then turn on the flight control system, upload the route to the flight control system, set the parameters for automatic takeoff and landing, and then check whether the data transmission system and the load system are operating normally; after the test results are normal, adjust the camera pitch angle (positive value for the lens upward and negative value for the lens downward), set the camera to the timed shooting mode, and then take off the drone to perform the flight mission. It should be noted that the optical camera takes images of the sea surface at a fixed time interval. In order to meet the requirements of the system memory and take into account the delay of the drone image transmission and data transmission system during the actual shooting process, the sampling time interval of the sea surface image data is greater than 2 seconds. During the execution of the route, the drone flies at a constant speed at a pre-set speed. The flight attitude data (pitch angle, roll angle, heading angle, altitude) and camera shooting parameters (camera pitch angle, heading angle, shooting time, longitude and latitude) are measured by the inertial measurement unit and recorded back through the data transmission system. The shooting time is directly recorded by the camera. When shooting the sea surface, the pitch angle of the camera is approximately -60°, and the angle between the main optical axis of the camera and the normal direction of the average sea level is about 30°.
[0053] Preferably, the camera payload carried by the drone is preferably photographed using a multi-camera device. In the specific implementation, a DJI Phantom 4 Multispectral Edition drone is used, and its onboard camera is used to obtain the original data set to be stitched. In this field, the DJI Phantom 4 Multispectral Edition drone is a simple and easy-to-use drone remote sensing system with a powerful single-frequency high-sensitivity GNSS, a multi-frequency and multi-system high-precision RTK GNSS receiver, a color sensor for visible light imaging, and 5 monochrome sensors for multispectral imaging. The effective pixels of a single sensor are 2.08 million, and aerial images can be acquired fully automatically. Its ground sampling distance (GSD) is (H / 18.9) cm / pixel, and H is the flight altitude of the aircraft relative to the mapping area (unit: meter).
[0054] S200: Preprocessing the remote sensing image to obtain a preprocessed remote sensing image.
[0055] When using drones for ocean mapping, it is impossible to collect images only under good weather conditions, so preprocessing is a very necessary step. In specific implementations, preprocessing the remote sensing images specifically includes distortion correction, radiation correction and image denoising preprocessing operations performed in sequence.
[0056] (i) In a specific implementation, the present invention adopts Zhang Zhengyou calibration method to perform distortion correction on remote sensing images. After distortion correction, the image clarity is enhanced and more in line with actual ground features.
[0057] In this field, Zhang Zhengyou calibration method is a camera calibration technology widely used in the field of computer vision. This calibration method mainly calibrates the camera's intrinsic parameters and distortion parameters by observing different viewing angles of a plane checkerboard calibration plate. Its core advantage is that it does not require expensive calibration equipment, is easy to operate, and has high accuracy and flexibility.
[0058] When performing the Zhang Zhengyou calibration method, you first need to prepare multiple calibration plate images taken at different positions and angles. The calibration plate is usually a black and white checkerboard pattern, and the coordinates of its inner corner points are detected in the image and used in the calibration process. Using a computer vision library such as OpenCV, the calibration process can be easily implemented. The calibration process mainly includes the following steps:
[0059] Corner detection: Use the findChessboardCorners function to detect chessboard corners in the image. Sub-pixel precision: Use the cornerSubPix function to perform sub-pixel precision on the corner position to improve calibration accuracy. Camera calibration: Use the calibrateCamera function to calculate the camera's intrinsic parameter matrix and distortion coefficients based on the image coordinates of the corners and the corresponding world coordinates. Distortion correction: Use the undistort function or initUndistortRectifyMap combined with the remap function to correct the image distortion.
[0060] The output of Zhang Zhengyou's calibration method includes the camera's intrinsic parameter matrix, distortion coefficients, and the rotation and translation vectors of each calibrated image. The intrinsic parameter matrix contains information such as the focal length and principal point coordinates of the camera, while the distortion coefficients describe the radial and tangential distortions of the image. After the calibration is completed, the reprojection error can be used to evaluate the accuracy of the calibration result. The reprojection error refers to the distance error between a point in the three-dimensional world coordinate system and the origin after projecting it into the image coordinate system and then back-projecting it back into three-dimensional space. A smaller reprojection error indicates that the calibration result is more accurate.
[0061] (ii) In the specific implementation, the histogram matching method is used to perform radiation correction on the remote sensing images of the drone. The histogram matching method is a commonly used image processing technology that is used to adjust the histogram of an image to match the histogram of a reference image. For remote sensing images acquired by drones, radiation correction is to ensure that the radiation value of the image can truly reflect the actual radiation characteristics of the ground object, thereby improving the comparability of the image and the accuracy of analysis.
[0062] In specific implementation, the steps of the histogram matching method are as follows:
[0063] 1. Select a reference image: Select an image with ideal radiation characteristics as the reference image. The histogram of this image represents the radiation characteristics that the adjusted image should have.
[0064] 2. Calculate the histogram: Calculate the histogram of the remote sensing image to be corrected acquired by the drone. Calculate the histogram of the reference image.
[0065] 3. Histogram Cumulative Distribution Function (CDF): Calculate the cumulative distribution function (CDF) of the image to be corrected and the reference image.
[0066] 4. Establish a mapping relationship: Establish a mapping relationship based on the cumulative distribution function of the image to be corrected and the cumulative distribution function of the reference image. The goal of this step is to adjust the pixel values of the image to be corrected so that its histogram matches the histogram of the reference image.
[0067] 5. Apply mapping relationship: Apply the mapping relationship to the image to be corrected and adjust its pixel values so that the radiation characteristics of the image are more consistent with the reference image.
[0068] In practical applications, the histogram matching method plays a very obvious role in remote sensing image radiation correction. It can effectively reduce the radiation differences caused by different imaging conditions (such as illumination, sensor response, etc.), improve the comparability of data, and provide a more reliable data basis for subsequent image classification, target recognition and change detection.
[0069] (III) Preprocessing remote sensing images is the first step in image analysis and processing, and denoising is a very important part of it. Remote sensing images may be affected by sensor noise, atmospheric interference, and other external factors during the acquisition process, resulting in noise in the image. These noises will affect subsequent image analysis and processing. Therefore, denoising is an important step to improve the quality of remote sensing images.
[0070] In the specific implementation, bilateral filtering is used to denoise the remote sensing image. Bilateral filtering smoothes the image by considering both the similarity of pixel values and spatial proximity, thereby removing noise while retaining the edge.
[0071] S300: extracting a plurality of feature points from the pre-processed remote sensing image.
[0072] In a specific implementation of the present invention, a plurality of feature points are extracted from the preprocessed remote sensing image. Specifically, a plurality of feature points are extracted from the preprocessed remote sensing image based on a SIFT algorithm.
[0073] In the present invention, the SIFT (Scale-Invariant Feature Transform) algorithm is used to extract feature points from pre-processed remote sensing images, which is an important step in tasks such as image matching, target recognition, and image stitching. The SIFT algorithm detects key points in the image and describes these key points so that the feature points are invariant to scale, rotation, and even some affine transformations. Specifically, the following steps are included:
[0074] (1) Scale-Space Construction:
[0075] By performing Gaussian blurring on remote sensing images at different scales, a series of images of different scales are generated, which constitute the scale space. Each pair of adjacent scale images is subtracted to generate a difference Gaussian pyramid (Difference of Gaussian, DoG).
[0076] (2) Keypoint Detection:
[0077] In the differential Gaussian pyramid, the SIFT algorithm detects potential key points by comparing each pixel with its 8 neighboring pixels at the current scale and adjacent scales and the extreme points in the 3×3 region. Key points with strong edge response but poor stability are deleted.
[0078] (3) Accurately locate key points (Keypoint Localization):
[0079] Each key point is precisely located, and the Taylor expansion is used to adjust the key point position at the sub-pixel level to eliminate unstable key points. The low-contrast key points caused by edge effects are further removed by calculating the ratio of the principal curvatures.
[0080] (4) Orientation Assignment:
[0081] The gradient direction histogram is calculated for the neighborhood of each key point, and the main direction is used as the directional feature of the key point. Through the direction assignment, the feature point is made invariant to rotation.
[0082] (5) Generate key point descriptor (Keypoint Descriptor Generation):
[0083] With the key point as the center, the gradient histogram is calculated in its neighborhood to generate a vector describing the key point features (i.e., feature descriptor). This feature descriptor is a 128-dimensional vector that can effectively describe the local image features of the key point.
[0084] (6) Feature point extraction:
[0085] The key points and corresponding feature descriptors obtained by the above processing are used as feature points of the image and retained for subsequent image processing tasks.
[0086] In remote sensing image processing, the SIFT algorithm can be used for tasks such as multi-temporal image matching, change detection, and target recognition. The scale and rotation invariance of SIFT feature points make it perform well in processing remote sensing images, especially in matching images acquired under different viewing angles and lighting conditions. By combining denoising and SIFT feature extraction in the preprocessing step, more robust and stable image features can be obtained, providing strong support for subsequent remote sensing data analysis and processing.
[0087] S400: performing initial matching on feature points of two pre-processed remote sensing images to be spliced that have overlapping areas to obtain a plurality of initial feature point pairs.
[0088] In the present invention, a plurality of initial feature point pairs are obtained by performing initial matching, specifically, a plurality of feature points are matched initially based on the FLANN algorithm to obtain a plurality of initial feature point pairs.
[0089] In the specific implementation, the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm is used to match the feature points of two pre-processed remote sensing images with overlapping areas, which can quickly and efficiently find the initial feature point pairs. FLANN is an algorithm library for large-scale feature matching, especially suitable for processing the nearest neighbor search of high-dimensional feature descriptors (such as SIFT descriptors). It significantly speeds up the nearest neighbor query by building an efficient data structure, and is particularly suitable for processing large-scale feature point sets. Among them, the steps for initial feature point matching based on the FLANN algorithm are:
[0090] (1) Extract feature points and their descriptors:
[0091] The SIFT algorithm is used to extract feature points from two pre-processed remote sensing images (including remote sensing image A and remote sensing image B) with overlapping areas to be stitched, and a 128-dimensional SIFT descriptor is calculated for each feature point.
[0092] (2) Build FLANN index:
[0093] All SIFT descriptors of one of the remote sensing images (for example, remote sensing image A) are constructed into a FLANN index. This step makes the search process more efficient by selecting appropriate indexing algorithms (such as KD trees, hash tables, etc.) and parameters.
[0094] (3) Nearest neighbor search:
[0095] For each SIFT descriptor of another remote sensing image (for example, remote sensing image B), a nearest neighbor search is performed in the FLANN index of remote sensing image A. FLANN will find the nearest neighbor matching point in image A for each descriptor. Usually using Euclidean distance as a metric, FLANN can find one or more feature points that are closest.
[0096] (4) Initial Matching:
[0097] The initial feature point pairs are obtained through nearest neighbor search, which are obtained based on the minimum distance matching between the feature point descriptors of the two images.
[0098] (4) Optional-Bidirectional Consistency Check:
[0099] In order to improve the accuracy of matching, bidirectional verification can be performed, that is, not only finding the nearest neighbor points of the feature points of remote sensing image B in remote sensing image A, but also reversely verifying whether the points in remote sensing image B are also the nearest neighbors of the points in remote sensing image A. If the nearest neighbor relationship of a pair of feature points in the two remote sensing images is consistent, the pair of points is considered to be a reliable matching point pair.
[0100] (5) Initial feature point pair:
[0101] The final matched feature point pairs are called initial feature point pairs. In remote sensing image stitching tasks, the FLANN algorithm can help quickly identify similar feature point pairs in two images, especially when the images contain a large number of feature points, the performance advantage of FLANN is more obvious. These initial feature point pairs provide a good starting point for image stitching and can be used for subsequent geometric correction, precise matching, and global optimization of image stitching.
[0102] It should be noted that although these feature point pairs have been optimized and accelerated by the FLANN algorithm, they may still contain some mismatched point pairs. Therefore, further precise matching and elimination of mismatched point pairs may be required. To this end, step S500 is used to eliminate mismatched point pairs.
[0103] S500: Perform secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs.
[0104] In the present invention, multiple initial feature point pairs are matched twice to obtain accurate feature point pairs. Specifically, multiple initial feature point pairs are matched twice based on the RANSAC algorithm to obtain accurate feature point pairs.
[0105] After obtaining multiple initial feature point pairs, there may be some mismatched points, which will affect the accuracy of subsequent image stitching. In order to remove these mismatched points and further improve the matching accuracy, the RANSAC (Random Sample Consensus) algorithm can be used to perform secondary matching on the initial feature point pairs to obtain accurate feature point pairs. Specifically, the steps of secondary matching based on the RANSAC algorithm include:
[0106] (1) Input the initial feature point pair:
[0107] The initial feature point pairs obtained from the FLANN algorithm are taken as input.
[0108] (2) Select a random sample:
[0109] The RANSAC algorithm first randomly selects a minimum sample set from the initial feature point pairs. For the two-dimensional image registration problem, four pairs of feature points are usually selected to estimate the transformation model between the images (such as affine transformation or homography matrix).
[0110] (3) Estimation transformation model:
[0111] Using the selected feature point pairs, calculate the transformation model between the images (such as an affine matrix or homography matrix). This model describes how to map points in one image to another image.
[0112] (4) Calculate the interior point set:
[0113] Apply this transformation model to all initial feature point pairs and calculate the residual of each point pair (i.e. the distance between the actual mapping position and the model predicted position). If the residual of a point pair is less than the set threshold, the point pair is considered an "inside point" (i.e. a correctly matched point pair); otherwise, it is considered an "outside point" (i.e. a mismatched point pair).
[0114] (5) Repeated iteration:
[0115] Repeat steps (2) to (4) for multiple iterations (usually a set number of iterations or until the number of inliers reaches a certain threshold), selecting a different set of random samples each time. In all iterations, the model with the largest number of inliers is selected as the optimal transformation model.
[0116] (6) Obtain accurate feature point pairs:
[0117] All the internal point sets of the optimal model are retained, and these point pairs are the precise feature point pairs screened by RANSAC.
[0118] (7) Apply the optimal model:
[0119] These precise feature point pairs are used to re-estimate the transformation model between images and apply it to image registration, stitching or other related operations.
[0120] S600: Based on the precise feature point pairs, two pre-processed remote sensing images to be stitched having overlapping areas are stitched together to output a stitched remote sensing image.
[0121] After image registration, the present invention selects the image transformation relationship, then optimizes the solution of the global transformation model, then optimizes the calculated image global transformation matrix, uses the optimized projection matrix for image stitching, and finally constrains the optimized global transformation matrix, thereby improving the algorithm's alignment ability in the image overlapping area and reducing the algorithm's distortion in the image non-overlapping area.
[0122] In the present invention, based on the precise feature point pairs, two pre-processed remote sensing images to be stitched with overlapping areas are stitched to output a stitched remote sensing image, which specifically includes the following steps:
[0123] S610: Using the pixel coordinates of the feature points of the precise feature point pairs, a global projection transformation matrix is obtained based on a DLT algorithm.
[0124] 1. Input precise feature point pairs:
[0125] The precise feature point pairs obtained from the RANSAC algorithm are input into this step, and these point pairs contain the pixel coordinates of the corresponding feature points in the two images.
[0126] 2. Construct equation:
[0127] For each pair of feature points, a linear system of equations is constructed using the DLT algorithm. DLT requires at least 4 pairs of non-collinear feature points to solve the homography matrix (also known as the global projective transformation matrix).
[0128] In the present invention, in image stitching, the feature matching point sets of the two images to be stitched are respectively recorded as X and X′. The relationship between the two is fitted by homography mapping transformation, which can be expressed as:
[0129]
[0130] Where (u, v, 1) T is the coordinate of the feature point in X′, (x, y, 1) T is the coordinate of the feature point in X, and H is the homography matrix between images, which represents the transformation relationship between images.
[0131] The derivation process of the DLT algorithm is as follows: Expand the above formula and divide the first two rows by the third row to get:
[0132] -h1x-h2y-h3+(h7x+h8y+h9)u=0
[0133] -h4x-h5y-h6+(h7x+h8y+h9)v=0;
[0134] Further, it can be organized into A i h=0; where Finally, the final global projection transformation matrix can be obtained by solving the singular value vector of the least squares solution of the equation.
[0135] In a preferred implementation, the accuracy of calculating the homography matrix is closely related to the accuracy of the coordinates of the corresponding points between the two images, so improving the accuracy of the pixel coordinates can effectively improve the accuracy of the homography matrix. Therefore, the present invention also optimizes the input coordinates of the DLT algorithm.
[0136] In order to improve the stability and accuracy of the solution, the present invention performs isotropic scaling and normalization processing on the pixel coordinates of the feature matching points of the input data. The normalized eight-point algorithm mentions that by adjusting the ratio of the pixel coordinates and translating the feature point pixels, the homogeneous coordinates of the feature point pixels are made equal to 1 on average and the center of mass of the feature point pixels are located at the origin of the coordinates, which will improve the accuracy of the algorithm calculation matrix. Assume that the image pixel coordinates (x coordinates) of a set of feature points are {501.5, 502.3, 498.7, ...}. If translated by 500, these numbers will become {1.5, 2.3, -1.3, ...}. Therefore, when the coordinate values are not converted, the effective values of the feature point coordinates are masked by the coordinate offset of 500.
[0137] The valid part of the coordinate value is only found in the third or fourth significant digit of the coordinate. This will have an adverse effect on the calculation of the homography matrix. The coordinates of the feature point pairs detected in each image to be stitched are translated (different translation for each image) so that the centroid of all feature points is moved to the origin. The pixel coordinates of the feature point pairs are then isotropically scaled and the average form of the pixels is x = (1, 1, 1) T , where the x and y coordinates of the pixel are scaled to the same ratio. The conversion process is as follows:
[0138] (1) Translate these points so that their center of mass is at the origin;
[0139] (2) Scale these points isotropically so that the average distance to the origin is equal to
[0140] Finally, such transformation is applied to the matching point pairs of the two images to be stitched, and the DLT algorithm is used to calculate the transformation matrix between the images after coordinate scaling. The transformation matrix is then denormalized to obtain the global transformation matrix of the original image.
[0141] S620: Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model.
[0142] In a preferred implementation, the overlapping areas of the image are aligned, the projection offset of the feature matching point pairs in the overlapping area is calculated, and the coordinates of the pixel projection points are corrected to obtain an optimized transformation matrix. Then, the local outliers of the image are removed. The Bayesian feature refinement probability model is used to remove local outliers based on the distribution probability of the image feature points. Finally, the image is adaptively smoothed and optimized. Since the global projection has the advantage of retaining the original structure of the image in the non-overlapping part, the optimized transformation model is constrained according to the pixel projection offset, thereby retaining the global similarity of the non-overlapping area on the basis of improving the algorithm's ability to align the overlapping areas of the image.
[0143] S630: Using a global projection transformation model, stitching two pre-processed remote sensing images to be stitched that have overlapping areas, and outputting a stitched remote sensing image.
[0144] In the specific implementation, the optimized global projection transformation model is used to perform geometric transformation on the two images to be stitched, and the two images are mapped into a unified coordinate system. The two transformed images are stitched according to their overlapping areas. A variety of stitching techniques can be used, such as linear blending, pyramid blending, etc., to ensure a smooth transition of the stitching area. Finally, the stitched remote sensing image is output. This image is the result of two pre-processed remote sensing images with overlapping areas being aligned and stitched through a global projection transformation model. There is no obvious seam in the stitching area, and the overall image quality is high.
[0145] In the preferred implementation, the deformation function of the overlapping area of the image is used to compensate the global transformation matrix of the image. However, if the transformation of the overlapping area is directly extrapolated to the non-overlapping area, overfitting will occur. In addition, the global projection has the advantage of retaining the original structure of the image in the non-overlapping part, so it is necessary to retain the result of the global projection in the non-overlapping area. Therefore, the deformation function g(x, y) is gradually reduced to 0 in the non-overlapping area, and the suppressed deformation function is g s (x, y) = η g (x, y); when the point (x, y) is far away from the overlapping area, the scale parameter η gradually changes from 1 to 0.
[0146] in, where [x l , x u ] and [y l ,y u ] are the boundaries of the overlapping areas in the x and y directions respectively.
[0147] Where ds is the smooth transition width in pixels, which is set proportional to the maximum deviation. s=K*max{|G(x, y)|} where K is the scale parameter. The suppressed deformation function g s (x, y) becomes 0 outside the overlapping area, so that the global transformation characteristics of the non-overlapping area are well preserved.
[0148] The present invention optimizes the DLT algorithm, introduces the technical combination of the normalized eight-point algorithm, and uses isotropically scaled image coordinates to reduce calculation errors. In the image global matrix optimization part: the TPS function and the Bayesian feature refinement probability model are used to calculate the pixel deviation function by minimizing energy, and then the feature point transformation result is compensated by the pixel deviation function, thereby improving the image alignment capability. In order to reduce the distortion of the non-overlapping areas of the image, the optimized transformation matrix is constrained to maintain the global transformation of the non-overlapping areas of the image. And the test results show that the surveying and mapping method of the present invention has better alignment capability for parallax image splicing.
[0149] The present invention also provides a marine remote sensing mapping system, which is used to implement and execute the marine remote sensing mapping method, including:
[0150] An acquisition module, which is used to acquire multiple remote sensing images, wherein the remote sensing images are obtained by photographing the target ocean area by a drone;
[0151] A preprocessing module, which is used to preprocess the remote sensing image to obtain a preprocessed remote sensing image;
[0152] An extraction module, which is used to extract multiple feature points from the pre-processed remote sensing image;
[0153] A first matching module is used to perform initial matching on feature points of two pre-processed remote sensing images to be spliced with overlapping areas to obtain a plurality of initial feature point pairs;
[0154] A second matching module is used to perform secondary matching on the multiple initial feature point pairs to obtain accurate feature point pairs;
[0155] The stitching module is used to stitch two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and output a stitched remote sensing image.
[0156] Preferably, the extraction module extracts a plurality of feature points from the preprocessed remote sensing image, specifically, extracts a plurality of feature points from the preprocessed remote sensing image based on a SIFT algorithm.
[0157] Preferably, the first matching module performs initial matching to obtain a plurality of initial feature point pairs, specifically, performs initial matching based on a plurality of feature points of a FLANN algorithm to obtain a plurality of initial feature point pairs.
[0158] Preferably, the second matching module performs secondary matching on the multiple initial feature point pairs to obtain accurate feature point pairs. Specifically, the second matching module performs secondary matching on the multiple initial feature point pairs based on the RANSAC algorithm to obtain accurate feature point pairs.
[0159] Preferably, the stitching module stitches two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and outputs a stitched remote sensing image, which specifically includes the following steps:
[0160] Using the pixel coordinates of the feature points of the precise feature point pairs, the global projection transformation matrix is solved based on the DLT algorithm;
[0161] Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model;
[0162] The global projection transformation model is used to stitch two pre-processed remote sensing images with overlapping areas to output a stitched remote sensing image.
[0163] For other structures of the offshore remote sensing surveying and mapping method and surveying and mapping system described in this embodiment, refer to the prior art.
[0164] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Therefore, any modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.
Claims
1. A method for remote sensing mapping at sea, characterized in that: The steps include: Acquire multiple remote sensing images, where the remote sensing images are obtained by photographing a target ocean area by a drone; Preprocessing the remote sensing image to obtain a preprocessed remote sensing image; Extracting a plurality of feature points from the preprocessed remote sensing image; Performing initial matching on the feature points of two pre-processed remote sensing images to be spliced with overlapping areas to obtain multiple initial feature point pairs; Perform secondary matching on multiple initial feature point pairs to obtain accurate feature point pairs; Based on the precise feature point pairs, two pre-processed remote sensing images to be stitched with overlapping areas are stitched to output a stitched remote sensing image.
2. A method for offshore remote sensing mapping according to claim 1, characterized in that: Extracting multiple feature points from the preprocessed remote sensing image, specifically, extracting multiple feature points from the preprocessed remote sensing image based on a SIFT algorithm.
3. A method for remote sensing mapping at sea according to claim 1, characterized in that: Performing initial matching to obtain multiple initial feature point pairs, specifically, performing initial matching on multiple feature points based on the FLANN algorithm to obtain multiple initial feature point pairs.
4. A method for remote sensing mapping at sea according to claim 1, characterized in that: A second matching is performed on the multiple initial feature point pairs to obtain accurate feature point pairs. Specifically, a second matching is performed on the multiple initial feature point pairs based on the RANSAC algorithm to obtain accurate feature point pairs.
5. The method for offshore remote sensing mapping according to claim 1, characterized in that: Based on the precise feature point pairs, two pre-processed remote sensing images to be stitched with overlapping areas are stitched together to output a stitched remote sensing image, which specifically includes the following steps: Using the pixel coordinates of the feature points of the precise feature point pairs, the global projection transformation matrix is solved based on the DLT algorithm; Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model; The global projection transformation model is used to stitch two pre-processed remote sensing images with overlapping areas to output a stitched remote sensing image.
6. A marine remote sensing mapping system, characterized in that: include: An acquisition module, which is used to acquire multiple remote sensing images, wherein the remote sensing images are obtained by photographing the target ocean area by a drone; A preprocessing module, which is used to preprocess the remote sensing image to obtain a preprocessed remote sensing image; An extraction module, which is used to extract multiple feature points from the pre-processed remote sensing image; A first matching module is used for performing initial matching on feature points of two pre-processed remote sensing images to be spliced having overlapping areas to obtain a plurality of initial feature point pairs; A second matching module is used to perform secondary matching on the multiple initial feature point pairs to obtain accurate feature point pairs; The stitching module is used to stitch two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and output a stitched remote sensing image.
7. A marine remote sensing mapping system according to claim 6, characterized in that: The extraction module extracts a plurality of feature points from the preprocessed remote sensing image, specifically, extracts a plurality of feature points from the preprocessed remote sensing image based on a SIFT algorithm.
8. The marine remote sensing mapping system according to claim 6, characterized in that: The first matching module performs initial matching to obtain a plurality of initial feature point pairs. Specifically, the first matching is performed on a plurality of feature points based on the FLANN algorithm to obtain a plurality of initial feature point pairs.
9. The marine remote sensing mapping system according to claim 6, characterized in that: The second matching module performs secondary matching on the multiple initial feature point pairs to obtain accurate feature point pairs. Specifically, the second matching module performs secondary matching on the multiple initial feature point pairs based on the RANSAC algorithm to obtain accurate feature point pairs.
10. The marine remote sensing mapping system according to claim 6, characterized in that: The stitching module stitches two pre-processed remote sensing images to be stitched with overlapping areas based on the precise feature point pairs, and outputs a stitched remote sensing image, which specifically includes the following steps: Using the pixel coordinates of the feature points of the precise feature point pairs, the global projection transformation matrix is solved based on the DLT algorithm; Perform compensation optimization on the global projection transformation matrix to obtain a global projection transformation model; The global projection transformation model is used to stitch two pre-processed remote sensing images with overlapping areas to output a stitched remote sensing image.
Citation Information
Cited By
Real-time remote sensing map generation method under denial condition
CN119785061A
A real-time remote sensing map generation method under denial condition
CN119785061B
