A method and system for stitching coastal drone orthophotos

Abstract

The present invention belongs to the technical field of marine surveying and mapping, and discloses a method and system for stitching orthophotos of coastal drones. The method first extracts, describes and matches feature points based on SIFT and RANSAC algorithms, calculates affine transformation matrices based on feature point pairs, and roughly determines image adjacency and overlapping area distribution based on drone tracks, yaw angles, etc.; secondly, generates an initial stitching network based on the Voronoi diagram, and extracts nodes and neighborhoods; finally, a seam line re-search model based on an improved A* algorithm is proposed, and the initial seam line is adjusted and optimized from coarse to fine to achieve seamless stitching of multiple images. The present invention can reduce the occurrence of erroneous feature point pairs and improve the efficiency of feature point extraction and matching through preprocessing of image connection graph construction and coarse determination of overlapping areas; the improved A* algorithm can improve the optimization efficiency while ensuring the optimization effect of the seam line, and provide effective technical support for coastal remote sensing analysis and engineering construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of ocean surveying and mapping, and in particular relates to a method and system for stitching orthophotos of unmanned aerial vehicles in a coastal zone. Background Art

[0002] With the development of marine resource surveys, marine environmental protection, marine engineering and other fields, coastal orthophotos are urgently needed to provide real texture and geographic information. UAV remote sensing technology has become the main means of obtaining coastal orthophotos due to its high efficiency and high flexibility. However, due to the small coverage of a single image obtained by drone aerial photography, it is impossible to perform subsequent operations such as remote sensing interpretation on a large coastal area. It is necessary to stitch a series of images with overlapping rates into a panoramic image. However, in the process of stitching coastal images, due to the special terrain and complex natural environment, it may cause problems such as sparse or uneven distribution of feature points, making it difficult to stitch multiple images into a panoramic orthophoto.

[0003] Among the existing methods, image registration is mainly based on the method of calculating the transformation matrix based on feature point matching. The SIFT algorithm constructs descriptors through gradient amplitude for feature point matching. It is the most widely used due to its extremely high stability, but its efficiency is relatively low. The improved feature extraction algorithm, the fast and robust feature extraction algorithm, is more efficient than the SIFT algorithm but has lower stability. The approximate nearest neighbor matching algorithm can perform nearest neighbor search in a large feature space and sacrifice accuracy in exchange for faster matching speed. The directional fast feature detection and rotational binary feature description algorithm and the fast adaptive edge feature description algorithm both methods quickly detect feature points by counting pixel values ​​within a certain range around the point and perform binary description of the feature points, sacrificing a certain degree of accuracy to improve processing efficiency.

[0004] For the problem of stitching multiple images, the simplest and most direct method is to convert the stitching problem of multiple images into the stitching problem between multiple pairs of images, and stitch the current image with the stitched images in turn. However, this stitching strategy is inefficient and there are a large number of intermediate results. In this regard, the stitching network method based on the Voronoi diagram effectively solves the problem of stitching multiple images. However, stitching based on the initial network may cause ghosting, dislocation and other problems. To address this problem, the seam line search algorithm can be used to make the transition line between images pass through the area with high similarity between images as much as possible, bypassing obvious objects. At present, many seam line search algorithms have been proposed. These methods can be roughly divided into two categories: pixel-based methods, such as the Snake model and the ant colony algorithm, can automatically construct seams through different energy functions such as color and texture similarity, but the result may be a local optimal solution rather than a global optimal solution. The seam lines obtained by the Dijkstra shortest path search algorithm are long and require a lot of calculations, so they are less efficient when processing high-resolution images. Methods based on auxiliary information or objects combine existing DSM data to search for seam lines under road and building constraints, and Pan J combines ground object recognition and classification to search for seam lines under image segmentation preprocessing. Methods based on auxiliary information and objects are more likely to bypass more obvious ground objects, but they require prior data or some preprocessing, and have certain limitations.

[0005] The research and application of coastal image stitching are currently focused on areas with obvious features, or with the help of target-assisted stitching. There are the following problems with stitching UAV images of the coast where there are no obvious features or where the features are extremely unevenly distributed: feature extraction and matching algorithms such as SIFT have problems such as uneven feature distribution and matching error points, which lead to stitching misalignment and ghosting problems; the stitching algorithm of commercial UAV image processing software will have water surface holes. In order to solve the problems of stitching misalignment and ghosting; the stitching algorithm of commercial UAV image processing software will have water surface holes, it is particularly important to explore a reasonable and effective method for stitching coastal UAV orthophotos. The present invention uses the airborne image data of the coast of Fenjiezhou Island in Hainan as the data basis to perform seamless stitching of orthophotos.

[0006] Through the above analysis, the problems and defects of the prior art are as follows:

[0007] (1) The existing SIFT algorithm is inefficient, and the improved feature extraction algorithm is less stable. The approximate nearest neighbor matching algorithm, directional fast feature detection, and rotational binary feature description algorithm are less accurate. There are also problems such as uneven feature distribution and wrong matching points, which lead to splicing misalignment and ghosting problems. The splicing algorithm of commercial drone image processing software will have water surface holes.

[0008] (2) The traditional method of converting the problem of stitching multiple images into the problem of stitching multiple pairs of images and stitching the current image with the stitched images in turn is inefficient and has a large number of intermediate results.

[0009] (3) Existing pixel-based methods such as the Snake algorithm may produce local optimal solutions rather than global optimal solutions. The stitching lines obtained by the Dijkstra shortest path search algorithm are relatively long and require a large amount of calculation, which makes them less efficient when processing high-resolution images. Methods based on auxiliary information and objects require prior data or preprocessing, which have limitations.

[0010] The defects solved by the present invention are:

[0011] (1) The present invention is based on the SIFT algorithm, utilizing its high accuracy and stability, and supplemented by track constraints, that is, determining images with overlapping areas and rough overlapping areas, thereby improving its efficiency while ensuring the accuracy of feature point extraction and matching.

[0012] (2) The present invention uses the Voronoi diagram to stitch multiple images, so that the images are evenly distributed, the effective stitching area of ​​the images is mostly located in the central area, and there is no intermediate result.

[0013] (3) The present invention utilizes the A* algorithm similar to the Dijkstra algorithm, simplifies its 8-direction search into a 5-direction search according to the initial suture line start-end direction, and constructs a coarse-to-fine search model, which greatly improves the search efficiency while ensuring the suture line effect and accuracy. Summary of the invention

[0014] In order to overcome the difficulty in stitching coastal airborne orthophotos in the related art, the disclosed embodiment of the present invention provides a method and system for stitching coastal drone orthophotos, and more particularly, relates to a method and system for stitching coastal drone orthophotos by combining track constraints with Voronoi diagram optimization. On the basis of extracting overlapping areas of images, an image connection diagram is constructed and image registration is performed based on it, and then the Voronoi diagram principle and the improved A* algorithm are combined to realize the construction and optimization of the stitching network. The technical solution is as follows:

[0015] The present invention is implemented in this way: a method for stitching orthophotos of a coastal zone unmanned aerial vehicle comprises:

[0016] S1, SIFT-RANSAC image registration under overlapping area constraints based on UAV images and track data;

[0017] S2, based on the center of the registered image, generates the initial splicing network according to the Voronoi diagram, and extracts nodes and neighborhoods;

[0018] S3, constructing a buffer zone, and optimizing the initial suture line from coarse to fine based on the improved A* algorithm.

[0019] In step S1, SIFT-RANSAC image registration under overlapping area constraints is performed based on the drone image and track data, including:

[0020] Draw an image connection diagram and roughly determine the overlapping area. Calculate the coordinates of the four corners of the image based on the image center coordinates, ground resolution, image size and heading angle; intersect the boundaries of adjacent images to obtain the coordinates of the image boundary intersection points INi (i = 0, 1, ..., n), where n is the number of boundary intersection points, and calculate the overlapping area S o , and finally calculate the overlap rate η;

[0021] Image registration, based on the image connection map and overlapping area information, uses the SIFT-RANSAC algorithm to achieve accurate matching of feature points and obtain the feature point pairs in image A and image B. A {p0,p1,p2...,p n-1} and Points B {q0,q1,q2...,q n-1}, Points A is the feature point set in image A, p 1…n-1 Points are the corresponding feature points in image A. B is the feature point set in image B, q 1…n-1 is the corresponding feature point in image B, according to Points A With Points B Calculate the transformation matrix; map each point (x, y) in the original image coordinate system to a point (X, Y) in the unified coordinate system through the affine transformation matrix and color interpolation.

[0022] Furthermore, the coordinates of the four corners of the image are calculated based on the image center coordinates, ground resolution, image size and heading angle. The expression is:

[0023]

[0024] Among them, (x, y) is the coordinate of the center point, (x1, y1) is the coordinate of the upper left point, (x2, y2) is the coordinate of the upper right point, (x3, y3) is the coordinate of the lower right point, (x4, y4) is the coordinate of the lower left point, L*W is the image size, φ is the heading angle, g is the ground resolution of the image, D is the actual distance on the ground corresponding to half the length of the image diagonal, θ is the angle between the image diagonal and the east-west direction, L is the horizontal resolution of the image, and W is the vertical resolution of the image.

[0025] In step S2, an initial splicing network is generated according to the Voronoi diagram, including:

[0026] The image is downsampled 10 times, and the geometric center of the registered image is used as the center point. For each pixel point in the stitching matrix, the Euclidean distance to each center point is calculated;

[0027] Mark the pixel as belonging to the image region where the nearest center point is located;

[0028] After each pixel is assigned a region, the resolution is restored through the nearest neighbor interpolation method to complete the initial splicing network construction.

[0029] Furthermore, the pixel points are marked as belonging to the image area where the nearest center point is located:

[0030] H(x j ,y j )=i,D ij =min

[0031] Where H represents the concatenation matrix, (x j ,y j ) is the pixel position, i is the Euclidean distance D ij The minimum principal point number.

[0032] In step S2, node and neighborhood extraction is performed, including:

[0033] For each node, in the initial network matrix, if the number of adjacent area labels is 3 and there is a background value, it is an external node; if the number of adjacent area labels is greater than or equal to 3 and there is no background value, it is an internal node; while extracting the node, the neighborhood description is performed to record the labels of the image areas that are commonly adjacent to the adjacent nodes.

[0034] In step S3, a buffer is constructed, including:

[0035] Based on the extracted nodes, a buffer is used to constrain the re-search area. The buffer is constructed with the line between the start point and the end point as the center. The buffer construction range is determined according to the following formula:

[0036]

[0037] Where d is the radius of the buffer constructed based on the initial stitching line, H is the vertical resolution of the original photo, W is the horizontal resolution of the original photo, and R h is the heading overlap ratio, R w is the lateral overlap ratio;

[0038] After all point pairs are processed, a buffer is obtained.

[0039] In step S3, the initial suture line is adjusted and optimized from coarse to fine based on the improved A* algorithm, including:

[0040] The cost function is constructed based on the color difference and gradient difference of the image; the cost function is downsampled, and a rough optimization is performed in the downsampled cost function buffer;

[0041] The search direction of the A* algorithm is optimized, and a rough search is performed using the improved A* algorithm based on the downsampled cost function; after restoring the resolution of the cost function, a fine search is performed using the improved A* algorithm.

[0042] Furthermore, the cost function consists of two parts: estimated cost and actual cost, expressed as:

[0043] F(n)=g(n)+h(n)

[0044] Among them, g(n) is the actual cost and h(n) is the estimated cost;

[0045] h(n) is defined as:

[0046]

[0047] Where C represents the point set consisting of the center lines of the rough suture line, c is a point in the point set C, and x n ,y n is the row and column position of the node to be visited, x c ,y c is the row and column position of point c;

[0048] The stitching network optimization is completed after a coarse-to-fine re-search of the initial seams between all node pairs.

[0049] Another object of the present invention is to provide a coastal zone UAV orthophoto stitching system, which is implemented by the coastal zone UAV orthophoto stitching method, and the system comprises:

[0050] The UAV image registration module is used to draw the image connection map and determine the overlapping area, and perform SIFT-RANSAC image registration under the constraints of the overlapping area based on the UAV image and track data;

[0051] The mosaic network construction module is used to generate the initial mosaic network according to the Voronoi diagram based on the center of the registered image;

[0052] The node and neighborhood extraction module is used to describe the neighborhood while extracting nodes, and record the labels of the image regions that are commonly adjacent to adjacent nodes;

[0053] Image stitching module, used to build buffers, and use the stitching optimization model based on the improved A* algorithm to adjust and optimize the initial stitching from coarse to fine.

[0054] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: In the coastal drone orthophoto stitching method of the present invention that combines track constraints and Voronoi diagram optimization, firstly, feature point extraction, description and matching are performed based on SIFT and RANSAC algorithms, and the affine transformation matrix is ​​calculated based on the feature point pairs, and the image adjacency relationship and overlapping area distribution are roughly determined based on the drone track, yaw angle, etc.; secondly, the initial stitching network is generated according to the Voronoi diagram, and nodes and neighborhoods are extracted; finally, a seam line re-search model based on the improved A* algorithm is proposed, and the initial seam line is adjusted and optimized from coarse to fine to achieve seamless stitching of multiple images. The present invention effectively solves the problem of difficulty in stitching coastal drone orthophotos, and provides effective technical support and solutions for further coastal remote sensing research and engineering construction.

[0055] The present invention can reduce the occurrence of erroneous feature point pairs and improve the efficiency of feature point extraction and matching by constructing an image connection graph and preprocessing the rough determination of overlapping areas. When processing images of coastal areas and other areas, direct splicing after alignment based on the existing SIFT-RANSAC method may cause misalignment. However, the present invention can effectively reduce the misalignment by adjusting and optimizing the initial suture line from coarse to fine based on the improved A* algorithm. The existing A* algorithm has a low efficiency in searching in eight directions based on the current pixel node when optimizing the suture line. The present invention reduces its nodes to 5 according to the starting point-end point direction of the suture line, and proposes a coarse-to-fine optimization method, which greatly improves the optimization efficiency while ensuring the optimization effect of the suture line. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings herein are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description, serve to explain the principles of the present disclosure;

[0057] Figure 1 is a flow chart of a method for stitching coastal drone orthophotos provided by an embodiment of the present invention;

[0058] Figure 2 is a schematic diagram of a method for stitching orthophotos of coastal drones provided in an embodiment of the present invention;

[0059] Figure 3 It is a schematic diagram of a connection graph construction and overlapping area determination method provided by an embodiment of the present invention;

[0060] Figure 4 is a schematic diagram of an initial network construction method provided by an embodiment of the present invention;

[0061] Figure 5 Schematic diagram of an improved method of the A* algorithm in suture optimization provided by an embodiment of the present invention;

[0062] Fig. 6A : is a schematic diagram comparing the SSIM index of each heading stitching line of the coastal zone drone orthophoto stitching method provided by the embodiment of the present invention and the method using only the SIFT-RANSAC algorithm;

[0063] Figure 6B It is a schematic diagram comparing the SSIM index of each lateral stitching line using the coastal zone drone orthophoto stitching method provided by an embodiment of the present invention and using only the SIFT-RANSAC algorithm. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0065] The existing A* algorithm searches in eight directions based on the current pixel node when optimizing the seam line, which is inefficient. The innovation of the present invention is to reduce its nodes to 5 based on the seam line start-end direction, and propose a coarse-to-fine optimization method: first, a rough seam line search under buffer constraints is performed in the actual cost function graph after downsampling; after restoring the cost function resolution, the cost is estimated based on the first rough seam line calculation and combined with the actual cost to perform a fine search to complete the seam line optimization. While ensuring the seam line optimization effect, the optimization efficiency is greatly improved.

[0066] Embodiment 1, as Figure 1 and Figure 2 As shown, the coastal zone drone orthophoto stitching method provided by the embodiment of the present invention includes the following steps:

[0067] Step 1: Perform SIFT-RANSAC image registration under overlapping area constraints based on UAV images and track data.

[0068] In order to convert the images into a unified coordinate system, the images need to be registered. At the same time, in order to improve the efficiency of image registration and prevent error accumulation, the overlapping relationship and overlapping area of ​​each image need to be extracted first.

[0069] The image connection diagram and overlapping area determination of the present invention are shown in Figure 3 .

[0070] Preferably, step 1 provided in the embodiment of the present invention specifically includes the following steps:

[0071] Step 1.1: Draw an image connection diagram and use it to roughly determine the overlapping area, that is, the feature point matching area. First, calculate the coordinates of the four corners of the image based on the image center coordinates, ground resolution, image size, and heading angle:

[0072]

[0073] Among them, (x, y) is the coordinate of the center point, (x1, y1) is the coordinate of the upper left point, (x2, y2) is the coordinate of the upper right point, (x3, y3) is the coordinate of the lower right point, (x4, y4) is the coordinate of the lower left point, L*W is the image size, φ is the heading angle, g is the image ground resolution, D is the actual distance on the ground corresponding to half the length of the image diagonal, θ is the angle between the image diagonal direction and the east-west direction, L is the horizontal resolution of the image, and W is the vertical resolution of the image. .

[0074] Secondly, the adjacent images are intersected by the boundary to obtain the image boundary intersection point IN i (i=0,1,…,n), where n is the number of boundary intersections, and the overlapping area S is calculated o :

[0075]

[0076] Among them, x k (k=1,2,…,m-1), x k+1 (k=1,2,…,m-1) represents the horizontal coordinate of the overlapping area boundary, y k (k=1,2,…,m-1),y k+1 (k=1,2,…,m-1) represents the vertical coordinate of the boundary of the overlapping area.

[0077] Finally, calculate the overlap rate η:

[0078]

[0079] Among them, S a is the initial image area. After the calculation is completed, the image connection diagram can be drawn and the location of the overlapping area can be recorded.

[0080] Step 1.2: Image registration. Based on the image connection graph and overlapping area information, the SIFT-RANSAC algorithm is used to achieve accurate matching of feature points and obtain the feature point pairs in image A and image B. A {p0,p1,p2...,p n-1} and Points B {q0,q1,q2...,q n-1}, where Points A is the feature point set in image A, p 1…n-1Points are the corresponding feature points in image A. B is the feature point set in image B, q 1…n-1 is the corresponding feature point in image B, according to Points A With Points B Calculate the transformation matrix. Commonly used transformation matrices are generally rigid transformation, affine transformation, and homography transformation. Affine transformation has 6 degrees of freedom. Between rigid transformation and homography transformation, it can suppress the perspective deformation caused by ground undulation to a certain extent, and will not cause excessive distortion of the image due to excessive degrees of freedom. It is an ideal matrix for aerial image space transformation. Its expression is:

[0081]

[0082] Among them, c and f are translation coefficients, a and e are scale coefficients, and b and d are rotation coefficients.

[0083] Map each point (x, y) in the original image coordinate system to a point (X, Y) in the unified coordinate system through affine transformation matrix and color interpolation:

[0084] X=f1(x,y),Y=f2(x,y) (5)

[0085] When faced with a large image data set, the existing technology needs to traverse the image to extract and match feature points. However, through the preprocessing of image connection graph construction and rough determination of overlapping areas in step 1 of the present invention, the occurrence of erroneous feature point pairs can be reduced and the efficiency of feature point extraction and matching can be improved.

[0086] Step 2: Generate the initial splicing network based on the Voronoi diagram based on the center of the registered image, and extract nodes and neighborhoods.

[0087] In order to avoid the appearance of intermediate results when stitching two images together and ensure that the registered images are evenly distributed when stitching, it is necessary to construct the initial stitching network based on the Voronoi diagram principle. At the same time, node extraction and neighborhood description are required for the subsequent optimization of the pairwise stitching lines.

[0088] The Voronoi diagram initial network construction method and results of the present invention are as follows Figure 4 shown.

[0089] Preferably, step 2 provided in the embodiment of the present invention specifically includes the following steps:

[0090] Step 2.1: Based on the center of the registered image, construct a uniformly distributed grid according to the Voronoi diagram principle. The basic principles are: there is only one discrete point in each polygon; the distance between the points in the polygon and the corresponding discrete points is the shortest; the distance between the points on the edge of the polygon and the discrete points on the other two sides is equal. The present invention uses a simpler and more intuitive way to construct the initial network: first, downsample the image by 10 times, take the geometric center of the registered image as the center point, and for each pixel point in the splicing matrix, calculate its Euclidean distance to each center point, that is:

[0091]

[0092] Among them, D ij is the Euclidean distance between each pixel point j and each center point i, x i With y i For each center point row and column position, x j With y j The row and column positions for each pixel.

[0093] Mark pixels as belonging to the image region whose center point is closest to them:

[0094] H(x j ,y j )=i,D ij =min (7)

[0095] Where H represents the concatenation matrix, (x j ,y j ) is the pixel position, i is the Euclidean distance D ij After allocating each pixel to a region, the nearest neighbor interpolation method is used to restore the resolution and complete the initial splicing network construction. The construction method is shown in Figure 4 .

[0096] Step 2.2: Node extraction and neighborhood description. For each node, in the initial network matrix, if the number of its adjacent region labels is 3 and there is a background value, it is an external node; if the number of its adjacent region labels is greater than or equal to 3 and there is no background value, it is an internal node. While extracting nodes, perform neighborhood description and record the labels of the image regions that are commonly adjacent to adjacent nodes.

[0097] Step 3: Based on the improved A* algorithm, the initial suture line is adjusted and optimized from coarse to fine.

[0098] After the registered images are directly spliced ​​based on the Voronoi diagram, ghosting and dislocation may occur. Therefore, the initial seam line is adjusted and optimized from coarse to fine based on the improved A* algorithm. First, a buffer is constructed to prevent the seam line from exceeding the overlapping area during the optimization process. At the same time, a seam line optimization model based on the improved A* algorithm is proposed to improve the optimization efficiency and effect.

[0099] Step 3.1: Buffer construction. Based on the extracted nodes, a buffer is used to constrain the re-search area. The buffer is constructed with the line between the start point and the end point as the center. The buffer construction range is determined according to formula (8):

[0100]

[0101] Where d is the radius of the buffer constructed based on the initial stitching line, H is the vertical resolution of the original photo, W is the horizontal resolution of the original photo, and R h is the heading overlap ratio, R w is the lateral overlap ratio, and the buffer zone can be obtained after processing all point pairs.

[0102] Step 3.2: Seam optimization based on the improved A* algorithm. The model construction steps are: constructing a cost function based on color and gradient differences; downsampling the cost function; using the improved A* algorithm to perform a rough search based on the downsampled cost function, and after restoring the resolution, using the improved A* algorithm to perform a fine search.

[0103] The cost function construction method based on image color difference and gradient difference is shown in formula (9):

[0104] EA=EC+EG (9)

[0105] Among them, EA is the total cost function, EC is the color difference of the overlapping part of the image, and EG is the gradient difference. EC and EG are defined as formula (10):

[0106]

[0107] Among them, C A and C B are the grayscale values ​​of image A and image B respectively. and is the horizontal gradient of images A and B, and is the vertical gradient of images A and B.

[0108] A rough optimization is performed in the cost function buffer after downsampling. The basic principle of the A* algorithm is: the comprehensive priority is calculated by the estimated cost and actual cost of the node to be visited. When selecting the next node to be traversed, the node with the highest comprehensive priority is selected to gradually find the optimal path. At this time, the A* algorithm only has the actual cost, as shown in formula (11):

[0109] F(n)=g(n) (11)

[0110] Among them, g(n) is the actual cost, that is, the cost function value EA. At the same time, the search direction of the A* algorithm is optimized, such as Figure 5 As shown in the figure, the approximate direction is determined based on the positions of the start node and the end node, reducing the original algorithm's 8 neighborhood nodes to be visited to 5, improving the search efficiency. The improved method is shown in Figure 5 .

[0111] After the rough optimization is completed, the cost function resolution is restored and the second A* optimization algorithm is executed. The purpose of this optimization is to refine the suture line. During the process, the A* algorithm is not restricted by the buffer zone, but the cost function consists of two parts: estimated cost and actual cost, as shown in formula (12):

[0112] F(n)=g(n)+h(n) (12)

[0113] Among them, g(n) is the actual cost, that is, the EA value, and h(n) is the estimated cost, which is defined as formula (13):

[0114]

[0115] Where C represents the point set consisting of the center lines of the rough suture line, c is a point in the point set C, and x n ,y n is the row and column position of the node to be visited, x c ,y c is the row and column position of point c, that is, the farther away from the center of the rough stitching line, the greater the estimated cost. The optimization of the splicing network can be completed by re-searching the initial stitching lines between all node pairs from coarse to fine.

[0116] When processing images of coastal areas and other regions, due to the problem of uneven feature distribution, direct stitching after alignment based on the existing SIFT-RANSAC method may cause misalignment. Therefore, the initial suture line is adjusted and optimized from coarse to fine based on the improved A* algorithm in step 3 of the present invention, which can effectively reduce the misalignment phenomenon. In step 3, the existing A* algorithm searches in eight directions based on the current pixel node when optimizing the suture line, which is inefficient. The present invention reduces its nodes to 5 according to the starting-end direction of the suture line, and proposes a coarse-to-fine optimization method: first, a rough search of the suture line under the buffer constraint is performed in the actual cost function graph after downsampling; after restoring the resolution of the cost function, the estimated cost is calculated based on the first rough suture line and combined with the actual cost to perform a fine search to complete the suture line optimization. Step 3 of the present invention greatly improves the optimization efficiency while ensuring the suture line optimization effect.

[0117] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] Since the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present invention, their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0119] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment.

[0120] According to an embodiment of the present application, the present invention also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0121] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0122] An embodiment of the present invention further provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, and switches.

[0123] An embodiment of the present invention further provides a server, which is used to provide a user input interface to implement the steps in the above method embodiments when executed on an electronic device.

[0124] An embodiment of the present invention further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0126] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] In order to verify the performance of the coastal zone drone orthophoto stitching method of the present invention, the experiment uses a DIJM300 RTK drone equipped with a DJI Zenmuse P1 camera for orthophoto acquisition. The orthophoto data used in the experiment was collected from the coastal area near Fenjiezhou Island, Hainan on September 28, 2023. When collecting data, the route parameters are: flight altitude 150m, heading overlap rate 80%, lateral overlap rate 70%, speed 10m / s; image data parameters: total number of images is 40, photo size is 3:2 (8192*5460), GSD is 1.88cm, focal length is 35mm, and the maximum regional terrain undulation does not exceed 15m. The experimental platform uses an AMDRyzen 5 4600Hwith Radeon Graphics 6-core processor, 32G RAM, Windows 10 64-bit operating system, programming software uses MATLAB 2022a, and programming is implemented in MATLAB language. The overall effect and some detail effects of the stitching method of the present invention are compared with those of stitching using only the SIFT-RANSAC algorithm. The present invention can effectively reduce the dislocation phenomenon, and the SSIN index (image structure similarity index) is used to quantitatively evaluate the stitching network.

[0128] Table 1 Average SSIM index of the paper algorithm and SIFT-RANSAC method

[0129]

[0130] The heading SSIM index of the SIFT-RANSAC algorithm is mostly in the interval [0.70, 0.95], and the heading SSIM index of the paper algorithm is mostly in the interval [0.75, 0.95]; the lateral SSIM index of the SIFT-RANSAC algorithm is mostly in the interval [0.55, 0.85], and the lateral SSIM index of the paper algorithm is mostly in the interval [0.75, 0.90]; and the SSIM index of a single seam line of the paper algorithm is greater than that of the SIFT-RANSAC algorithm. As shown in Table 2, the average heading SSIM of the SIFT-RANSAC algorithm is 0.856, and the average heading SSIM of the paper algorithm is 0.905. The SSIM index of the paper algorithm is 5.7% higher than that of the SIFT-RANSAC algorithm; the average lateral SSIM of the SIFT-RANSAC algorithm is 0.725, and the average lateral SSIM of the paper algorithm is 0.835. The SSIM index of the paper algorithm is 15.1% higher than that of the SIFT-RANSAC algorithm.

[0131] In summary, the present invention provides a method for stitching coastal drone orthophotos by combining track constraints with Voronoi diagram optimization. First, feature point extraction, description and matching are performed based on SIFT and RANSAC algorithms, and the affine transformation matrix is ​​calculated based on feature point pairs, and image adjacency and overlapping area distribution are roughly determined based on drone track, yaw angle, etc.; secondly, an initial stitching network is generated based on the Voronoi diagram to extract nodes and neighborhoods; finally, a seam line re-search model based on an improved A* algorithm is proposed to adjust and optimize the initial seam line from coarse to fine, so as to achieve seamless stitching of multiple images. The present invention solves the problem of difficulty in stitching coastal images, and provides effective technical support and solutions for further coastal remote sensing research and engineering construction.

[0132] 1. The present invention can be expanded from this experimental area to the collection of orthophotos of 18km of coastline across the country, providing orthophoto data with high precision, high density and unified coordinate system for water resource management and utilization. It provides important technical support for the urgent needs of ecology, surveying and mapping, water conservancy, transportation and other fields for high-precision geographic texture information in various complex environments, and generates significant economic benefits and commercial value.

[0133] 2. This invention fills the technical gap at home and abroad to a certain extent. In terms of image feature point extraction and matching, the image connection map is constructed based on the track and the overlapping area is roughly determined, which improves the efficiency; in terms of seam line search, without the need for image preprocessing and prior data, the seam line search effect and efficiency are better balanced.

[0134] 3. With the development of fields such as marine resource surveys, marine environmental protection, and marine engineering, there is an urgent need for coastal orthophotos to provide real texture and geographic information. UAV remote sensing technology has become the main means of acquiring coastal orthophotos due to its high efficiency and high flexibility. However, due to the small coverage of a single image acquired by drone aerial photography, it is impossible to perform subsequent operations such as remote sensing interpretation on a large coastal area. It is necessary to splice a series of images with overlapping rates into a panoramic image. However, in the process of stitching coastal images, due to the special terrain and complex natural environment, it may cause problems such as sparse or uneven distribution of feature points, making it difficult to stitch multiple images into a panoramic orthophoto. The reasonable and effective coastal drone orthophoto stitching method proposed in the present invention solves this problem to a certain extent.

[0135] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for stitching coastal drone orthophotos, characterized in that: The method includes: S1, SIFT-RANSAC image registration under overlapping area constraints based on UAV images and track data; S2, based on the center of the registered image, generates the initial splicing network according to the Voronoi diagram, and extracts nodes and neighborhoods; S3, constructing a buffer zone, and optimizing the initial suture line from coarse to fine adjustment based on the improved A* algorithm; SIFT-RANSAC image registration based on UAV images and track data under overlapping area constraints, including: Draw an image connection diagram and roughly determine the overlapping area. Calculate the coordinates of the four corners of the image based on the image center coordinates, ground resolution, image size and heading angle; intersect the boundaries of adjacent images to obtain the image boundary intersection points. The coordinates of n is the number of boundary intersections, and the overlapping area is calculated , and finally calculate the overlap rate ; Image registration, based on the image connection map and overlapping area information, uses the SIFT-RANSAC algorithm to achieve accurate matching of feature points and obtain images and images The feature point pairs in and , is the feature point set in image A, is the corresponding feature point in image A, is the feature point set in image B, is the corresponding feature point in image B, according to and Calculate the transformation matrix; transform each point in the original image coordinate system , mapped to points in a unified coordinate system through affine transformation matrix and color interpolation ; In step S2, an initial splicing network is generated according to the Voronoi diagram, including: The image is downsampled 10 times, and the geometric center of the registered image is used as the center point. For each pixel point in the stitching matrix, the Euclidean distance to each center point is calculated; Mark the pixel as belonging to the image region where the nearest center point is located; After assigning each pixel to a region, the resolution is restored through the nearest neighbor interpolation method to complete the initial splicing network construction; In step S3, a buffer is constructed, including: Based on the extracted nodes, a buffer is used to constrain the re-search area. The buffer is constructed with the line between the start point and the end point as the center. The buffer construction range is determined according to the following formula: ； in, Construct a radius for the buffer based on the initial seam line, is the vertical resolution of the original photo, is the horizontal resolution of the original photo, is the heading overlap ratio, is the lateral overlap ratio; After all point pairs are processed, a buffer is obtained.

2. The coastal zone drone orthophoto stitching method according to claim 1 is characterized in that: The coordinates of the four corners of the image are calculated based on the image center coordinates, ground resolution, image size and heading angle. The expression is: ； in, is the center point coordinate, is the coordinate of the upper left point, is the coordinate of the upper right point, is the coordinate of the lower right point, is the coordinate of the lower left point, is the image size, is the heading angle, is the image ground resolution, is the actual distance on the ground corresponding to half the length of the image diagonal. is the angle between the image diagonal and the east-west direction, is the horizontal resolution of the image, is the vertical resolution of the image.

3. The coastal zone drone orthophoto stitching method according to claim 1 is characterized in that: The image region where the pixel is marked as belonging to the nearest center point is: ； in, represents the concatenation matrix, is the pixel position, is the Euclidean distance The minimum principal point number.

4. The coastal zone drone orthophoto stitching method according to claim 1 is characterized in that: In step S2, node and neighborhood extraction is performed, including: For each node, in the initial network matrix, if the number of adjacent area labels is 3 and there is a background value, it is an external node; if the number of adjacent area labels is greater than or equal to 3 and there is no background value, it is an internal node; while extracting the node, the neighborhood description is performed to record the labels of the image areas that are commonly adjacent to the adjacent nodes.

5. The coastal zone drone orthophoto stitching method according to claim 1 is characterized in that: In step S3, the initial suture line is adjusted and optimized from coarse to fine based on the improved A* algorithm, including: The cost function is constructed based on the color difference and gradient difference of the image; the cost function is downsampled, and a rough optimization is performed in the downsampled cost function buffer; The search direction of the A* algorithm is optimized, and a rough search is performed using the improved A* algorithm based on the downsampled cost function; after restoring the resolution of the cost function, a fine search is performed using the improved A* algorithm.

6. The coastal zone drone orthophoto stitching method according to claim 5 is characterized in that: The cost function consists of two parts: estimated cost and actual cost, and the expression is: ； in, For the actual cost, To estimate the cost; Defined as: ； in, represents the set of points consisting of the center lines of the rough seam lines, Point Set One point in , is the row and column position of the node to be visited, , For point The row and column position of The stitching network optimization is completed after a coarse-to-fine re-search of the initial seams between all node pairs.

7. A coastal drone orthophoto stitching system, characterized in that: The system is implemented by the coastal zone drone orthophoto stitching method according to any one of claims 1 to 6, and the system comprises: The UAV image registration module is used to draw the image connection map and determine the overlapping area, and perform SIFT-RANSAC image registration under the constraints of the overlapping area based on the UAV image and track data; The mosaic network construction module is used to generate the initial mosaic network according to the Voronoi diagram based on the center of the registered image; The node and neighborhood extraction module is used to describe the neighborhood while extracting nodes, and record the labels of the image regions that are commonly adjacent to adjacent nodes; The image stitching module is used to construct a buffer zone and use the stitching optimization model based on the improved A* algorithm to adjust and optimize the initial stitching line from coarse to fine.

Images