An automatic labeling method for target GCPs

Abstract

The present application provides a method for automatically labeling a target GCP, comprising the following steps: establishing a coarse mesh model of a set of aerial images of a specified area; determining the labeling points of the target GCP on a reference image; projecting the labeling points onto the surface of the coarse mesh model and intercepting a first projection surface element; searching and determining the optimal position and optimal orientation of the first projection surface element based on the correlation of images within a reprojection window obtained by reprojecting the first projection surface element onto several evaluation images; projecting the image within a reference window on the reference image onto the first projection surface element with the optimal position and optimal orientation to obtain a second projection surface element, and searching and determining the corresponding imaging position of the target GCP on the image to be labeled based on the expected image obtained by reprojecting the second projection surface element onto the image to be labeled. The method provided by the present application effectively improves the credibility and accuracy of automatically labeling target GCPs on the image to be labeled.

Description

Technical Field

[0001] The present application belongs to the technical field of three-dimensional terrain reconstruction based on aerial photography measurement, and relates to GCP recognition technology in aerial image processing. Specifically, it provides an automatic marking method for target GCP. Background Art

[0002] Aerial photogrammetry is a remote sensing technology that uses aircraft (such as airplanes, helicopters, or drones) equipped with photographic equipment to capture designated ground areas and analyze and process these image data to obtain three-dimensional information and measurement data of terrain and land features. It combines the principles of aerial photography and surveying and can be widely used in land and resource exploration, surveying of major infrastructure projects, natural disaster monitoring and ecological and environmental protection, geographic information systems (GIS), environmental monitoring, three-dimensional terrain modeling, land and resource management, and other fields.

[0003] Ground Control Points (GCPs) play a vital role in three-dimensional terrain modeling based on aerial photography. As basic data, they are of key significance in improving product accuracy. Marking GCPs is an essential step after triangulation of aerial images. By marking GCPs on aerial pictures or images, the positioning accuracy of the data can be improved and the measurement results can be effectively integrated into the coordinate system.

[0004] The traditional GCP marking method is to manually mark GCPs on each image, which not only consumes a lot of time, but also significantly reduces the efficiency of product production; in order to improve the speed of GCP identification and marking, feature detection algorithms can be used to automatically extract GCPs in images; in recent years, with the continuous development of artificial intelligence and deep learning technology, methods for automatic GCP identification using various deep learning networks have emerged (Gong Xiaoqiang, Zou Jingui, Meng Liyuan. Automatic extraction of close-range photogrammetric control points based on target detection and corner detection [J]. Bulletin of Surveying and Mapping, 2020, (S1): 173-175+180; Ding Tao. Research on automatic layout and identification of image control points for UAV oblique photogrammetry [D]. Anhui University of Science and Technology, 2021).

[0005] However, the above-mentioned existing technologies still have problems when facing small fields of view, large deformation, large inclination angles and large amounts of aerial image data. The reasons are: First, in images with different perspectives, due to perspective deformation, the same GCP may show obvious deformation and image distortion in different pictures. When the degree of deformation exceeds the training range of the deep learning network used for recognition, accurate recognition will not be possible; in addition, the images of GCPs located at the edge of some pictures cannot be fully displayed, and there is a lack of complete information for feature detection, which makes it impossible for existing algorithms to use complete GCP image features to identify them; more importantly, various existing automatic recognition algorithms simply compare and match parts with similar texture features on pictures with different perspectives, without considering the relationship between the positions of the same GCP in pictures with different perspectives. This will cause a GCP to be incorrectly marked at the location of other GCPs during the automatic recognition process. Summary of the Invention

[0006] To solve the problems existing in the above-mentioned prior art, the present application provides a method for automatically marking target GCPs, which includes the following operations:

[0007] a. Build a coarse grid model based on a collection of aerial images of a specified area, where the target GCPs are located within the specified area;

[0008] b. determining a marking point of a target GCP on a reference image, where the marking point is located in an image of the target GCP on the reference image;

[0009] c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element;

[0010] d. searching and determining an optimal position and an optimal orientation of the first projection plane based on correlation of images within a reprojection window obtained by reprojecting the first projection plane onto a plurality of evaluation images;

[0011] e. Projecting an image within a reference window on the reference image onto a first projection plane having an optimal position and optimal orientation to obtain a second projection plane, wherein the image of the target GCP on the reference image is located within the reference window;

[0012] f. Based on the desired image obtained by reprojecting the second projection bin onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled.

[0013] Furthermore, the coarse grid model is generated by performing an unconstrained bundle adjustment operation on a set of aerial pictures of a designated area.

[0014] Furthermore, the coarse grid model uses as grid points points in a sparse point cloud obtained by performing an unconstrained bundle adjustment on a set of aerial pictures of a designated area.

[0015] Preferably, the similarity between the image of the target GCP on the reference image and its orthographic image is greater than a preset first similarity threshold, or the deviation is less than a preset first deviation threshold.

[0016] Preferably, the similarity between the image of the target GCP on the evaluation picture and its orthographic image or its image on the reference picture is greater than a preset second similarity threshold, or the deviation is less than a preset second deviation threshold, wherein the first similarity threshold is greater than or equal to the second similarity threshold, and the first deviation threshold is less than or equal to the second deviation threshold.

[0017] Furthermore, the operation c described in projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element includes: projecting the marked point along its reference projection line corresponding to the reference image onto the surface of the coarse mesh model and obtaining its projection point on the surface of the coarse mesh, wherein the reference projection line of the marked point corresponding to the reference image is determined based on the camera orientation element of the reference image.

[0018] Furthermore, the intercepted first projection surface element in operation c is determined by intercepting a region surrounding the projection point on the surface of the coarse mesh model.

[0019] Furthermore, in operation d, reprojecting the first projection bin to a plurality of evaluation images includes:

[0020] Each point on the first projection surface element is reprojected onto each evaluation picture along its projection line corresponding to each evaluation picture, and the edge of each point reprojected onto each evaluation picture is used as its reprojection window on each evaluation picture, wherein the projection line of each point on the first projection surface element corresponding to each evaluation picture is determined based on the camera orientation element of each evaluation picture.

[0021] Furthermore, the searching and determining of the optimal position and optimal orientation of the first projection plane element in operation d includes:

[0022] Taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to perform a translation operation or a rotation operation along the reference projection line;

[0023] Reproject the result of each translation or rotation operation onto each evaluation image to obtain a reprojection window, and calculate the correlation of the image of each evaluation image in its reprojection window;

[0024] When the calculation result meets the correlation criterion, the position and orientation of the first projection plane element at this time are taken as the optimal position and optimal orientation of the first projection plane element, otherwise the translation operation or rotation operation of the first projection plane element is performed again.

[0025] Preferably, the correlation criterion is: the correlation statistics of the images of each evaluation picture within its reprojection window meet a preset correlation threshold.

[0026] Preferably, the lower limit of the area of the first projection bin is 3×3 pixels, and the upper limit of the area is determined based on the correlation statistics results of the images in the reprojection window of each evaluation picture.

[0027] Furthermore, the expected image is an image formed by reprojecting each point on the second projection plane onto each picture to be labeled along its projection line corresponding to each picture to be labeled.

[0028] Furthermore, the search and determination of the imaging position corresponding to the target GCP on the image to be marked in operation f includes:

[0029] Extracting edges of the desired image to construct a matching window, wherein the matching window is constructed only by the edges of the desired image, or jointly by the edges of the desired image and the edges of the picture to be marked where the desired image is located;

[0030] The best position of the matching window is searched on the image to be marked, so as to mark the imaging position corresponding to the target GCP on the image to be marked.

[0031] Preferably, the best position of the matching window is searched on the image to be marked by using a least squares image matching algorithm.

[0032] Preferably, in the process of searching for the best position of the matching window on the image to be marked, the matching window is only translated on the image to be marked.

[0033] An embodiment of the present application provides a method for automatic labeling of a target GCP, which provides an initial "index" for the step of searching for the real spatial position and posture of the target GCP by generating a coarse grid model of a specified area; after projecting the target GCP accurately marked on a reference image onto the coarse grid, under the constraints of the coarse grid model, the most likely position and orientation of the target GCP in the actual physical space is searched based on the result of reprojection to the evaluation image; further, based on this most likely position and orientation, the expected position, shape and texture of the target GCP on the image to be evaluated are obtained, and this is used as the basis for searching the position of the target GCP on the image to be evaluated, thereby effectively avoiding the problems of existing solutions for automatic GCP labeling that rely solely on texture features, such as poor recognition effect when the distortion is large or part of the texture is missing, and incorrect association of different GCPs with similar texture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of an existing three-dimensional terrain modeling method based on aerial photogrammetry;

[0035] Figure 2 is a schematic diagram of a pattern of ground control points;

[0036] Figure 3 This is a schematic diagram of the principle of aerial triangulation based on aerial images;

[0037] Figure 4 This is a specific flow chart of an existing aerial triangulation using multiple aerial images and GCP information;

[0038] Figure 5 This is a schematic diagram of the image of the same GCP on several aerial photos;

[0039] Figure 6 Flowchart of the automatic labeling method for target GCPs provided in this application;

[0040] Figure 7 A schematic diagram of a specific implementation process of the automatic labeling method for target GCPs provided according to some embodiments of the present application;

[0041] Figure 8 A schematic diagram of a specific implementation process for establishing a coarse grid model of a specified area in some embodiments;

[0042] Figure 9 is a schematic diagram of a reference image and several evaluation images on both sides thereof in a specific embodiment;

[0043] Figure 10 A schematic diagram illustrating the operation principle of a first projection plane element in some specific embodiments;

[0044] Figure 11 A schematic diagram of the principle of searching and determining the optimal position and optimal orientation of the first projection plane element in some specific embodiments;

[0045] Figure 12 A flowchart for searching and determining the optimal position and optimal orientation of the first projection plane in some specific embodiments;

[0046] Figure 13 is a schematic diagram showing how the mean NCC value of the image in each reprojection window changes with the size of the first projection bin in a specific embodiment;

[0047] Figure 14 Schematic diagram of searching for an imaging position of a target GCP on a to-be-labeled image in some embodiments;

[0048] Figure 15 Schematic diagram of searching for an imaging position of a target GCP on a to-be-labeled image in some embodiments;

[0049] Figure 16 A schematic diagram of an aerial image of a designated area provided according to an embodiment of the present application;

[0050] Figure 17 A schematic diagram of a three-dimensional sparse point cloud of a specified area provided according to an embodiment of the present application;

[0051] Figure 18 This is a schematic diagram of a coarse grid model of a specified area provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Hereinafter, the present application will be further described based on preferred embodiments with reference to the accompanying drawings.

[0053] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the products of the embodiments of the present application are usually placed when in use. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0054] The vocabulary in this specification is used to illustrate the embodiments of the present application, but is not intended to limit the present application. It should also be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, an indirect connection through an intermediate medium, or a communication between the two components. For those skilled in the art, the specific meanings of the above terms in this application can be specifically understood.

[0055] In order to clearly illustrate the automatic marking method of target feature points provided in this application, the process of the existing three-dimensional terrain modeling method based on aerial photography measurement and its existing problems are first introduced.

[0056] Figure 1 A flowchart of an existing three-dimensional terrain modeling method based on aerial photography is shown. Figure 1 As shown, in order to generate a three-dimensional terrain model of a specified area, first, in step 100, several GCPs (Ground Control Points) are laid out in the specified area, and the spatial position of each GCP is measured with high precision using a total station, GPS, etc.

[0057] Generally, the shape of the ground control point is a square, and its side length can be determined according to the aerial photography height, shooting resolution, etc. For example, it can be made of 50cm×50cm or 1m×1m. It can be made of various boards or soft fabrics so that it can be placed on the ground, buildings or natural objects in the designated area. In order to facilitate identification from aerial images, its surface can be printed or painted. Figure 2 The pattern shown has more distinct features and higher contrast.

[0058] Step 200 involves developing an aerial photography plan. This involves selecting an appropriate aerial photography platform (aircraft, helicopter, drone, etc.) and camera based on the terrain of the designated area, and planning the flight altitude, flight path, and photography parameters (such as shooting angle and overlap). Generally, the overlap between consecutive aerial images is set between 60% and 80% to ensure the reliability of subsequent feature matching.

[0059] In step 300, aerial photography is performed on a designated area according to an aerial photography plan to obtain an aerial image of the area.

[0060] Generally, aerial images are multiple aerial pictures or continuous aerial videos taken according to an aerial photography plan. The original aerial images generally need to be converted according to the requirements of subsequent data processing, such as unifying the image size or converting continuous videos into multiple aerial pictures to meet the needs of subsequent data processing.

[0061] Next, in step 400 , aerial triangulation is performed using multiple aerial images and GCP information of a designated area.

[0062] Aerial triangulation is abbreviated as aerial triangulation. It is one of the key steps in the three-dimensional terrain modeling method based on aerial photography. Figure 3 As shown, in this step, the camera's orientation elements are estimated and the positions of the three-dimensional points in the scene are reconstructed through geometric analysis of multiple overlapping aerial images, thereby obtaining a three-dimensional sparse point cloud of the specified area.

[0063] The number of aerial images required for aerial triangulation depends on many factors, including the size, complexity, required resolution, and overlap of the specified area. For example, for small to medium-sized areas (such as small towns and markets), 20 to 50 aerial images are usually required. For high-precision measurement and analysis of large areas (such as mines, forests, cities, etc.), more than 50 to hundreds of high-resolution aerial images may be required to ensure adequate coverage and details.

[0064] Figure 4 The specific process of aerial triangulation (step 400 ) performed using multiple aerial images and GCP information in an existing embodiment is shown.

[0065] like Figure 4 As shown, first, in step 410, the GCPs in each aerial image are marked. When the aerial triangulation algorithm adopts the conventional constrained bundle adjustment method to determine the camera parameters and the three-dimensional position of each feature point, marking the GCPs is a necessary prerequisite. This is because the GCPs have accurate spatial position information. Therefore, they can be used as constraints in the subsequent constrained bundle adjustment to ensure that the output results are consistent with the known geographic reference.

[0066] The means of marking GCPs can be performed as described in the background art, by manually calibrating GCPs on each aerial image, or by automatically extracting GCPs from each aerial image using a trained deep learning network.

[0067] Generally, Figure 2 After the GCP shown is arranged in the specified area, the image it forms on the aerial image is a collection of pixels with a certain area. Furthermore, marking the GCP on the aerial image means specifying at least one pixel for the GCP to represent the position of the GCP on the aerial image. The pixel is also called the annotation point (or marking point) of the GCP on the aerial image.

[0068] Figure 5The figure shows the images of the same GCP on several aerial images, as well as the results of their labeling. Ideally, the pixel corresponding to the center of the GCP on the aerial image can be used as the label point; in the actual labeling process, the pixel position identified as the label point may deviate by several pixels from the actual pixel position corresponding to the center of the GCP on the aerial image. Therefore, when the deviation between the GCP label point and its actual position on a particular aerial image does not exceed a preset threshold, the GCP can be considered to have been accurately labeled on the aerial image.

[0069] Next, in step 420, feature points are extracted from each aerial image (algorithms such as SIFT and SURF can be used). Generally, these feature points are points with high contrast, edges, or corners that are automatically extracted from the aerial image by the feature extraction algorithm. After the feature points are extracted, in step 430, the different aerial images are registered. Specifically, the same feature points can be matched between different aerial images through a feature matching algorithm. This step can establish a correspondence between the various aerial images and identify the corresponding feature points at different shooting angles.

[0070] It should be noted that the pixel points of the image formed by GCP in the aerial image can also be regarded as a kind of feature points, because they also contain specific texture information in the image, and therefore can also participate in the feature point matching process.

[0071] After completing steps 420 and 430, in step 440, under the constraints of the GCP information, the camera's internal orientation elements (including the camera's focal length, principal point, distortion coefficient, etc.) and external orientation elements (including the camera's position and orientation) can be iteratively estimated through bundle adjustment. Based on the matching relationship between the camera's orientation elements (internal / external orientation elements) and the feature points, the three-dimensional coordinates of each feature point are iteratively estimated, thereby establishing a three-dimensional sparse point cloud of the specified area.

[0072] Bundle adjustment is a mathematical method for optimizing camera parameters and 3D point positions. After determining an initial estimate of the camera's orientation elements through simple geometric relationships (such as triangulation), it uses an iterative optimization method (such as the Gauss-Newton method) to repeatedly adjust the camera's orientation elements and 3D point coordinates, aiming to minimize the sum of the squares of all reprojection errors, until convergence is achieved. Obviously, when the coordinates of known GCPs are introduced as constraints during the iterative optimization process, their precise known positions ensure that the camera parameters and 3D point estimates conform to the known GCP coordinates, thereby improving the accuracy and precision of the 3D point estimates.

[0073] The 3D sparse point cloud generated in step 400 has too low a density to be directly used to build a 3D terrain model. Therefore, in step 500, the 3D sparse point cloud is further densified using image matching and reconstruction algorithms such as PatchMatch and SGM (Semi-Global Matching), thereby obtaining a 3D dense point cloud for the specified area. Then, in step 600, mesh reconstruction algorithms such as Delaunay triangulation and Poisson reconstruction are used to convert the 3D dense point cloud into a 3D fine mesh model, thus completing the construction of the 3D terrain model for the specified area.

[0074] Preferably, in step 700 , the three-dimensional fine grid model of the designated area may be colored, rendered, or visualized, or terrain features such as elevation and slope may be extracted, or further processed such as measurement and analysis may be performed.

[0075] Obviously, in the above process of three-dimensional terrain modeling of the specified area, whether the GCPs in each aerial image can be accurately marked through step 410 determines whether the correspondence between the imaging position of the GCP in the aerial image and its actual position coordinates is correct (that is, whether the bundle adjustment can be correctly constrained), which in turn has a key impact on the estimation results of the camera parameters and the three-dimensional coordinates of the feature points.

[0076] Manual processing can accurately identify and mark each GCP in each aerial image and accurately associate the same GCP in different aerial images. However, when the number of aerial images is large, the processing speed is obviously unable to meet the processing speed requirements.

[0077] In order to solve the problem of low efficiency of manual GCP marking mentioned above, several methods have emerged to automatically identify GCP images from various aerial pictures. For example, Gong Xiaoqiang et al. used the Faster R-CNN network combined with corner detection to identify GCP images in pictures (Gong Xiaoqiang, Zou Jingui, Meng Liyuan. Automatic extraction of close-range photogrammetry control points based on target detection and corner detection [J]. Bulletin of Surveying and Mapping, 2020, (S1): 173-175+180); for example, Ding Tao used the deep learning SSD enhancement algorithm to train the model, combined with the LSD line segment detection algorithm to extract the image image angle threshold of the control point, so as to identify the geometric center mark of the control point in the screened image (Ding Tao. Research on automatic layout and identification of control points in UAV oblique photogrammetry [D]. Anhui University of Science and Technology, 2021).

[0078] Although the above solution can automatically recognize the image of the landmark, the following problems still exist in its application:

[0079] 1) The same GCP may have significant shape deformation and imaging area differences in aerial images taken from different perspectives. For example, a GCP image that appears square in one aerial image (e.g., when the camera is shooting directly at the GCP on the ground) may be compressed into a very thin rectangle in another aerial image (e.g., when the camera is shooting at an angle that is almost perpendicular to the normal of the GCP on the ground). Due to the large difference in texture information between the two, neither correlation nor deep networks can resolve this large discrepancy.

[0080] 2) When a GCP is imaged at the edge of an aerial image, the image may only contain a partial image of the GCP. In this case, conventional image recognition methods may fail to correctly identify the GCP due to the lack of complete image information.

[0081] 3) The image of a GCP in a particular aerial image may be mistakenly associated with the image of another GCP in another aerial image for some reason. This problem may be caused by image distortion caused by perspective distortion or by the terrain structure of the specified area. For example, there are some relatively similar buildings with periodic arrangement, which causes the recognition algorithm to identify two different GCPs with different spatial locations but similar texture features in aerial images taken from two different locations as the same.

[0082] The applicant analyzed the causes of the above-mentioned problems and found that when using feature extraction or deep learning networks for automatic GCP identification, since the camera posture and shooting angle information have not yet been obtained through bundle adjustment, the identification basis is only the texture features of the image formed by the GCP in each aerial picture, and does not consider the position and shape of the GCP in the aerial picture with a specific shooting angle. That is, even if the above-mentioned recognition algorithm can extract the image of each ground feature point from each aerial picture through texture features, it cannot associate the same ground feature points on different aerial pictures. Therefore, no matter what kind of optimization is performed on the above-mentioned feature extraction algorithm or deep learning network structure, or how the training level of the deep learning network is increased, the root cause of the above-mentioned problem cannot be eliminated.

[0083] In order to solve the above problems, an embodiment of the present application provides a new method for automatic marking of target GCPs, which can replace the current method of manually marking aerial images one by one. It can automatically mark any GCP specified by a human (in the embodiment of the present application, the human-specified GCP is called the target GCP) in each aerial image that captures the GCP, and at the same time can significantly improve the shortcomings of the current GCP automatic marking method in the implementation process.

[0084] Figure 6A flow chart of an automatic labeling method for target GCPs provided by the present application is shown, Figure 6 As shown, the method includes the following operations:

[0085] a. Build a coarse grid model based on a collection of aerial images of a specified area, where the target GCPs are located within the specified area;

[0086] b. determining a marking point of a target GCP on a reference image, where the marking point is located in an image of the target GCP on the reference image;

[0087] c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element;

[0088] d. searching and determining an optimal position and an optimal orientation of the first projection plane based on correlation of images within a reprojection window obtained by reprojecting the first projection plane onto a plurality of evaluation images;

[0089] e. Projecting an image within a reference window on the reference image onto a first projection plane having an optimal position and optimal orientation to obtain a second projection plane, wherein the image of the target GCP on the reference image is located within the reference window;

[0090] f. Based on the desired image obtained by reprojecting the second projection bin onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled.

[0091] Figure 7 In some embodiments, the specific implementation process of the method is shown. Figure 6 、 Figure 7 As shown in FIG, the method mainly includes three sub-processes, wherein sub-process I (corresponding to operation a) is used to generate a coarse grid model of the specified area, which can provide an initial "index" for the subsequent step of searching for the real space position and posture of the GCP; in sub-process II (corresponding to operations b to d), after the target GCP accurately marked on the reference image is projected onto the coarse grid, under the constraint of the coarse grid model, the most likely position and orientation of the target GCP in the actual physical space is searched based on the result of reprojection to the evaluation image; in sub-process III (corresponding to operations e and f), the expected position, shape and texture of the target GCP on the image to be evaluated are obtained based on this most likely position and orientation, and this is used as the basis for searching the position of the target GCP on the image to be evaluated, thereby effectively avoiding the problems of existing schemes that rely solely on texture features for automatic GCP labeling, such as poor recognition effect when the distortion is large or part of the texture is missing, and incorrect association of different GCPs with similar texture features.

[0092] The specific implementation of this method is described in detail below.

[0093] Sub-process I (establishing a coarse grid model of a specified area):

[0094] like Figure 7 As shown, in sub-process I, first, a set of aerial pictures of the specified area is read through step 810. Specifically, as described above for the three-dimensional terrain modeling technology based on aerial photography, a number of GCPs can be pre-deployed in the specified area, and then the specified area is aerially photographed according to the aerial photography plan, and the acquired pictures, videos, etc. are processed to generate multiple aerial pictures that meet the overlap requirements (obviously, any GCP arranged in the specified area will be imaged in at least one aerial picture). The aerial picture set composed of all aerial pictures of the above-mentioned specified area can be stored in a database and read and processed in the process of modeling the specified area.

[0095] After the aerial image set of the designated area is read, a coarse grid model of the designated area can be established in step 820. The coarse grid model can be constructed by performing a bundle adjustment operation on each aerial image in the aerial image set.

[0096] It should be emphasized that in the method provided in the present application, since GCPs are not marked on the aerial images before executing step 820, this step adopts the bundle adjustment method that does not use GCP position coordinates for constraints, that is, in the process of establishing a coarse grid model through a set of aerial images, no external known ground control points are relied upon for constraints.

[0097] Figure 8 The specific implementation process of step 820 in some preferred embodiments is shown, wherein the estimation of the camera orientation elements and the three-dimensional coordinates of the feature points adopts the unconstrained bundle method free network adjustment. Specifically, first in step 821, feature points are extracted from each aerial image by a feature detection algorithm (such as SIFT, SURF or ORB, etc.); then in step 822, feature points of different aerial images are matched. This step can use methods such as nearest neighbor search to match the same feature points in aerial images of different perspectives; then, the unconstrained bundle method free network adjustment operation described in steps 823 to 827 is used to iteratively optimize the estimated values of the camera orientation elements and the three-dimensional coordinates of each feature point until the objective function converges or the number of iterations reaches an upper limit.

[0098] The estimation of camera orientation elements and three-dimensional coordinates of feature points involved in the above-mentioned bundle free network adjustment operation, as well as the form of the objective function, are all known to those skilled in the art. For example, initial estimates of the camera's internal and external orientation elements can be made using documentation of camera parameter specifications and the flight position and attitude records of the aircraft during aerial photography. Preliminary estimates of the three-dimensional coordinates of the feature points can also be made using triangulation methods. These estimates are then updated using nonlinear least squares methods such as Levenberg-Marquardt. A common form of the objective function is generally to minimize the reprojection error. Without departing from the inventive concept of this application, a specific algorithm can be reasonably selected to implement the above-mentioned unconstrained bundle adjustment operation.

[0099] After completing the above operations, the final estimated values of the camera orientation elements and the three-dimensional coordinates of the feature points can be determined (step 827). The multiple feature points with three-dimensional coordinate values mentioned above are used as point cloud points to form a sparse point cloud of the specified area; each point cloud point with a determined three-dimensional coordinate value is used as a grid point, and algorithms such as Delaunay triangulation are used to obtain a coarse grid model of the specified area (step 828).

[0100] like Figure 8 As shown, the camera orientation elements obtained in step 820 and the sparse point cloud and coarse mesh model of the specified area are stored in the database for subsequent use in the implementation of sub-process II and sub-process III.

[0101] It should be noted that in existing 3D terrain modeling methods based on aerial photography, after obtaining a sparse point cloud, a coarse mesh model is not generated based on it. Instead, it is directly densified to obtain a fine mesh model of the specified area and perform operations such as texture mapping. In the embodiments of the present application, the purpose of establishing a coarse mesh model is to provide a basis for estimating the actual pose of the target GCP in sub-process II.

[0102] Sub-process II (estimating the actual pose of the target GCP in the specified area):

[0103] The surface of the coarse mesh model reflects the general shape of the three-dimensional terrain of the specified area, that is, the normal direction and coordinates of each plane on the surface of the coarse mesh model can roughly characterize the orientation and spatial position of the specified area at that location. Therefore, a landmark point of a target GCP accurately calibrated on a reference image can be projected onto the coarse mesh model to obtain an initial facet as the search starting point. The orientation and projection distance of the facet are continuously changed within a small constrained range and reprojected to several evaluated images. The correlation of the images in the reprojected area on each evaluated image is used to determine the optimal orientation and projection distance of the facet. In this way, the most likely orientation and position of the target GCP when it is deployed in the specified area can be obtained.

[0104] Back to Figure 7 In some specific embodiments, sub-process II includes steps 830 to 860.

[0105] In step 830 , a reference picture may be selected and several evaluation pictures may be selected around the reference picture based on the imaging conditions of the target GCP to be automatically marked on each aerial picture.

[0106] In order to ensure the accuracy of subsequent search results, the reference image should be selected so that the target GCP is photographed at an orthographic angle as much as possible. That is, the aerial image whose image corresponding to the target GCP is close to its orthographic shape should be selected as the reference image. For example, when the actual shape of the target GCP is Figure 2 When the target GCP is a square, aerial images that capture the target GCP and whose image shape is also substantially square should be selected from the aerial image collection. In some preferred embodiments, deviation or similarity can be used to evaluate the deviation or similarity between the shape of the image formed by a GCP in an aerial image and the shape it should have when photographed from an orthographic perspective. For example, the Hausdorff distance, average distance, or similarity ratio can be used to measure the geometric difference between the image and the ideal square. For another example, the difference between the lengths of the sides (or perimeter) of the image formed by the GCP in the aerial image and the lengths of the sides (or perimeter) of a standard square of the same area can be used as a deviation indicator:

[0107]

[0108] Among them, D is the deviation index, l i is the length of each side of the image formed by a GCP on the aerial picture, l square is the side length of a square of equal area to the image.

[0109] Obviously, a preset similarity threshold (first similarity threshold) or deviation threshold (first deviation threshold) can be set. When the similarity between the shape of the image formed by a GCP on an aerial image and the shape of its orthographic image is greater than the first similarity threshold, or the deviation is less than the first deviation threshold, the aerial image is used as a reference image with the GCP as the target GCP.

[0110] After determining the benchmark image of the target GCP, several aerial images can be selected from the "surroundings" of the benchmark image as evaluation images. Here, "surroundings" means that the evaluation images have camera positions and viewing angles similar to those of the benchmark image. Since aerial images are generally numbered consecutively according to the order in which they were taken, the evaluation images in the aerial image collection are generally the images before and after the benchmark image.

[0111] Figure 9 A schematic diagram of a reference image and several evaluation images on both sides thereof in a specific embodiment is shown, wherein the reference image is marked with annotation points of the target GCP.

[0112] like Figure 9 As shown, the evaluation pictures may include pictures of the target GCP taken at a perspective close to the orthographic projection, or pictures of the target GCP taken at a perspective more deviated from the reference picture. Similarly, the evaluation pictures may also be screened based on the similarity or deviation between the shape of the image of the target GCP formed thereon and the shape of the orthographic projection image of the GCP (or the shape of its image formed on the reference picture), that is, a second similarity threshold or a second deviation threshold may be preset, and only aerial pictures whose similarity with the orthographic projection image of the target GCP (or its image formed on the reference picture) is greater than the second similarity threshold, or whose deviation is less than the second deviation threshold, are selected as evaluation pictures.

[0113] Preferably, the image of the target GCP on the reference image is closer to its positive projection image than the image of the surrounding evaluation image. Therefore, in some preferred embodiments, the first similarity threshold is greater than or equal to the second similarity threshold; in other preferred embodiments, the first deviation threshold is less than or equal to the second deviation threshold.

[0114] In some preferred embodiments, the number of evaluation pictures is not less than 6. For example, 3 aerial pictures can be selected in sequence on both sides of the reference picture as 6 evaluation pictures. In other preferred embodiments, the number of evaluation pictures is not more than 10. For example, 5 aerial pictures can be selected in sequence on both sides of the reference picture as 10 evaluation pictures.

[0115] Step 840 is used to mark the annotation point corresponding to the target GCP on the reference image. The marking in this step can be performed manually, for example, manually selecting a pixel in the image of the target GCP on the reference image (preferably the pixel corresponding to the center of the target GCP), and using the pixel as the annotation point of the target GCP on the reference image; or, it can also be performed semi-automatically in a human-computer interactive manner, for example, first manually defining a rough range, identifying the GCP image therein through an automatic recognition algorithm and marking one of the pixels therein as the annotation point, and then manually adjusting the position of the annotation point.

[0116] It should be noted that although the operation of marking the target GCP in step 840 occurs after the selection of the reference image in step 830, the preferred principle for selecting the reference image is whether a certain GCP can be imaged in an aerial image in a normal projection or a direction close to normal projection. The purpose is to facilitate the accurate marking of the GCP in the aerial image. Therefore, for different target GCPs, different aerial images may need to be selected as their reference images.

[0117] After the target GCP is accurately marked on the reference image, in step 850, as shown in FIG. Figure 10 As shown, first, the marked points of the target GCP are projected onto the coarse grid model along their projection lines corresponding to the reference image (in the embodiment of the present application, the projection lines corresponding to the marked points on the reference image are called reference projection lines), wherein the reference projection lines of the marked points can be determined by the camera orientation elements of the reference image obtained by steps 823 to 827 and stored in the database.

[0118] Camera orientation elements play an important role in fields such as photogrammetry, computer vision, and image processing. They include the camera's intrinsic orientation elements and extrinsic orientation elements. The intrinsic orientation elements describe the camera's internal parameters, which are related to the camera's geometric structure and optical properties. These parameters mainly include focal length, principal point, radial distortion coefficients, tangential distortion coefficients, etc., which are used to characterize how to project from a three-dimensional space point to a two-dimensional image plane; the extrinsic orientation elements describe the camera's spatial position and orientation during shooting, mainly including the camera's three-dimensional position coordinates and rotation angles (Roll, Pitch, Yaw or the corresponding rotation matrix / quaternion), which are used to characterize the camera's position and posture relative to the world coordinate system.

[0119] refer to Figure 3Once the camera orientation elements of an aerial image are determined, for a three-dimensional space point, the straight line from the camera center to this space point is the projection line of the space point corresponding to the aerial image. The intersection of this projection line and the aerial image is the position of its image on the aerial image. Therefore, the projection line shows how the space point is mapped to the image through the internal and external orientation elements of the camera.

[0120] In order to clearly indicate the direction of the projection operation, in the embodiments of the present application, the operation of projecting one or several elements (such as pixels, geometric features, or texture features) from the aerial image to the coarse mesh model along their corresponding projection lines is called projection; the operation of projecting the elements on the coarse mesh model to the aerial image along their projection lines is called reprojection.

[0121] As analyzed above, the coarse mesh model obtained by unconstrained bundle adjustment roughly reflects the three-dimensional terrain of the specified area. The camera orientation element provides the direction in which each pixel on the reference image is projected to its corresponding spatial point. Therefore, after projecting the annotated point of the target GCP onto the surface of the coarse mesh model according to its corresponding reference projection line on the reference image, a projection point will be obtained. The three-dimensional coordinates of the projection point and the orientation of the plane on which it is located on the coarse mesh model (i.e., the normal direction) represent the approximate spatial position and orientation of the target GCP when it is deployed in the specified area.

[0122] Next, step 850 is continued to intercept a surface element around the projection point on the plane where the projection point is located on the coarse mesh model. In this application, this surface element is referred to as the first projection surface element. The first projection surface element can preferably be intercepted with the projection point as the center, or it can be adjusted accordingly based on the actual intersection of the various planes constituting the coarse mesh model (for example, if the surface element intercepted with the projection point as the center happens to cross two planes constituting the coarse mesh model, it is necessary to appropriately offset the first projection surface element so that it is within the same plane). It is only necessary to ensure that the intercepted surface element contains the projection point.

[0123] The shape and size of the first projection surface element can be selected according to the scale of the designated area, the resolution parameters of the aerial image, etc. In some preferred embodiments, the first projection surface element can be in the shape of a square with the projection point as the center of the square. In other embodiments, the shape of the first projection surface element can also be set accordingly according to the actual shape of the target GCP, such as a rectangle, a circle, etc.

[0124] Obviously, the first projection surface element is equivalent to a Mask. Based on the approximate position and orientation information of the target GCP reflected by the projection point on the coarse grid model, it constructs a "window" with a limited shape and size (number of pixels). This is used as the starting point of the search, and the actual position and orientation of the target GCP are obtained by iterative search in step 860.

[0125] In an embodiment of the present application, the search for the actual position and orientation of the target GCP is performed by the following operations: taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to continuously perform translation or rotation operations along the reference projection line, reprojecting the result of each operation onto each evaluation image to obtain a reprojection window for evaluating the correlation, and calculating the correlation of the images in the reprojection window on each evaluation image. When the calculation result meets the correlation criterion, the position of the first projection surface element at this time (the position can be represented by the three-dimensional coordinates of the intersection of the first projection surface element and the reference projection line) and the normal direction can be used as the estimated results of the actual position and orientation of the target GCP.

[0126] Figure 11 、 Figure 12 The operation principle and specific implementation process of step 860 in some preferred embodiments are shown respectively, with reference to Figure 11 、 Figure 12 In step 861, the first projection plane is constructed according to its current position and orientation, and the camera orientation element information of each evaluation picture read from the database is used to construct the projection line of each point on it to each evaluation picture, and reprojection is performed to each evaluation picture along the projection line corresponding to each evaluation picture. After reprojection to the evaluation picture, the area covered on the evaluation picture is as follows Figure 11 The edge of this region, shown in red, constitutes the reprojection window.

[0127] Obviously, during the first reprojection operation, the intersection of the first projection element and the reference projection line is the projection point on the coarse mesh model, and the position and orientation of the first projection element are the position and orientation of the element intercepted in step 850.

[0128] refer to Figure 11 The reprojection window represents the expected position, size and shape (i.e. the position of the reprojection window, the number of pixels covered, and the shape formed) of the target GCP on the corresponding aerial image when it is in a certain alternative posture after being photographed by the cameras at various positions. Obviously, only the first projection surface element that is consistent with the actual posture of the target GCP, the texture features of the image within the reprojection window obtained by reprojecting it to each evaluation image, have the greatest correlation with the target GCP.

[0129] It can be seen that since the rough mesh model has already obtained the approximate pose of the target GCP, using it to determine the initial pose of the first projection surface element can significantly reduce the computational cost of the search. Only a slight translation or rotation around the initial pose is required to accurately estimate the true pose of the target GCP. At the same time, for any alternative pose of the first projection surface element, its position determines the position of the reprojection window on the evaluation image, and its orientation determines the shape and size of the reprojection window on the evaluation image. It can be seen that the texture features of the image within the reprojection window are simultaneously restricted and constrained by factors such as position, pose, shape, and area. Any reprojection window generated according to a pose that deviates from the actual pose by more than a certain degree will cause the image it covers to deviate significantly from the true imaging result of the target GCP on the evaluation image in one or more of the following: position, shape, and area, and further cause a significant decrease in the correlation of the image texture within the reprojection window. Therefore, the above method can greatly improve the accuracy of estimating the actual pose of the target GCP.

[0130] Step 862 is used to evaluate the correlation of the images in each reprojection window. An image correlation evaluation index known to those skilled in the art can be selected to calculate the correlation of the images in the reprojection windows on any two evaluation pictures. For example, an optional correlation evaluation index is the Normalized Cross-Correlation (NCC). Alternatively, other indicators for image correlation can also be selected.

[0131] In some preferred embodiments, the reference picture is also used as the evaluation picture, that is, the first projection bin is also reprojected to the reference picture, and the image in its reprojection window is used to participate in the correlation evaluation.

[0132] After completing the correlation calculation of the images in each reprojection window on each evaluation picture, it is determined whether the calculation results meet the correlation criterion, and step 863 or step 864 is selected for execution based on the judgment result.

[0133] In some optional embodiments, the correlation of the images in each reprojection window may be first statistically analyzed to obtain any one or more of the statistical characteristics, such as the mean, median, maximum, or minimum, and then determine whether the statistical characteristics are greater than or equal to a preset correlation threshold. If the determination result is negative, the first projection plane is translated or rotated in step 863 (see Figure 12, the translation or rotation of the first projection surface element is always performed with the intersection of the first projection surface element and the reference projection line as the movement base point, that is, the projection result of the target GCP's annotation point to its spatial point is always "bound" to the reference projection line), and then return to step 861 to re-project and recalculate the correlation. When the judgment result is yes, execute step 864, and use the current posture of the first projection surface element as its optimal position and optimal orientation (that is, use it as the actual position and orientation of the target GCP obtained by the search), and store the search results in the database.

[0134] Figure 13 The figure shows how the mean NCC of the image in each reprojection window changes with the size of the first projection bin in a specific embodiment. In the figure, the first projection bins are all square bins, and the horizontal axis represents the number of pixels contained in each side of the bin.

[0135] like Figure 13 As shown, in the area where the first projection bin size is small, the NCC value generally shows an upward trend as the bin size increases, indicating that the matching accuracy improves as the bin size increases. However, as the size further increases, the NCC value no longer increases significantly and even fluctuates (such as the fluctuation between 11×11 pixels and 17×17 pixels).

[0136] Analyzing the texture features of each evaluation image reveals that this correlation metric no longer significantly improves, and may even decrease, after the bin size increases to a certain extent. This indicates that as the number of pixels used for correlation calculation increases, additional image noise may be introduced, resulting in a decrease in the contribution of the increased bin size to pose optimization, or even a negative impact. Therefore, the size of the first projection bin cannot be too small, otherwise there will be insufficient information to evaluate the cross-correlation during the subsequent search process. Preferably, the lower limit of the size of the first projection bin is 3×3 pixels; at the same time, the size of the first projection bin does not need to be too large, as further increasing the size will no longer improve the cross-correlation. In some preferred embodiments, the upper limit of the size of the first projection bin can be determined by calculating the maximum value of the correlation statistics of the images within each reprojection window.

[0137] Sub-process III (automatically marking target GCPs on the image to be marked):

[0138] After searching and estimating the actual pose of the target GCP in the specified area through sub-process II, it can be used as a geometric constraint to search and mark its imaging position on the image to be marked. Obviously, using this constraint can effectively limit the search for the target GCP on the image to be marked to a specific area, thereby avoiding the problem of identifying different GCP images in two aerial images as the same GCP.

[0139] In some preferred embodiments, Figure 7 As shown, first, a reference window is created on the reference image in step 870. Specifically, the reference window is determined based on the image of the target GCP formed on it. For example, the reference window can be generated by manually generating the image edge corresponding to the target GCP in the reference image, or by interactive operation of a manual and automatic algorithm. Alternatively, after completely surrounding the image corresponding to the target GCP, the image edge can be appropriately extended by a few pixels from the edge to obtain a larger reference window that surrounds the image corresponding to the target GCP.

[0140] Next, in step 880, the image in the reference window is projected onto the first projection surface element along the reference projection line to obtain the second projection surface element; then in step 890, the second projection surface element is reprojected onto the image to be marked along the projection line corresponding to the image to be marked, and the desired image on the image to be marked can be obtained; after obtaining the desired image, the edge of the desired image is extracted in step 900 to construct a matching window; finally, in step 910, the best position of the matching window is searched on the image to be marked, thereby marking the imaging position corresponding to the target GCP on the image to be marked. In the embodiment of the present application, Figure 7 As shown, after the target GCP is marked on a picture to be marked, the picture to be marked with the target GCP can be replaced and the above operation can be repeated.

[0141] The search for the optimal position of the matching window in step 910 can be implemented using a least squares image matching algorithm. Figure 14 In some preferred embodiments, the implementation principle of step 910 is shown as follows: Figure 14 As shown in the figure, the matching window can be slid translationally on the image to be marked according to the set sliding step size, and the image within the matching window is extracted, and the minimum error square calculation is performed on its texture features and the texture features of the expected image. The above process is repeated until the position with the minimum error is found. The image surrounded by the matching window at this position represents the position where the target GCP is imaged, thereby completing the automatic marking of the target GCP.

[0142] Obviously, the position and orientation of the second projection surface element are consistent with the first projection surface element with the optimal position and orientation. The difference between the two is that the second projection surface element adds the shape and texture feature information expected when the corresponding image of the target GCP on the reference image is projected to the actual pose of the target GCP. Therefore, when the multiple information with this specific position, orientation, shape and texture features is further reprojected onto the image to be marked, the obtained expected image can roughly limit the expected imaging position of the target GCP on the image to be marked and the expected shape, size and texture of the image. The target GCP is searched using this strongly constrained information, which eliminates the possibility of identifying different GCPs in two aerial images as the same one. At the same time, it only needs to translate and slide the matching window in a smaller area around the expected image, without the need for large-scale search and rotation operations in conventional image matching retrieval, which undoubtedly greatly improves the search speed.

[0143] Figure 14 In the illustrated embodiment, since the desired image is completely displayed on the picture to be marked, the matching window can be constructed only through the edge of the desired image. Figure 15 The illustrated embodiment further demonstrates the search for a target GCP on an aerial image that partially displays the target GCP. When the target GCP is located at the edge of an aerial image, only a portion of it may be imaged on the aerial image. For conventional automatic labeling methods based solely on texture information, the texture features used for search and comparison are obviously missing, making recognition difficult. However, with the method of the present application, since the desired image is imaged at the edge of the aerial image according to its expected position and only a portion of it is located on the aerial image, the edge of the desired image and the edge of the aerial image itself can be used to jointly construct a matching window, and its optimal position can be searched on the image to be labeled. In other words, the method of the present application can complete the identification and labeling of the target GCP using only partial information on the image to be labeled. Specific embodiments

[0145] This embodiment adopts the above-mentioned automatic labeling method of target GCP to automatically label target GCP in a collection of aerial images (Dort-ZECHE dataset) published by the International Society for Photogrammetry and Remote Sensing (ISPRS) with a building as the designated area.

[0146] Figure 16 shows some aerial pictures obtained by aerial photography of the designated area, such as Figure 16 As shown, different aerial pictures capture different GCPs in different numbers, positions, and shapes. The target GCP automatic labeling method provided in this application is used to automatically label the target GCPs in the aerial picture set, where the size of the first projection bin is 11×11 pixels. Figure 17 、 Figure 18 Schematic diagrams of the three-dimensional sparse point cloud and coarse mesh model generated during the implementation process are shown respectively.

[0147] In order to evaluate the effectiveness of the method provided in this application, 10 GCPs set on the buildings were used as target GCPs, and the target GCPs were automatically marked on the aerial images in the dataset. The following Table 1 summarizes the automatic marking results.

[0148] Table 1 Statistics of target GCP automatic labeling results

[0149]

[0150] It can be seen from Table 1 that when the reference window size is 15×15, a total of 482 points are successfully marked, with a success rate of about 67.3%, and the root mean square error between the coordinates of these 482 marked points and the true coordinates is 0.564 pixels; when the window size is 25×25, a total of 683 points are successfully marked, with a success rate of about 95.5%, and the root mean square error between the coordinates of these 683 marked points and their true coordinates is 0.549 pixels. The statistical results show that the target GCP automatic marking method used in this application can automatically mark more than 700 GCPs to be marked within 1 minute, and as the reference window size increases to 25×25 pixels, its marking success rate exceeds 95%. At the same time, the comprehensive accuracy of marking each target GCP reaches the sub-pixel level, proving its comprehensive improvement in marking speed and marking accuracy compared to manual marking methods.

[0151] The above is a detailed introduction to the specific implementation methods of the present application. For those skilled in the art, several improvements and modifications can be made to the present application without departing from the principles of the present application. These improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for automatically labeling target GCPs, characterized in that: The following operations are included: a. Build a coarse grid model based on a collection of aerial images of a specified area, where the target GCPs are located within the specified area; b. determining a marking point of a target GCP on a reference image, where the marking point is located in an image of the target GCP on the reference image; c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element; d. searching and determining an optimal position and an optimal orientation of the first projection plane based on correlation of images within a reprojection window obtained by reprojecting the first projection plane onto a plurality of evaluation images; e. Projecting an image within a reference window on the reference image onto a first projection plane having an optimal position and optimal orientation to obtain a second projection plane, wherein the image of the target GCP on the reference image is located within the reference window; f. Based on the desired image obtained by reprojecting the second projection bin onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled; The similarity between the image of the target GCP on the reference image and its orthographic image is greater than a preset first similarity threshold, or the deviation is less than a preset first deviation threshold; The similarity between the image of the target GCP on the evaluation picture and its orthographic image or its image on the reference picture is greater than a preset second similarity threshold, or the deviation is less than a preset second deviation threshold, wherein the first similarity threshold is greater than or equal to the second similarity threshold, and the first deviation threshold is less than or equal to the second deviation threshold.

2. The automatic marking method of target GCP according to claim 1, characterized in that The coarse grid model is generated by performing an unconstrained bundle adjustment operation on a set of aerial pictures of a specified area, wherein the coarse grid model uses each point in a sparse point cloud obtained by performing an unconstrained bundle adjustment on a set of aerial pictures of a specified area as a grid point.

3. The automatic marking method of target GCP according to claim 1, characterized in that, Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element in operation c includes: The marked point is projected onto the surface of the coarse mesh model along a reference projection line corresponding to the reference image to obtain its projection point on the surface of the coarse mesh, wherein the reference projection line corresponding to the reference image of the marked point is determined based on the camera orientation element of the reference image.

4. The automatic marking method of target GCP according to claim 3, characterized in that, The first projection surface element in operation c is determined by intercepting a region surrounding the projection point on the surface of the coarse mesh model.

5. The automatic marking method of target GCP according to claim 1, characterized in that, In operation d, the first projection bin is reprojected onto a plurality of evaluation images, including: Each point on the first projection surface element is reprojected onto each evaluation picture along its projection line corresponding to each evaluation picture, and the edge of each point reprojected onto each evaluation picture is used as its reprojection window on each evaluation picture, wherein the projection line of each point on the first projection surface element corresponding to each evaluation picture is determined based on the camera orientation element of each evaluation picture.

6. The automatic marking method of target GCP according to claim 5, characterized in that: The searching and determining of the optimal position and optimal orientation of the first projection plane element in operation d includes: Taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to perform a translation operation or a rotation operation along the reference projection line; Reproject the result of each translation or rotation operation onto each evaluation image to obtain a reprojection window, and calculate the correlation of the image of each evaluation image in its reprojection window; When the calculation result meets the correlation criterion, the position and orientation of the first projection plane element at this time are taken as the optimal position and optimal orientation of the first projection plane element, otherwise the translation operation or rotation operation of the first projection plane element is performed again.

7. The automatic marking method of target GCP according to claim 6, characterized in that: The correlation criterion is that the correlation statistics of the images of each evaluation picture within its reprojection window meet a preset correlation threshold; The lower limit of the area of the first projection bin is 3×3 pixels, and the upper limit of the area is determined based on the correlation statistics results of the images in the reprojection window of each evaluation picture.

8. The automatic marking method of target GCP according to claim 1, characterized in that: The expected image is an image formed by reprojecting each point on the second projection plane onto each picture to be marked along its projection line corresponding to each picture to be marked; as well as, The search in operation f to determine the imaging position corresponding to the target GCP on the image to be labeled includes: Extracting edges of the desired image to construct a matching window, wherein the matching window is constructed only by the edges of the desired image, or jointly by the edges of the desired image and the edges of the picture to be marked where the desired image is located; The best position of the matching window is searched on the image to be marked, so as to mark the imaging position corresponding to the target GCP on the image to be marked.

9. The automatic marking method for target GCP according to claim 8, characterized in that: In the process of searching for the best position of the matching window on the image to be marked, the matching window is only translated on the image to be marked.

Images