Automatic marking method for target GCP

Abstract

The invention provides an automatic marking method for a target GCP. The automatic marking method comprises the following steps: establishing a rough mesh model based on an aerial photo set of a specified area; determining a marking point of the target GCP on a reference picture; projecting the marking point to the surface of the rough mesh model and intercepting a first projection surface element; searching and determining the optimal position and the optimal orientation of the first projection surface element based on the correlation of images in a re-projection window obtained by re-projecting the first projection surface element to the plurality of evaluation pictures; and projecting an image in a reference window on the reference picture to the first projection surface element with the optimal position and the optimal orientation to obtain a second projection surface element, and searching and determining a corresponding imaging position of the target GCP on the to-be-marked picture based on an expected image obtained by re-projecting the second projection surface element to the to-be-marked picture. According to the method provided by the invention, the credibility and accuracy of automatically marking the target GCP on the to-be-marked picture are effectively improved.

Description

Technical Field

[0001] The present application belongs to the technical field of three-dimensional terrain reconstruction based on aerial photography measurement, and relates to GCP recognition technology in aerial image processing. Specifically, a method for automatically marking a target GCP is provided. Background Art

[0002] Aerial photogrammetry is a remote sensing technology that uses aircraft (such as airplanes, helicopters or drones) equipped with photographic equipment to photograph designated ground areas, and obtains three-dimensional information and measurement data of terrain and objects by analyzing and processing these image data. It combines the principles of aerial photography and surveying, and can be widely used in land and resources exploration, surveying of major infrastructure projects, natural disaster monitoring and ecological environmental protection, geographic information systems (GIS), environmental monitoring, three-dimensional terrain modeling, land and resources management and other fields.

[0003] Ground Control Points (GCPs) play a vital role in three-dimensional terrain modeling based on aerial photography. As basic data, they are of key significance in improving product accuracy. After triangulation of aerial images, marking GCPs is an essential step. By marking GCPs on aerial pictures or images, the positioning accuracy of the data can be improved and the measurement results can be effectively integrated into the coordinate system.

[0004] The traditional GCP marking method is to manually mark GCP on each image, which not only consumes a lot of time, but also significantly reduces the efficiency of product production. In order to improve the speed of GCP identification and marking, feature detection algorithms can be used to automatically extract GCPs in images. In recent years, with the continuous development of artificial intelligence and deep learning technology, methods for automatic GCP identification using various deep learning networks have emerged (Gong Xiaoqiang, Zou Jingui, Meng Liyuan. Automatic extraction of close-range photogrammetry control points based on target detection and corner detection [J]. Bulletin of Surveying and Mapping, 2020, (S1): 173-175+180; Ding Tao. Research on automatic layout and identification of image control points for UAV oblique photogrammetry [D]. Anhui University of Science and Technology, 2021).

[0005] However, the above-mentioned existing technologies still have problems when facing small fields of view, large deformations, large inclinations and a large number of aerial image data. The reasons are: first, in images with different perspectives, due to perspective deformation, the same GCP may show obvious deformation and image distortion in different pictures. When the degree of deformation exceeds the training range of the deep learning network used for recognition, accurate recognition will not be possible; in addition, the images of GCPs located at the edge of some pictures cannot be fully displayed, and lack complete information for feature detection, resulting in the inability of existing algorithms to use complete GCP image features to identify them; more importantly, various existing automatic recognition algorithms simply compare and match parts with similar texture features on pictures with different perspectives, without considering the connection between the positions of the same GCP in pictures with different perspectives, which will cause a GCP to be incorrectly marked at the location of other GCPs during the automatic recognition process. Summary of the invention

[0006] In order to solve the problems existing in the above-mentioned prior art, the present application provides a method for automatically marking a target GCP, which comprises the following operations:

[0007] a. Establish a coarse grid model based on a set of aerial photos of a specified area, where the target GCP is located in the specified area;

[0008] b. determining a marking point of a target GCP on a reference picture, where the marking point is located in an image of the target GCP on the reference picture;

[0009] c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection facet;

[0010] d. searching and determining the best position and the best orientation of the first projection surface element based on the correlation of the images in the reprojection window obtained by reprojecting the first projection surface element onto a plurality of evaluation images;

[0011] e. Projecting an image in a reference window on the reference picture to a first projection plane element having an optimal position and an optimal orientation to obtain a second projection plane element, wherein the image of the target GCP on the reference picture is located in the reference window;

[0012] f. Based on the desired image obtained by reprojecting the second projection surface element onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled.

[0013] Furthermore, the coarse grid model is generated by performing an unconstrained bundle adjustment operation on a set of aerial pictures of a designated area.

[0014] Furthermore, the coarse grid model uses points in a sparse point cloud obtained by performing an unconstrained bundle adjustment on a set of aerial pictures of a designated area as grid points.

[0015] Preferably, the similarity between the image of the target GCP on the reference image and its orthographic image is greater than a preset first similarity threshold, or the deviation is less than a preset first deviation threshold.

[0016] Preferably, the similarity between the image formed by the target GCP on the evaluation picture and its orthographic image or its image formed on the reference picture is greater than a preset second similarity threshold, or the deviation is less than a preset second deviation threshold, wherein the first similarity threshold is greater than or equal to the second similarity threshold, and the first deviation threshold is less than or equal to the second deviation threshold.

[0017] Furthermore, the operation c described in projecting the marked point to the surface of the coarse mesh model and intercepting the first projection face element includes: projecting the marked point along its reference projection line corresponding to the reference image to the surface of the coarse mesh model and obtaining its projection point on the surface of the coarse mesh, wherein the reference projection line of the marked point corresponding to the reference image is determined based on the camera orientation element of the reference image.

[0018] Furthermore, the first projection surface element intercepted in operation c is determined by intercepting a region surrounding the projection point on the surface of the coarse mesh model.

[0019] Furthermore, in operation d, the first projection surface element is reprojected to a plurality of evaluation pictures, including:

[0020] Each point on the first projection surface element is reprojected to each evaluation picture along its projection line corresponding to each evaluation picture, and the edge of each point reprojected to each evaluation picture is used as its reprojection window on each evaluation picture, wherein the projection line corresponding to each evaluation picture of each point on the first projection surface element is determined based on the camera orientation element of each evaluation picture.

[0021] Furthermore, the searching and determining of the best position and the best orientation of the first projection plane element in operation d includes:

[0022] Taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to perform a translation operation or a rotation operation along the reference projection line;

[0023] Reproject the result of each translation operation or rotation operation onto each evaluation image to obtain a reprojection window, and calculate the correlation of the image of each evaluation image in its reprojection window;

[0024] When the calculation result satisfies the correlation criterion, the position and orientation of the first projection plane element at this time are taken as the optimal position and optimal orientation of the first projection plane element, otherwise the translation operation or rotation operation of the first projection plane element is performed again.

[0025] Preferably, the correlation criterion is that the correlation statistics of the images of each evaluation picture within its reprojection window meet a preset correlation threshold.

[0026] Preferably, the lower limit of the area of ​​the first projection surface element is 3×3 pixels, and the upper limit of the area is determined based on the correlation statistics of the images in the reprojection window of each evaluation picture.

[0027] Furthermore, the expected image is an image obtained by reprojecting each point on the second projection plane element onto each picture to be marked along its projection line corresponding to each picture to be marked.

[0028] Furthermore, the searching and determining of the imaging position corresponding to the target GCP on the image to be marked in operation f includes:

[0029] Extracting the edge of the expected image to construct a matching window, wherein the matching window is constructed only by the edge of the expected image, or is jointly constructed by the edge of the expected image and the edge of the picture to be marked where the expected image is located;

[0030] The best position of the matching window is searched on the image to be marked, so as to mark the imaging position corresponding to the target GCP on the image to be marked.

[0031] Preferably, the best position of the matching window is searched on the image to be marked by using a least squares image matching algorithm.

[0032] Preferably, in the process of searching for the best position of the matching window on the image to be marked, the matching window is only translated on the image to be marked.

[0033] An embodiment of the present application provides an automatic labeling method for a target GCP, which generates a coarse grid model of a specified area to provide an initial "index" for the step of searching for the real spatial position and posture of the target GCP; after projecting the target GCP accurately marked on a reference image to the coarse grid, under the constraint of the coarse grid model, the most likely position and orientation of the target GCP in the actual physical space is searched according to the result of reprojection to the evaluation image; further, based on this most likely position and orientation, the expected position, shape and texture of the target GCP on the image to be evaluated are obtained, and this is used as the basis for searching the position of the target GCP on the image to be evaluated, thereby effectively avoiding the problems of existing solutions that rely solely on texture features for automatic GCP labeling, such as poor recognition effect when the distortion is large or part of the texture is missing, and incorrect association of different GCPs with similar texture features. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flowchart of an existing three-dimensional terrain modeling method based on aerial photography measurement;

[0035] Figure 2 is a schematic diagram of a pattern of ground control points;

[0036] Figure 3 This is a schematic diagram of the principle of aerial triangulation based on aerial images;

[0037] Figure 4 It is a specific flow chart of an existing aerial triangulation measurement using multiple aerial photos and GCP information;

[0038] Figure 5 This is a schematic diagram of the image of the same GCP on several aerial photos;

[0039] Figure 6 A flow chart of the automatic labeling method for the target GCP provided in this application;

[0040] Figure 7 A schematic diagram of a specific implementation process of the automatic labeling method of target GCP provided according to some embodiments of the present application;

[0041] Figure 8 A schematic diagram of a specific implementation process of establishing a coarse grid model of a specified area in some embodiments;

[0042] Fig. 9 is a schematic diagram of a reference picture and a plurality of evaluation pictures on both sides thereof in a specific embodiment;

[0043] Fig.10 A schematic diagram of the operation principle of the first projection plane element is captured in some specific embodiments;

[0044] Fig.11 A schematic diagram of the principle of searching and determining the best position and the best orientation of the first projection plane element in some specific embodiments;

[0045] Fig.12 A specific implementation flow chart of searching and determining the best position and the best orientation of the first projection plane element in some specific embodiments;

[0046] Fig.13 is a schematic diagram showing, in a specific embodiment, how the NCC mean value of the image in each reprojection window changes with the change of the size of the first projection bin;

[0047] Fig.14 A schematic diagram of searching for an imaging position of a target GCP on a to-be-marked picture in some embodiments;

[0048] Fig.15 A schematic diagram of searching for an imaging position of a target GCP on a to-be-marked picture in some embodiments;

[0049] Fig.16 A schematic diagram of an aerial picture of a designated area provided according to an embodiment of the present application;

[0050] Fig.17 A schematic diagram of a three-dimensional sparse point cloud of a specified area provided according to an embodiment of the present application;

[0051] Fig.18 A schematic diagram of a coarse grid model of a specified area provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] Hereinafter, the present application will be further described based on preferred embodiments with reference to the accompanying drawings.

[0053] In the description of the embodiments of the present application, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear to indicate an orientation or position relationship, it is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the products of the embodiments of the present application are usually placed when used. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0054] The vocabulary in this specification is used to illustrate the embodiments of the present application, but is not intended to limit the present application. It should also be noted that, unless otherwise clearly specified and limited, the terms "disposed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, an indirect connection through an intermediate medium, or a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in this application can be specifically understood.

[0055] In order to clearly explain the automatic marking method of target feature points provided in the present application, the process of the existing three-dimensional terrain modeling method based on aerial photography measurement and its existing problems are first introduced.

[0056] Figure 1 A flowchart of an existing three-dimensional terrain modeling method based on aerial photography is shown. Figure 1 As shown, in order to generate a three-dimensional terrain model of a specified area, firstly, a number of GCPs (Ground Control Points) are arranged in the specified area through step 100, and the spatial position of each GCP is measured with high precision using a total station, GPS, etc.

[0057] Generally, the shape of the ground control point is a square, and its side length can be determined according to the aerial photography height, shooting resolution, etc. For example, it can be made of 50cm×50cm, or 1m×1m, etc. It can be made of various plates or soft fabrics to facilitate placement on the ground, buildings or natural objects in the designated area. In order to facilitate identification from aerial photos, its surface can be printed or painted, such as Figure 2 The pattern shown has more distinct features and higher contrast.

[0058] Step 200 is used to formulate an aerial photography plan, including selecting a suitable aerial photography platform (aircraft, helicopter, drone, etc.) and aerial photography camera according to the terrain conditions of the designated area, and planning the flight altitude, route, and aerial photography parameters (such as shooting angle, aerial photography overlap, etc.). Generally, the overlap between consecutive aerial photos is usually set between 60% and 80% to ensure the reliability of subsequent feature matching.

[0059] In step 300, aerial photography is performed on a designated area according to an aerial photography plan to obtain an aerial image of the area.

[0060] Generally, aerial images are multiple aerial pictures or continuous aerial videos taken according to an aerial photography plan. The original aerial images generally need to be converted according to the requirements of subsequent data processing, such as unifying the image size, or converting continuous videos into multiple aerial pictures to meet the needs of subsequent data processing.

[0061] Next, in step 400, aerial triangulation is performed using multiple aerial images and GCP information of the designated area.

[0062] Aerial triangulation, referred to as aerial triangulation, is one of the key steps in the three-dimensional terrain modeling method based on aerial photography. Figure 3 As shown, in this step, through geometric analysis of multiple overlapping aerial images, the camera's orientation elements are estimated and the positions of the three-dimensional points in the scene are reconstructed, thereby obtaining a three-dimensional sparse point cloud of the specified area.

[0063] The number of aerial images required for aerial triangulation depends on many factors, including the size, complexity, required resolution, overlap, etc. of the specified area. For example, for small to medium-sized areas (such as small towns, markets, etc.), 20 to 50 aerial images are usually required. For high-precision measurement and analysis of large areas (such as mines, forests, cities, etc.), more than 50 to hundreds of high-resolution aerial images may be required to ensure adequate coverage and details.

[0064] Figure 4 The specific process of aerial triangulation (step 400) performed using multiple aerial images and GCP information in an existing embodiment is shown.

[0065] like Figure 4 As shown, first in step 410, the GCPs in each aerial image are marked. When the aerial triangulation algorithm adopts the conventional constrained bundle adjustment method to determine the camera parameters and the three-dimensional position of each feature point, marking the GCP is a necessary prerequisite step. This is because the GCP has accurate spatial position information. Therefore, it can be used as a constraint condition in the subsequent constrained bundle adjustment to ensure the consistency of the output result with the known geographic reference.

[0066] The means of marking GCPs can be as described in the background technology, by manually calibrating GCPs on each aerial image, or by automatically extracting GCPs from each aerial image using a trained deep learning network.

[0067] Generally, Figure 2 After the GCP shown is arranged in the specified area, the image formed on the aerial picture is a collection of a group of pixels with a certain area. Furthermore, marking the GCP on the aerial picture means specifying at least one pixel for the GCP to represent the position of the GCP on the aerial picture. The pixel is also called the annotation point (or marking point) of the GCP on the aerial picture.

[0068] Figure 5The figure shows the images of the same GCP on several aerial images and the results of marking them. Obviously, in an ideal situation, the pixel corresponding to the center of the GCP on the aerial image can be used as the marking point; in the actual marking process, the pixel position identified as the marking point may have a deviation of several pixels from the pixel position actually corresponding to the center of the GCP on the aerial image. Therefore, when the deviation between the GCP's marking point and its actual position on a certain aerial image does not exceed a preset threshold, it can be considered that the GCP has been accurately marked on the aerial image.

[0069] Next, in step 420, feature points are extracted from each aerial image (algorithms such as SIFT and SURF can be used). Generally, these feature points are points with features such as high contrast, edges or corners that are automatically extracted from the aerial images by the feature extraction algorithm. After the feature points are extracted, in step 430, different aerial images are registered. Specifically, the same feature points can be matched between different aerial images through a feature matching algorithm. This step can establish the correspondence between the aerial images and identify the corresponding feature points at different shooting angles.

[0070] It should be noted that the pixel points of the image formed by GCP in the aerial image can also be regarded as a kind of feature points, because they also contain specific texture information in the image, and therefore can also participate in the feature point matching process.

[0071] After completing steps 420 and 430, in step 440, under the constraint of the GCP information, the camera's internal orientation elements (including the camera's focal length, principal point, distortion coefficient, etc.) and external orientation elements (including the camera's position and orientation) are iteratively estimated through bundle adjustment, and based on the matching relationship between the camera's orientation elements (internal / external orientation elements) and feature points, the three-dimensional coordinates of each feature point are iteratively estimated, thereby establishing a three-dimensional sparse point cloud of the specified area.

[0072] Bundle adjustment is a mathematical method for optimizing camera parameters and 3D point positions. After determining the initial estimated values ​​of the camera orientation elements through simple geometric relationships (such as triangulation), it uses an iterative optimization method (such as the Gauss-Newton method) to repeatedly adjust the camera orientation elements and 3D point coordinates with the goal of minimizing the sum of squares of all reprojection errors until convergence conditions are reached. Obviously, when the known GCP coordinates are introduced as constraints in the iterative optimization process, the characteristics of their known precise positions can be used to ensure that the estimated values ​​of the camera parameters and 3D points conform to the known GCP coordinates, thereby improving the accuracy and precision of the 3D point estimation.

[0073] The density of the 3D sparse point cloud generated in step 400 is too low to be directly used to build a 3D terrain model. Therefore, in step 500, the 3D sparse point cloud is further densified by using image matching and reconstruction algorithms such as PatchMatch and SGM (Semi-Global Matching), thereby obtaining a 3D dense point cloud of the specified area. Then, in step 600, the 3D dense point cloud is converted into a 3D fine mesh model using mesh reconstruction algorithms such as Delaunay triangulation and Poisson reconstruction, thereby completing the construction of a 3D terrain model of the specified area.

[0074] Preferably, in step 700, the three-dimensional fine grid model of the specified area may be colored, rendered, and visualized, or terrain features such as elevation and slope may be extracted, or further processed such as measurement and analysis may be performed.

[0075] Obviously, in the above process of three-dimensional terrain modeling for the specified area, whether the GCP in each aerial image can be accurately marked through step 410 determines whether the correspondence between the imaging position of the GCP in the aerial image and its actual position coordinates is correct (that is, whether the bundle adjustment can be correctly constrained), which in turn has a key impact on the estimation results of the camera parameters and the three-dimensional coordinates of the feature points.

[0076] Manual processing can accurately identify and mark each GCP in each aerial image, and accurately associate the same GCP in different aerial images. However, when the number of aerial images is large, the processing speed obviously cannot meet the processing speed requirements.

[0077] In order to solve the problem of low efficiency of manual GCP marking mentioned above, several methods have emerged to automatically identify GCP images from various aerial photos. For example, Gong Xiaoqiang et al. used the Faster R-CNN network combined with corner detection to identify GCP images in pictures (Gong Xiaoqiang, Zou Jingui, Meng Liyuan. Automatic extraction of close-range photogrammetry control points based on target detection and corner detection [J]. Bulletin of Surveying and Mapping, 2020, (S1): 173-175+180); for example, Ding Tao used the deep learning SSD enhancement algorithm to train the model, combined with the LSD line segment detection algorithm to extract the image angle threshold of the control point, so as to identify the geometric center mark of the control point of the screened image (Ding Tao. Research on automatic layout and identification of image control points in UAV oblique photogrammetry [D]. Anhui University of Science and Technology, 2021).

[0078] Although the above scheme can automatically identify the image of the landmark point, there are still the following problems in its application:

[0079] 1) The same GCP may have great shape deformation and imaging area differences in aerial images of different perspectives. For example, a GCP image that is square in one aerial image (for example, the shooting perspective is just facing the GCP on the ground) may be compressed into a very thin rectangle in another aerial image (for example, the shooting perspective is almost perpendicular to the normal direction of the GCP on the ground). Since the texture information of the two is too different, neither calculating correlation nor relying on deep networks can solve this problem.

[0080] 2) When a GCP is imaged at the edge of an aerial image, the image may only include a partial image of the GCP. In this case, conventional image recognition methods may fail to correctly identify the GCP due to the lack of complete image information.

[0081] 3) The image of a GCP in a certain aerial image may be mistakenly associated with the image of another GCP in another aerial image for some reason. The reason for this problem may be image distortion caused by perspective deformation, or it may be caused by the terrain structure of the specified area. For example, there are some relatively similar and periodically arranged buildings, which causes the recognition algorithm to recognize two different GCPs with different spatial positions but similar texture features in aerial images taken at two different locations as the same.

[0082] The applicant analyzed the causes of the above-mentioned problems and found that when using feature extraction or deep learning networks for automatic GCP identification, since the camera posture and shooting angle information have not been obtained through bundle adjustment, the identification basis is only the texture features of the image formed by the GCP in each aerial picture, and does not consider the position and shape of the GCP in the aerial pictures with a specific shooting angle. That is, even if the above-mentioned recognition algorithm can extract the image of each ground feature point from each aerial picture through texture features, it cannot associate the same ground feature points on different aerial pictures. Therefore, no matter how the feature extraction algorithm or deep learning network structure is optimized, or how the training degree of the deep learning network is increased, the root cause of the above-mentioned problems cannot be eliminated.

[0083] In order to solve the above problems, an embodiment of the present application provides a new method for automatic marking of target GCPs, which can replace the current method of manually marking aerial images one by one, and can automatically mark any artificially specified GCP (in the embodiment of the present application, the artificially specified GCP is called the target GCP) in each aerial image that captures the GCP, and at the same time can significantly improve the deficiencies existing in the implementation process of the current GCP automatic marking method.

[0084] Figure 6A flow chart of an automatic labeling method for a target GCP provided by the present application is shown, such as Figure 6 As shown, the method includes the following operations:

[0085] a. Establish a coarse grid model based on a set of aerial photos of a specified area, where the target GCP is located in the specified area;

[0086] b. determining a marking point of a target GCP on a reference picture, where the marking point is located in an image of the target GCP on the reference picture;

[0087] c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection facet;

[0088] d. searching and determining the best position and the best orientation of the first projection surface element based on the correlation of the images in the reprojection window obtained by reprojecting the first projection surface element onto a plurality of evaluation images;

[0089] e. Projecting an image in a reference window on the reference picture to a first projection plane element having an optimal position and an optimal orientation to obtain a second projection plane element, wherein the image of the target GCP on the reference picture is located in the reference window;

[0090] f. Based on the desired image obtained by reprojecting the second projection surface element onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled.

[0091] Figure 7 In some embodiments, the specific implementation process of the method is shown. Figure 6 , Figure 7 As shown, the method mainly includes three sub-processes, among which, sub-process I (corresponding to operation a) is used to generate a coarse grid model of the specified area, and the coarse grid model can provide an initial "index" for the step of searching the real space position and posture of the GCP described later; in sub-process II (corresponding to operations b to d), after the target GCP accurately marked on the reference image is projected to the coarse grid, under the constraint of the coarse grid model, the most likely position and orientation of the target GCP in the actual physical space are searched according to the result of reprojection to the evaluation image; in sub-process III (corresponding to operations e and f), the expected position, shape and texture of the target GCP on the image to be evaluated are obtained according to this most likely position and orientation, and this is used as the basis for searching the position of the target GCP on the image to be evaluated, thereby effectively avoiding the existing scheme of automatic GCP marking that relies solely on texture features, which has poor recognition effect when the distortion is large or part of the texture is missing, and the problem of incorrectly associating different GCPs with similar texture features.

[0092] The specific implementation of this method is described in detail below.

[0093] Sub-process I (establishing a coarse grid model of a specified area):

[0094] like Figure 7 As shown, in sub-process I, first, step 810 is used to read a set of aerial pictures of the specified area. Specifically, as described above in the three-dimensional terrain modeling technology based on aerial photography, a number of GCPs can be pre-deployed in the specified area, and then the designated area is aerially photographed according to the aerial photography plan, and the acquired pictures, videos, etc. are processed to generate multiple aerial pictures that meet the overlap requirements (obviously, any GCP arranged in the specified area will be imaged in at least one aerial picture). The aerial picture set consisting of all aerial pictures of the above-mentioned specified area can be stored in a database and read and processed in the process of modeling the specified area.

[0095] After the aerial picture set of the designated area is read, a coarse grid model of the designated area can be established in step 820. The construction of the coarse grid model can be achieved by performing a bundle adjustment operation on each aerial picture in the aerial picture set.

[0096] It should be emphasized that in the method provided in the present application, since GCPs are not marked on the aerial images before executing step 820, this step adopts the bundle adjustment that does not use GCP position coordinates for constraints, that is, in the process of establishing a coarse grid model through a set of aerial images, it does not rely on external known ground control points for constraints.

[0097] Figure 8 The specific implementation process of step 820 is shown in some preferred embodiments, wherein the estimation of the camera orientation elements and the three-dimensional coordinates of the feature points adopts the unconstrained bundle method free network adjustment. Specifically, first in step 821, feature points are extracted in each aerial picture by a feature detection algorithm (such as SIFT, SURF or ORB, etc.); then in step 822, feature points of different aerial pictures are matched, and this step can use methods such as nearest neighbor search to match the same feature points in aerial pictures of different perspectives; then, the unconstrained bundle method free network adjustment operation described in steps 823 to 827 is used to iteratively optimize the estimated values ​​of the camera orientation elements and the three-dimensional coordinates of each feature point until the objective function converges or the number of iterations reaches an upper limit.

[0098] The estimation of the camera orientation elements and the three-dimensional coordinates of the feature points involved in the above-mentioned bundle free network adjustment operation, as well as the form of the objective function are all known to those skilled in the art. For example, the camera internal orientation elements and external orientation elements can be initially estimated through the description documents of the camera parameter specifications and the flight position and attitude records of the aircraft during aerial photography, and the preliminary estimation of the three-dimensional coordinates of the feature points can be performed respectively through the triangulation method, and the estimated values ​​are updated through the nonlinear least squares method such as Levenberg-Marquardt. The common form of the objective function is generally the minimization of the reprojection error. Without departing from the inventive concept of the present application, a specific algorithm can be reasonably selected to implement the above-mentioned unconstrained bundle adjustment operation.

[0099] After completing the above operations, the final estimated values ​​of the camera orientation elements and the three-dimensional coordinates of the feature points can be determined (step 827). The above-mentioned multiple feature points with three-dimensional coordinate values ​​are used as point cloud points to form a sparse point cloud of the specified area. Each point cloud point with a determined three-dimensional coordinate value is used as a grid point, and algorithms such as Delaunay triangulation are used to obtain a coarse grid model of the specified area (step 828).

[0100] like Figure 8 As shown, the camera orientation elements obtained in step 820 and the sparse point cloud and coarse grid model of the specified area are stored in the database for subsequent calling in the implementation of sub-process II and sub-process III.

[0101] It should be noted that in the existing 3D terrain modeling method based on aerial photography measurement, after obtaining the sparse point cloud, a coarse mesh model is not generated based on it, but it is directly densified to obtain a fine mesh model of the specified area and perform operations such as texture mapping. In the embodiment of the present application, the purpose of establishing the coarse mesh model is to provide a basis for estimating the actual pose of the target GCP in sub-process II.

[0102] Sub-process II (estimating the actual position of the target GCP in the specified area):

[0103] The surface of the coarse grid model reflects the general shape of the three-dimensional terrain of the specified area, that is, the normal direction and coordinates of each plane on the surface of the coarse grid model can roughly characterize the orientation and spatial position of the specified area at that location. Therefore, a landmark point of a target GCP accurately calibrated on a reference image can be projected onto the coarse grid model to obtain an initial face element as a search starting point. The orientation and projection distance of the face element are continuously changed within a smaller constrained range and reprojected to several evaluated images. The correlation of the images in the reprojected area on each evaluated image is used to determine the optimal orientation and projection distance of the face element. In this way, the most likely orientation and position of the target GCP when it is deployed in the specified area can be obtained.

[0104] Back to Figure 7 In some specific embodiments, sub-process II includes steps 830 to 860.

[0105] In step 830, a reference picture may be selected and several evaluation pictures may be selected around the reference picture according to the imaging conditions of the target GCP to be automatically marked on each aerial picture.

[0106] In order to ensure the accuracy of subsequent search results, the reference image should be selected so that the target GCP is photographed at an orthographic angle as much as possible. That is, the aerial image whose image corresponding to the target GCP is close to its orthographic shape should be selected as the reference image. For example, when the actual shape of the target GCP is Figure 2 When the square shown is selected, the aerial pictures in which the target GCP is captured and the image shape is also basically a square should be selected from the aerial picture collection. In some preferred embodiments, the deviation or similarity can be used to evaluate the deviation or similarity between the shape of the image formed by a GCP in an aerial picture and the shape it should have when it is photographed according to the projection angle. For example, the Hausdorff distance, average distance or similarity ratio can be used to measure the geometric difference between the imaged image and the ideal square; for another example, the difference between the length of each side (or perimeter) of the image formed by the GCP in the aerial picture and the length of each side (or perimeter) of a standard square with the same area can be used as a deviation index:

[0107]

[0108] Among them, D is the deviation index, l i is the length of each side of the image formed by a GCP on the aerial photo, l square is the side length of a square of equal area to the image.

[0109] Obviously, a preset similarity threshold (first similarity threshold) or deviation threshold (first deviation threshold) can be set. When the similarity between the shape of the image formed by a GCP in an aerial image and the shape of its orthographic image is greater than the first similarity threshold, or the deviation is less than the first deviation threshold, the aerial image is used as a reference image with the GCP as the target GCP.

[0110] After determining the benchmark image of the target GCP, several aerial images can be selected "around" the benchmark image as evaluation images. Here, "around" means that the evaluation images have a camera position and viewing angle close to those of the benchmark image. Since aerial images are generally numbered consecutively in the order of shooting, generally, the evaluation images in the aerial image collection are several adjacent images before and after the benchmark image.

[0111] Fig. 9 A schematic diagram of a reference image and several evaluation images on both sides thereof in a specific embodiment is shown, wherein the reference image is marked with annotation points of the target GCP.

[0112] like Fig. 9 As shown, the evaluation picture may include a picture of the target GCP taken at a perspective close to the orthographic projection, or a picture of the target GCP taken at a perspective more deviated from the reference picture. Similarly, the evaluation picture may also be screened according to the similarity or deviation between the shape of the image formed by the target GCP thereon and the shape of the orthographic projection image of the GCP (or the shape of its image formed on the reference picture), that is, a second similarity threshold or a second deviation threshold may be preset, and only aerial pictures whose similarity with the orthographic projection image of the target GCP (or its image formed on the reference picture) is greater than the second similarity threshold, or whose deviation is less than the second deviation threshold, are selected as evaluation pictures.

[0113] Preferably, the image of the target GCP on the reference image is closer to its positive projection image than the image of the surrounding evaluation image. Therefore, in some preferred embodiments, the first similarity threshold is greater than or equal to the second similarity threshold; in other preferred embodiments, the first deviation threshold is less than or equal to the second deviation threshold.

[0114] In some preferred embodiments, the number of evaluation pictures is not less than 6. For example, 3 aerial pictures can be selected in sequence on both sides of the reference picture as 6 evaluation pictures. In other preferred embodiments, the number of evaluation pictures is not more than 10. For example, 5 aerial pictures can be selected in sequence on both sides of the reference picture as 10 evaluation pictures.

[0115] Step 840 is used to mark the annotation point corresponding to the target GCP on the reference image. The marking in this step can be performed manually, for example, manually selecting a pixel in the image formed by the target GCP on the reference image (preferably the pixel corresponding to the center of the target GCP), and using the pixel as the annotation point of the target GCP on the reference image; or, it can also be performed in a semi-automatic manner of human-computer interaction, for example, first manually defining a rough range, identifying the GCP image therein through an automatic recognition algorithm and marking a pixel therein as a annotation point, and then manually adjusting the position of the annotation point.

[0116] It should be noted that although the operation of marking the target GCP in step 840 occurs after the selection of the reference image in step 830, the preferred principle for selecting the reference image is whether a certain GCP can be imaged in an aerial image in a normal projection or a direction close to normal projection. The purpose is to facilitate accurate marking of the GCP in the aerial image. Therefore, for different target GCPs, different aerial images may need to be selected as their reference images.

[0117] After the target GCP is accurately marked on the reference image, in step 850, Fig.10 As shown, first, the marked points of the target GCP are projected onto the coarse grid model along their projection lines corresponding to the reference image (in the embodiments of the present application, the projection lines corresponding to the marked points on the reference image are referred to as reference projection lines), wherein the reference projection lines of the marked points can be determined by the camera orientation elements of the reference image obtained through steps 823 to 827 and stored in the database.

[0118] Camera orientation elements play an important role in photogrammetry, computer vision, image processing and other fields. They include the camera's intrinsic orientation elements and extrinsic orientation elements. The intrinsic orientation elements describe the camera's internal parameters, which are related to the camera's geometric structure and optical properties, mainly including focal length, principal point, radial distortion coefficient, tangential distortion coefficient, etc., which are used to characterize how to project from a three-dimensional space point to a two-dimensional image plane; the extrinsic orientation elements describe the camera's spatial position and orientation during shooting, mainly including the camera's three-dimensional position coordinates and rotation angles (Roll, Pitch, Yaw or the corresponding rotation matrix / quaternion), which are used to characterize the camera's position and posture relative to the world coordinate system.

[0119] refer to Figure 3Once the camera orientation elements of an aerial image are determined, for a three-dimensional space point, the straight line from the camera center to this space point is the projection line of the space point corresponding to the aerial image. The intersection of this projection line and the aerial image is the position of its imaging on the aerial image. Therefore, the projection line shows how the space point is mapped to the image through the internal and external orientation elements of the camera.

[0120] In order to clearly indicate the direction of the projection operation, in the embodiments of the present application, the operation of projecting one or several elements (such as pixels, geometric features, or texture features) from the aerial image to the coarse mesh model along their corresponding projection lines is called projection; the operation of projecting the elements on the coarse mesh model to the aerial image along their projection lines is called reprojection.

[0121] As analyzed above, the coarse mesh model obtained by unconstrained bundle adjustment roughly reflects the three-dimensional terrain of the specified area, and the camera orientation element provides the direction of projection of each pixel on the reference image to its corresponding spatial point. Therefore, after projecting the marked point of the target GCP onto the surface of the coarse mesh model according to its corresponding reference projection line on the reference image, a projection point will be obtained. The three-dimensional coordinates of the projection point and the orientation of the plane on which it is located on the coarse mesh model (i.e., the normal direction) represent the approximate spatial position and orientation of the target GCP when it is arranged in the specified area.

[0122] Next, continue to intercept a face element around the projection point on the plane where the projection point is located on the coarse mesh model through step 850. In this application, this face element is called the first projection face element. The first projection face element can preferably be intercepted with the projection point as the center, or it can also be adjusted accordingly according to the actual intersection of the various planes constituting the coarse mesh model (for example, if the face element intercepted with the projection point as the center just crosses two planes constituting the coarse mesh model, it is necessary to appropriately offset the first projection face element so that it is in the same plane), and it only needs to ensure that the intercepted face element contains the projection point.

[0123] The shape and size of the first projection surface element can be selected according to the scale of the designated area, the resolution parameters of the aerial image, etc. In some preferred embodiments, the first projection surface element can be in the shape of a square with the projection point as the center of the square. In other embodiments, the shape of the first projection surface element can also be set accordingly according to the actual shape of the target GCP, such as a rectangle, a circle, etc.

[0124] Obviously, the first projection surface element is equivalent to a Mask, which constructs a "window" with limited shape and size (number of pixels) based on the approximate position and orientation information of the target GCP reflected by the projection point on the coarse grid model. This is used as the starting point of the search, and the actual position and orientation of the target GCP are obtained by iterative search in step 860.

[0125] In an embodiment of the present application, the search for the actual position and orientation of the target GCP is performed by the following operations: taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to continuously perform translation or rotation operations along the reference projection line, reprojecting the result of each operation onto each evaluation image to obtain a reprojection window for evaluating the correlation, and calculating the correlation of the images in the reprojection window on each evaluation image. When the calculation result meets the correlation criterion, the position of the first projection surface element at this time (the position can be represented by the three-dimensional coordinates of the intersection of the first projection surface element and the reference projection line) and the normal direction can be used as the estimated results of the actual position and orientation of the target GCP.

[0126] Fig.11 , Fig.12 The operation principle and specific implementation process of step 860 in some preferred embodiments are shown respectively, referring to Fig.11 , Fig.12 In step 861, the first projection plane element is constructed according to its current position and orientation, and the projection lines of each point on it are projected to each evaluation picture according to the camera orientation element information of each evaluation picture read from the database, and reprojected to each evaluation picture along the projection line corresponding to each evaluation picture. After reprojection to the evaluation picture, the area covered on the evaluation picture is as follows Fig.11 The edge of this region, shown in red, constitutes the reprojection window.

[0127] Obviously, during the first reprojection operation, the intersection of the first projection element and the reference projection line is the projection point on the coarse mesh model, and the position and orientation of the first projection element are the position and orientation of the element intercepted in step 850.

[0128] refer to Fig.11 The reprojection window represents the expected position, size and shape (i.e., the position of the reprojection window, the number of pixels covered, and the shape formed) of the target GCP on the corresponding aerial image after it is photographed by the cameras at various positions when the target GCP is in a certain alternative posture. Obviously, only the first projection facet that is consistent with the actual posture of the target GCP, whose texture features of the image in the reprojection window obtained by reprojecting it to each evaluation image, has the greatest correlation with the target GCP.

[0129] It can be seen that since the rough grid model has obtained the approximate pose of the target GCP, using it to determine the initial pose of the first projection surface element can significantly reduce the amount of search calculations, and only a slight translation or rotation around the initial pose is required to accurately estimate the true pose of the target GCP; at the same time, for any candidate pose of the first projection surface element, its position determines the position of the reprojection window on the evaluation image, and its orientation determines the shape and size of the reprojection window on the evaluation image. It can be seen that the texture features of the image in the reprojection window are also restricted and constrained by factors such as position, posture, shape, and area. The reprojection window generated according to any pose that deviates from the actual pose by more than a certain degree will cause the image it covers to deviate significantly from the actual imaging result of the target GCP on the evaluation image in one or more of the position, shape, and area, and further cause a significant decrease in the correlation of the image texture in the reprojection window. Therefore, the above method can greatly improve the accuracy of estimating the actual pose of the target GCP.

[0130] Step 862 is used to evaluate the correlation of images in each reprojection window. An image correlation evaluation index known to those skilled in the art may be selected to calculate the correlation of images in the reprojection windows on any two evaluation pictures. For example, an optional correlation evaluation index is the Normalized Cross-Correlation (NCC). Alternatively, other indicators for image correlation may be selected.

[0131] In some preferred embodiments, the reference picture is also used as the evaluation picture, that is, the first projection bin is also reprojected to the reference picture, and the image in its reprojection window is used to participate in the evaluation of the correlation.

[0132] After completing the correlation calculation of the images in each reprojection window on each evaluation picture, it is determined whether the calculation result meets the correlation criterion, and step 863 or step 864 is executed according to the determination result.

[0133] In some optional embodiments, the correlation of the images in each reprojection window may be firstly counted to obtain any one or more of the statistical characteristics such as the mean, median, maximum or minimum, and then determine whether it is greater than or equal to a preset correlation threshold. When the determination result is no, the first projection plane element is translated or rotated in step 863 (refer to Fig.12, the translation or rotation of the first projection surface element is always performed with the intersection of the first projection surface element and the reference projection line as the movement base point, that is, the projection result of the annotation point of the target GCP to its spatial point is always "bound" to the reference projection line), and then return to step 861 to re-project and recalculate the correlation. When the judgment result is yes, execute step 864, and use the posture of the first projection surface element at this time as its optimal position and optimal orientation (that is, use it as the actual position and orientation of the target GCP obtained by the search), and store the search results in the database.

[0134] Fig.13 The figure shows how the NCC mean of the image in each reprojection window changes with the size of the first projection bin in a specific embodiment. In the figure, the first projection bins are all square bins, and the horizontal axis represents the number of pixels contained in each side of the bin.

[0135] like Fig.13 As shown, in the area where the first projection pixel size is smaller, the NCC value generally shows an upward trend as the pixel size increases, indicating that the matching accuracy is improved as the pixel size increases. However, as the size further increases, the NCC value no longer increases significantly, and even fluctuates (such as the fluctuation between 11×11 pixels and 17×17 pixels).

[0136] By analyzing the image texture features on each evaluation image, it can be seen that this correlation index no longer increases significantly after the pixel size increases to a certain extent, and even decreases. This indicates that as the number of pixels used for correlation calculation increases, other image noise may be introduced, resulting in a decrease in the contribution of the pixel size increase to the pose optimization, or even a negative impact. Therefore, the size of the first projection pixel cannot be too small, otherwise there will be a lack of sufficient information for evaluating the cross-correlation in the subsequent search process. Preferably, the lower limit of the size of the first projection pixel is 3×3 pixels; at the same time, the size of the first projection pixel does not need to be too large, because further increasing the size will not continue to improve the cross-correlation. In some preferred embodiments, the upper limit of the size of the first projection pixel can be determined by calculating the maximum value of the correlation statistics of the images in each reprojection window.

[0137] Sub-process III (automatically marking target GCPs on the image to be marked):

[0138] After searching and estimating the actual pose of the target GCP in the specified area through sub-process II, it can be used as a geometric constraint to search and mark the imaging position on the image to be marked. Obviously, the use of this constraint can effectively limit the search for the target GCP on the image to be marked to a specific area, thereby avoiding the problem of identifying different GCP images in two aerial images as the same GCP.

[0139] In some preferred embodiments, Figure 7 As shown, first, a reference window is created on the reference picture through step 870. Specifically, the reference window is determined based on the image formed by the target GCP on it. For example, the image edge corresponding to the target GCP can be generated on the reference picture by manual operation or manual plus automatic algorithm interaction, so as to obtain the reference window; or, on the basis of completely surrounding the image corresponding to the target GCP, a few pixels can be appropriately extended from its edge to obtain a larger reference window surrounding the image corresponding to the target GCP.

[0140] Next, in step 880, the image in the reference window is projected onto the first projection plane element along the reference projection line to obtain the second projection plane element; then in step 890, the second projection plane element is reprojected onto the image to be marked along the projection line corresponding to the image to be marked, and the desired image on the image to be marked can be obtained; after the desired image is obtained, the edge of the desired image is extracted in step 900 to construct a matching window; finally, in step 910, the best position of the matching window is searched on the image to be marked, so as to mark the imaging position corresponding to the target GCP on the image to be marked. In the embodiment of the present application, Figure 7 As shown, after the target GCP is marked on a picture to be marked, the picture to be marked in which the target GCP is captured can be replaced and the above operation can be repeated.

[0141] The search for the optimal position of the matching window in step 910 can be implemented using a least squares image matching algorithm. Fig.14 In some preferred embodiments, the implementation principle of step 910 is shown, such as Fig.14 As shown, the matching window can be slid translationally on the image to be marked according to the set sliding step size, and the image in the matching window can be extracted, and the minimum error square calculation is performed on its texture features and the texture features of the expected image. The above process is repeated until the position with the smallest error is found. The image surrounded by the matching window at this position represents the position where the target GCP is imaged, thereby completing the automatic marking of the target GCP.

[0142] Obviously, the position and orientation of the second projection surface element are consistent with the first projection surface element with the optimal position and orientation. The difference between the two is that the second projection surface element adds the shape and texture feature information expected when the corresponding image of the target GCP on the reference image is projected to the real posture of the target GCP. Therefore, when the multiple information with this specific position, orientation, shape and texture features is further reprojected onto the image to be marked, the obtained expected image can roughly limit the expected imaging position of the target GCP on the image to be marked and the expected shape, size and texture of the image. The target GCP is searched using this information with strong constraints, which eliminates the possibility of identifying different GCPs in two aerial images as the same one. At the same time, it only needs to translate and slide the matching window in a smaller area around the expected image, without the need for large-scale search and rotation operations in conventional image matching retrieval, which undoubtedly greatly improves the search speed.

[0143] Fig.14 In the illustrated embodiment, since the desired image is completely displayed on the picture to be marked, the matching window can be constructed only through the edge of the desired image. Fig.15 The illustrated embodiment further demonstrates the situation of searching for a target GCP on an aerial image that partially displays the target GCP. When the target GCP is located at the edge of an aerial image, only a portion of it may be imaged on the aerial image. For conventional automatic labeling methods based solely on texture information, it is obvious that the texture features used for search and comparison are missing, which makes recognition difficult. However, for the method of the present application, since the desired image is imaged at the edge of the aerial image according to its expected position, and only a portion of it is located on the aerial image, the edge of the desired image and the edge of the aerial image itself can be used to jointly construct a matching window, and its optimal position can be searched on the image to be labeled. That is, through the method of the present application, it is possible to complete its identification and labeling only through partial information of the target GCP on the image to be labeled. Specific embodiments

[0145] This embodiment adopts the above-mentioned automatic labeling method of target GCP to automatically label the target GCP of the aerial picture collection (Dort-ZECHE data set) with a certain building as the designated area published by the International Society for Photogrammetry and Remote Sensing (ISPRS).

[0146] Fig.16 shows some aerial pictures obtained by aerial photography of the specified area, such as Fig.16 As shown, different aerial pictures capture different GCPs in number, position and form. The target GCP automatic labeling method provided by the present application is used to automatically label the target GCP of the aerial picture set, wherein the size of the first projection plane is 11×11 pixels. Fig.17 , Fig.18 Schematic diagrams of the three-dimensional sparse point cloud and the coarse grid model generated during the implementation process are shown respectively.

[0147] In order to evaluate the effectiveness of the method provided in this application, 10 GCPs set on the buildings were used as target GCPs, and the target GCPs were automatically marked on the aerial images in the data set. The following Table 1 summarizes the automatic marking results.

[0148] Table 1 Statistics of target GCP automatic labeling results

[0149]

[0150] It can be seen from Table 1 that when the reference window size is 15×15, a total of 482 points are successfully marked, with a success rate of about 67.3%, and the root mean square error between the coordinates of these 482 marked points and the true coordinates is 0.564 pixels; when the window size is 25×25, a total of 683 points are successfully marked, with a success rate of about 95.5%, and the root mean square error between the coordinates of these 683 marked points and their true coordinates is 0.549 pixels. The statistical results show that the target GCP automatic marking method used in this application can automatically mark more than 700 GCPs to be marked within 1 minute, and as the reference window size increases to 25×25 pixels, its marking success rate exceeds 95%. At the same time, the comprehensive accuracy of marking each target GCP reaches the sub-pixel level, proving that it is a comprehensive improvement in marking speed and marking accuracy compared to manual marking methods.

[0151] The above is a detailed introduction to the specific implementation methods of the present application. For those skilled in the art, several improvements and modifications may be made to the present application without departing from the principles of the present application. These improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A method for automatically marking a target GCP, characterized in that: The following operations are included: a. Establish a coarse grid model based on a set of aerial photos of a specified area, where the target GCP is located in the specified area; b. determining a marking point of a target GCP on a reference picture, where the marking point is located in an image of the target GCP on the reference picture; c. Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection facet; d. searching and determining the best position and the best orientation of the first projection surface element based on the correlation of the images in the reprojection window obtained by reprojecting the first projection surface element onto a plurality of evaluation images; e. Projecting an image in a reference window on the reference picture to a first projection plane element having an optimal position and an optimal orientation to obtain a second projection plane element, wherein the image of the target GCP on the reference picture is located in the reference window; f. Based on the desired image obtained by reprojecting the second projection surface element onto the image to be labeled, search and determine the imaging position corresponding to the target GCP on the image to be labeled.

2. The automatic marking method of target GCP according to claim 1, characterized in that: The coarse grid model is generated by performing an unconstrained bundle adjustment operation on a set of aerial pictures of a specified area, wherein the coarse grid model uses each point in a sparse point cloud obtained by performing an unconstrained bundle adjustment on a set of aerial pictures of a specified area as a grid point.

3. The automatic marking method of target GCP according to claim 1, characterized in that: The similarity between the image of the target GCP on the reference image and its orthographic image is greater than a preset first similarity threshold, or the deviation is less than a preset first deviation threshold; The similarity between the image formed by the target GCP on the evaluation picture and its orthographic image or its image formed on the reference picture is greater than a preset second similarity threshold, or the deviation is less than a preset second deviation threshold, wherein the first similarity threshold is greater than or equal to the second similarity threshold, and the first deviation threshold is less than or equal to the second deviation threshold.

4. The automatic marking method of target GCP according to claim 1, characterized in that: Projecting the marked point onto the surface of the coarse mesh model and intercepting the first projection surface element in operation c includes: The marked point is projected onto the surface of the coarse mesh model along a reference projection line corresponding to the reference picture to obtain its projection point on the surface of the coarse mesh, wherein the reference projection line corresponding to the reference picture of the marked point is determined based on the camera orientation element of the reference picture.

5. The automatic marking method of target GCP according to claim 4, characterized in that: In operation c, the first projection surface element is determined by intercepting a region surrounding the projection point on the surface of the coarse mesh model.

6. The automatic labeling method of target GCP according to claim 1, characterized in that: In operation d, the first projection surface element is reprojected to a plurality of evaluation images, including: Each point on the first projection surface element is reprojected to each evaluation picture along its projection line corresponding to each evaluation picture, and the edge of each point reprojected to each evaluation picture is used as its reprojection window on each evaluation picture, wherein the projection line corresponding to each evaluation picture of each point on the first projection surface element is determined based on the camera orientation element of each evaluation picture.

7. The automatic marking method of target GCP according to claim 6, characterized in that: The searching and determining of the best position and the best orientation of the first projection plane element in operation d includes: Taking the intersection of the first projection surface element and the reference projection line as the movement base point, driving the first projection surface element to perform a translation operation or a rotation operation along the reference projection line; Reproject the result of each translation operation or rotation operation onto each evaluation image to obtain a reprojection window, and calculate the correlation of the image of each evaluation image in its reprojection window; When the calculation result satisfies the correlation criterion, the position and orientation of the first projection plane element at this time are taken as the optimal position and optimal orientation of the first projection plane element, otherwise the translation operation or rotation operation of the first projection plane element is performed again.

8. The automatic marking method of target GCP according to claim 7, characterized in that: The correlation criterion is: the correlation statistics of the images of each evaluation picture in its reprojection window meet the preset correlation threshold; The lower limit of the area of ​​the first projection surface element is 3×3 pixels, and the upper limit of the area is determined based on the correlation statistics results of the images in the reprojection window of each evaluation picture.

9. The automatic marking method of target GCP according to claim 1, characterized in that: The expected image is an image obtained by reprojecting each point on the second projection plane element onto each picture to be marked along its projection line corresponding to each picture to be marked; as well as, The search in operation f to determine the imaging position corresponding to the target GCP on the image to be marked includes: Extracting the edge of the expected image to construct a matching window, wherein the matching window is constructed only by the edge of the expected image, or is jointly constructed by the edge of the expected image and the edge of the picture to be marked where the expected image is located; The best position of the matching window is searched on the image to be marked, so as to mark the imaging position corresponding to the target GCP on the image to be marked.

10. The automatic marking method of target GCP according to claim 9, characterized in that: In the process of searching for the best position of the matching window on the image to be marked, the matching window is only translated on the image to be marked.

Images