Ground control point generating device, image processing system, ground control point generating method, image processing method, ground control point generating program, and image processing program

Abstract

To provide a ground control point with high accuracy in a height direction over a wide range.SOLUTION: A ground control point generation device 100 generates ground control point data using a reference image and terrain data. An object detection unit 10 outputs an object class and pixel coordinates for an object installed at a height approximately equal to the average elevation of the surrounding land. A small area image extraction unit 20 extracts a small area image in the reference image from the pixel coordinates of the object. An on-map coordinate acquisition unit 30 acquires on-map coordinates of the object using a correspondence relationship between the reference image and the on-map coordinates. A height acquisition unit 40 acquires the height of the object by referring to the terrain data corresponding to the pixel coordinates of the object. A data registration unit 50 registers the ground control point data in which the small area image is associated with three-dimensional coordinates composed of the on-map coordinates and the height of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a ground control point generation device, an image processing device, a ground control point generation method, an image processing method, a ground control point generation program, and an image processing program. In particular, the present disclosure relates to a ground control point generation device that generates ground control points used to correct position and orientation information of an image captured from the sky by an aircraft, and an image processing device that corrects the position and orientation information of an image using ground control points. [Background technology]

[0002] Images of the ground taken from above by aerial vehicles such as satellites, aircraft, or drones can be accurately associated with locations on maps and used for various purposes, such as monitoring ground events linked to location information, overlaying images taken from different platforms, and integrating them with other geospatial data.

[0003] Correspondence between such images taken from the sky and locations on a map is generally assigned on the system side through systematic correction using the platform's position and orientation data, a geometric model of the Earth, and the geometric characteristics data of the sensor.

[0004] RPC is a general-purpose format for position and orientation information assigned by the system. RPC is an abbreviation for Rational Polynomial Coefficients. RPC is a set of coefficients for a mathematical formula that associates a location on Earth (latitude, longitude, and altitude) with a pixel location in an image. Such a formula is a rational polynomial. RPC does not directly use platform-specific parameters such as the geometric characteristic data of the sensor or the position and orientation at the time of observation. RPC enables the association of an image with a location on a map by using the coefficients obtained by applying information such as the geometric characteristic data of the sensor or platform-specific parameters such as the position and orientation at the time of observation to a specified mathematical formula.

[0005] However, due to various error factors, such as insufficient accuracy of the platform's position and orientation data, errors are included in the position and orientation information provided by the system. These errors cause discrepancies between the position shown on the image and the actual position shown on the map. In the case of satellite images, for example, such discrepancies can be on the order of several meters or more.

[0006] Therefore, ground control points in the real space are used to correct or modify the position and orientation information of an image to be corrected, such as an aerial photograph or a satellite image.

[0007] The technology in Patent Document 1 uses, as ground control points, objects whose positions and shapes can be accurately determined both in real space and on the image to be corrected, such as the center of a road intersection or the corner of a building, and discloses a technology that uses this to correct the image to be corrected with high precision.

[0008] The conventional method of generating ground control points involves acquiring survey data using techniques such as GNSS surveying, and generating ground control point data that stores coordinates on a map along with information that identifies the location where the data was acquired, such as on-site photographs. GNSS is an abbreviation for Global Navigation Satellite System. In addition, conventional image processing methods identify the positions of ground control points on an image, correct the position and orientation information based on the correspondence between the coordinates of the ground control points on a map and the positions on the image, and then use the results to perform map projection or orthorectification to correct the image to be corrected. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Patent No. 4231279 Summary of the Invention [Problem to be solved by the invention]

[0010] However, while GNSS surveying can obtain accurate height information for ground control points, it is not available over a wide area, making it impossible to obtain accurate height information for any point on the Earth's surface. On the other hand, there are terrain data such as digital elevation models (DEM) and digital surface models (DSM) that provide height information over a wide area. DEM is an abbreviation for Digital Elevation Model. DSM is an abbreviation for Digital Surface Model. These types of terrain data will be referred to as DEM and DSM terrain data below. In topographic data such as DEM and DSM, the mesh for height measurement intervals is coarse, so there is a discrepancy between the height of the points selected as ground control points and the height represented by topographic data such as DEM and DSM. Therefore, it is not possible to obtain highly accurate ground control points. As a result, there was an issue that ground control points whose height information is extracted from topographic data such as DEM and DSM cannot be used to accurately correct the position information of the image to be corrected.

[0011] The present disclosure aims to obtain ground control points with high accuracy in the vertical direction over a wide range where reference images and topographical data exist. [Means for solving the problem]

[0012] A ground control point generation device according to the present disclosure generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, the device comprising: an object detection unit that outputs an object class representing the type of object and pixel coordinates of the object for an object installed at a height approximately equal to the average elevation of the surrounding land; a small-area image extraction unit that extracts an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition unit that acquires map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition unit that acquires a height of the object by referring to the topographical data corresponding to pixel coordinates of the object; a data registration unit that registers data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; Equipped with. [Effects of the Invention]

[0013] According to the ground control point generating device of the present disclosure, ground control points with high accuracy in the height direction can be obtained over a wide range where reference images and topographical data exist. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a ground control point generating device according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the configuration of a ground control point database according to the first embodiment. [Figure 3] FIG. 2 is a diagram showing a specific example of the configuration of an object detection unit according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing a specific example of the configuration of an object detection unit having a learning function according to the first embodiment. [Figure 5] 3 is a flowchart showing the operation of the ground control point generating device according to the first embodiment. [Figure 6] 4 is a flowchart showing details of the operation of the object detection unit according to the first embodiment. [Figure 7] 4 is a flowchart showing the operation of a learning device used in an object detection unit having a learning function according to the first embodiment. [Figure 8] 4 is a flowchart showing details of the operation of the object detection unit having a learning function according to the first embodiment. [Figure 9] FIG. 1 is a diagram showing an example of the hardware configuration of a ground control point generating device according to a first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of the hardware configuration of a ground control point generation device according to a modified example of the first embodiment. [Figure 11] FIG. 10 is a diagram showing an example of the configuration of an image processing device according to a second embodiment. [Figure 12] FIG. 10 is a flowchart showing the operation of the image processing device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] The present embodiment will be described below with reference to the drawings. In each drawing, identical or corresponding parts are designated by the same reference numerals. In the description of the embodiment, the description of identical or corresponding parts will be omitted or simplified as appropriate. Arrows in the drawings mainly indicate the flow of data or the flow of processing. Furthermore, the sized relationships between components in the following drawings may differ from the actual relationships. Furthermore, in the description of the embodiment, directions or positions such as up, down, left, right, front, rear, front and back may be indicated. These notations are used for convenience of explanation and do not limit the placement, direction or orientation of devices, instruments, parts, etc.

[0016] Embodiment 1 ***Configuration Description*** FIG. 1 is a diagram showing an example of the configuration of a ground control point generating device 100 according to this embodiment. The ground control point generation device 100 generates ground control points used to correct the position and orientation information of an image captured from the sky by an air vehicle such as an artificial satellite, an aircraft, or a drone. An image to be corrected using the ground control points is set as a correction target image.

[0017] The input data to the ground control point generation device 100 are a reference image REF, topographical data DEM, and attribute information MAP. The output data of the ground control point generation device 100 is ground control point data GCP. The ground control point generating device 100 comprises an object detecting unit 10, a small area image extracting unit 20, a map coordinate acquiring unit 30, an elevation acquiring unit 40, and a data registering unit 50. The ground control point generation device 100 is also connected to a ground control point database 200, and registers the ground control point data GCP, which is output data, in the ground control point database 200.

[0018] First, the input data to the ground control point generation device 100 will be described. The reference image REF is an image that includes at least the same field of view as the image to be corrected and is associated with map coordinates. An example of a reference image REF is an image that has been geometrically corrected with high precision using ground control points obtained by GSNN surveying of a previously acquired aerial photograph or satellite image. Alternatively, orthorectified aerial photographs or satellite images published by the Geospatial Information Authority of Japan or other organizations may be used. These reference images REF are provided with meta-information that associates pixel coordinates, such as pixel or line coordinates in the image, with map coordinates. The reference images REF can be used to obtain the horizontal position of an object on the Earth's surface from the position of the object in the image using the meta-information.

[0019] DEM topographic data is data such as DEM or DSM. DEM is data that shows the height of the ground excluding buildings and trees. DSM is data that shows the height of the Earth's surface including buildings and trees. These topographic data are available in high resolution, acquired by aerial laser surveying and drones. However, the areas where this topographic data is available are limited. Meanwhile, topographical data covering all of Japan includes 5m and 10m mesh DEMs published by the Geospatial Information Authority of Japan. Also, topographical data covering the entire globe includes the 30m mesh ASTER GDEM DEM published jointly by the Ministry of Economy, Trade and Industry and NASA. Furthermore, topographical data covering the entire globe includes the 30m mesh DSM of the Global Digital Elevation Data (AW3D30) published by JAXA.

[0020] This embodiment assumes the use of such coarse topographic data with a mesh size of approximately 5m to 30m, which has been compiled at the national and global levels. In particular, the Geospatial Information Authority of Japan's 5m mesh DEM is created based on data acquired by airborne laser surveying. Therefore, the Geospatial Information Authority of Japan's 5m mesh DEM has high accuracy in the vertical direction, allowing accurate height information to be obtained over a wide range.

[0021] Next, the output data from the ground control point generation device 100 will be described. FIG. 2 is a diagram showing an example of the configuration of the ground control point database 200 according to this embodiment.

[0022] The ground control point database 200 is a sequence of ground control point data GCPs, each of which is a pair of a feature that serves as a ground control point, a small area image that is an image of the surrounding area, and the three-dimensional coordinates of the ground control point. The small area image may be a feature extracted from the image. The ground control point generating device 100 outputs the ground control point data GCPs that make up the ground control point database 200.

[0023] Next, each component of the ground control point generating device 100 will be described with reference to FIG. The ground control point generating device 100 comprises, as functional elements, an object detecting unit 10, a small area image extracting unit 20, a map coordinate acquiring unit 30, an altitude acquiring unit 40, and a data registering unit 50.

[0024] The object detection unit 10 identifies and locates objects based on local image features of the reference image REF, and outputs the object class L and pixel coordinates (i, j) of the location where the desired object is captured.

[0025] Here, the desired object is an object that can be easily positioned as a ground control point in an image taken from the sky and is installed at a height approximately equal to the average elevation of the surrounding land. The desired object is, for example, an intersection of white lines painted on a road or a diamond mark. The object detection unit 10 extracts such an object from the reference image REF using a method such as an object detection algorithm. The object detection unit 10 then outputs an object class L representing the type of object and the pixel coordinates (i, j) of a representative point of the object. The object class L is, for example, the type of object, such as an intersection of white lines or a diamond mark.

[0026] FIG. 3 is a diagram showing a specific example of the configuration of the object detection unit 10 according to this embodiment. The object detection unit 10 comprises a feature extraction unit 101 , a discrimination unit 102 , a position identification unit 103 , and a selection unit 104 .

[0027] The feature extraction unit 101 extracts local image features from the reference image REF. For example, the image features can be extracted using techniques such as HOG and SIFT, which are used in processes such as image recognition and object detection. HOG is an abbreviation for Histograms of Oriented Gradients. SIFT is an abbreviation for Scale-Invariant Feature Transform. The feature extraction unit 101 extracts a predetermined number N (N is a natural number) of values ​​related to image features, such as the direction or fineness of the brightness gradient, from each local window in the image as image features, to form an N-dimensional vector. N is the number of dimensions of the image feature. The feature extraction unit 101 calculates the image feature for each position in the reference image REF by sliding the local window within the image. The feature extraction unit 101 supplies data linking the image feature and its position to the classification unit 102 and the position identification unit 103. Note that the image feature is not limited to the above-mentioned HOG or SIFT, and a filter individually designed to be suitable for detecting the desired object may also be used. Furthermore, a feature extractor acquired by learning, such as a neural network, may also be used, as will be described later.

[0028] The classification unit 102 classifies at least one object class having a predetermined image feature based on the image feature amount. Specifically, the image feature extracted by the feature extraction unit 101 is compared with feature predetermined for each object class to be detected. The classification unit 102 classifies the image feature based on an index such as the similarity between the image feature and the feature predetermined for each object class. The similarity is, for example, the distance between feature vectors. The identification unit 102 then supplies the label L of the classification result to the selection unit 104 as the object class L. The label L may also include information that the object to be detected does not fall into any of the classes. Classification may be performed using a threshold value individually designed to identify the desired object. Alternatively, a classifier acquired by learning, such as a support vector machine or a neural network, may be used, as will be described later.

[0029] The position identification unit 103 detects the position of a representative point of an image region having the identified predetermined image feature. The position identification unit 103 receives information about the local window position at which the image feature identified as belonging to the predetermined object class was calculated from the identification unit 102. This local window position indicates the position of a rectangle containing the object to be identified in the reference image REF. The position identification unit 103 identifies the position of the representative point from the position of the rectangle containing the object to be identified and supplies this to the selection unit 104 as the pixel coordinates (i, j) of the object. For example, when detecting an intersection of white lines on a road as an object, the position of the representative point is determined by a technique such as fitting the coordinates of the intersection of the white lines from the image within the rectangle, and this is used as the representative point. Alternatively, the coordinates of the upper left corner or center of the rectangle may be used as the representative point without detecting the coordinates of the representative point.

[0030] The selection unit 104 outputs the object to be detected and the pixel coordinates (i, j) of the object based on the object class L identified by the identification unit 102 and the pixel coordinates (i, j) of the object identified by the position identification unit 103.

[0031] The object detection unit 10 may further include a mask image creation unit 105. The mask image creation unit 105 generates a mask image based on attribute information MAP indicating attributes of features corresponding to coordinates on a map and topographical data DEM. The attribute information MAP can be, for example, a land cover classification map. A land cover classification map is raster data labeled with information on land cover on the earth's surface, such as buildings, roads, vegetation, and water bodies. The mask image creation unit 105 creates a mask image from the land cover classification map by validating only the pixel coordinates of attributes containing the target object. For example, if the target object is an intersection of white lines on a road, the mask image creation unit 105 creates a mask image by validating only the pixel coordinates of the road in the land cover classification map. Another example of the attribute information MAP may be vector data of buildings, roads, water bodies, etc. The mask image creation unit 105 performs area expansion on this vector data through buffer processing, then converts it into raster data, which is used to create a mask image in the same manner as described above. Furthermore, the mask image creation unit 105 may create a mask image by extracting areas below a certain elevation or areas with small spatial changes in elevation from the topographical data DEM.

[0032] Based on the mask image created by the mask image creation unit 105, the selection unit 104 compares the class of the object corresponding to the pixel coordinates (i, j) of the object with the attributes shown in the mask image to select the object to be registered in the ground control point data.

[0033] FIG. 4 is a diagram showing a specific example of the configuration of the object detection unit 11 having a learning function according to this embodiment. The object detection unit may be configured entirely using a learning function such as a neural network. Fig. 4 shows an object detection unit 11 configured using a learning function such as a neural network.

[0034] The learning device 300 for the object detection unit 11 includes a data acquisition unit 121, a model generation unit 122, and a learned model storage unit 13.

[0035] The data acquisition unit 121 acquires a reference image REF', an object class, and pixel coordinates (correct answers) as learning data.

[0036] The reference image REF' is an image taken from the sky, and has approximately the same image quality as the reference image REF input to the object detection unit 11. Equivalent image quality means, for example, that the resolution, sharpness, noise level, and image processing conditions are the same. However, it does not necessarily have to include the same field of view as the reference image REF, nor does it necessarily have to be associated with map coordinates.

[0037] The object class and pixel coordinates (correct answer) are data annotated with the class label of the object to be detected and the pixel coordinates of that object in the reference image REF'. The pixel coordinates may be the coordinates of a rectangle containing the object to be detected, or the coordinates of a representative point of the object to be detected. The object to be detected may be, for example, an intersection of white lines painted on a road, or a diamond mark. The rectangle is annotated to surround these objects. Furthermore, the representative point is annotated to specify the intersection of the white lines, or the center or node of the diamond mark.

[0038] The model generation unit 122 learns the object class and pixel coordinates based on learning data created based on a combination of the reference image REF' output from the data acquisition unit 121 and the object class and pixel coordinates (correct answer). That is, the model generation unit 122 generates a learned model that infers the optimal object class and pixel coordinates from the reference image REF' of the object detection unit 10 and the object class and pixel coordinates (correct answer). Here, the learning data is data that associates the reference image REF' with the object class and pixel coordinates (correct answer).

[0039] The model generation unit 122 learns the object class and pixel coordinates, for example, by so-called supervised learning according to a neural network model. Here, supervised learning refers to a technique in which pairs of input and result data are provided to the learning device 300, and the learning device 300 learns the features of the learning data and infers the result from the input.

[0040] A neural network consists of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer, or two or more layers.

[0041] In this embodiment, the neural network learns a model that infers the object class and pixel coordinates by so-called supervised learning in accordance with learning data created based on a combination of a reference image REF', an object class, and pixel coordinates (correct answer). The reference image REF', the object class, and pixel coordinates (correct answer) are output from the data acquisition unit 121.

[0042] That is, the neural network learns by inputting a reference image REF' into the input layer and adjusting the weights so that the result output from the output layer approaches the object class and pixel coordinates (correct answer).

[0043] The model generation unit 122 generates and outputs a trained model by executing the above-described learning.

[0044] The trained model storage unit 13 stores the trained model output from the model generation unit 122.

[0045] Next, the configuration of the object detection unit 11, which corresponds to the inference device, will be described. The configuration of the object detection unit 11 differs from that of the object detection unit 10 in that the object detection unit 11 includes a data acquisition unit 111 and an inference unit 112. The selection unit 104 and the mask image creation unit 105 have the same configuration as those of the object detection unit 10, and therefore will not be described below.

[0046] The data acquisition unit 111 acquires a reference image REF.

[0047] The inference unit 112 infers the object class and pixel coordinates obtained using the trained model. That is, by inputting the reference image REF acquired by the data acquisition unit 111 to this trained model, the object class and pixel coordinates inferred from the reference image REF can be output.

[0048] In FIG. 4, the configurations of the learning device 300 and the object detection unit 11 are shown in parallel, but the learning device 300 may be a device separate from the ground control point generation device 100.

[0049] The neural network used in the object detection unit 11 may be a neural network that performs object detection, such as YOLO or SSD. YOLO is an abbreviation for You Only Look Once. SSD is an abbreviation for Single Shot MultiBox Detector. However, the present embodiment is not limited to these methods.

[0050] Returning to the description of the configuration of the ground control point generating device 100 in FIG.

[0051] The small area image extraction unit 20 extracts an image of a small area of ​​the reference image REF containing the object from the pixel coordinates (i, j) of the object output by the object detection unit 10, or image features of the small area, as a small area image IMG. If the small area image IMG is simply image data, the small area image extraction unit 20 extracts an area of ​​P x P pixels surrounding the pixel coordinates (i, j) of the object. If the small area image IMG is image features of the small area, the small area image extraction unit 20 calculates the image features using data of Q x Q pixels surrounding the pixel coordinates (i, j) of the object. For example, the above-mentioned SIFT can be used as the image features. Alternatively, feature extraction obtained by a learning function such as a neural network may be used.

[0052] The map coordinate acquisition unit 30 acquires the map coordinates (X, Y) of the object from the pixel coordinates (i, j) of the object using the correspondence between the reference image REF and the map coordinates. Specifically, the map coordinate acquisition unit 30 converts pixel coordinates into map coordinates using meta-information assigned to the reference image REF for associating pixel coordinates (pixel and line coordinates) with map coordinates. Map coordinate systems include latitude-longitude coordinate systems, UTM coordinate systems, and pseudo-Mercator coordinate systems. Meanwhile, the coordinate system included in the meta-information of the reference image REF is determined according to the projection coordinate system of the reference image REF used as input. The map coordinate acquisition unit 30 first converts pixel coordinates (i, j) into map coordinates in the projection coordinate system of the reference image REF, and then converts them into the coordinate system of the map coordinates of the ground control point data GCP to be output. For example, a latitude and longitude coordinate system is used as the coordinate system of the map coordinates of the ground control point data GCP.

[0053] The height acquisition unit 40 acquires the height Z of an object by referencing the topographical data DEM corresponding to the pixel coordinates (i, j) of the object. The topographical data DEM, like the reference image REF, includes meta-information that associates pixel coordinates in the topographical data DEM with map coordinates. The height acquisition unit 40 uses the meta-information of the reference image REF to obtain map coordinates from pixel coordinates (i, j) in the reference image REF, and then calculates pixel coordinates in the topographical data DEM from the map coordinates. The height acquisition unit 40 then acquires pixel values ​​of the topographical data DEM at the position of the pixel coordinates in the corresponding topographical data DEM, thereby acquiring the height Z corresponding to the pixel coordinates (i, j) of the object.

[0054] The data registration unit 50 registers data in the ground control point database 200, in which the small area image IMG is linked to three-dimensional coordinates (X, Y, Z) consisting of the coordinates X, Y on the map and the height Z of the object, as ground control point data GCP.

[0055] The ground control point generation device 100 generates a ground control point database 200 consisting of one or more ground control point data GCPs using the configuration described above.

[0056] Next, a description will be given of the operation of the ground control point generation device 100. Fig. 5 is a flowchart showing the operation of the ground control point generation device 100 of the first embodiment.

[0057] ***Explanation of Operation*** Next, the operation of the ground control point generation device 100 according to this embodiment will be described. The operation procedure of the ground control point generation device 100 corresponds to a ground control point generation method. Furthermore, the program that realizes the operation of the ground control point generation device 100 corresponds to a ground control point generation program.

[0058] FIG. 5 is a flow diagram showing the operation of the ground control point generation device 100 according to this embodiment. In step S110, the ground control point generation device 100 acquires the reference image REF, the topographical data DEM, and the attribute information MAP. In step S120, the object detection unit 10 detects objects from the reference image REF. The object detection unit 10 outputs pixel coordinates (i, j) of one or more detected objects. The processing of step S120 will be described in detail later. The following processing steps S130 to S160 are repeated for each of the pixel coordinates of all detected objects.

[0059] In step S130, the small area image extracting unit 20 extracts a small area image for the pixel coordinates (i, j) of the object. In step S140, the map coordinate acquisition unit 30 acquires the map coordinates (X, Y) of the object from the pixel coordinates (i, j) of the object using the correspondence between the reference image REF and the map coordinates. In step S150, the height acquisition unit 40 acquires the height Z of the object by referring to the topographical data DEM corresponding to the pixel coordinates (i, j) of the object. In step S160, the data registration unit 50 registers data linking the small area image IMG with the three-dimensional coordinates (X, Y, Z) consisting of the map coordinates X, Y and height Z of the object as ground control point data GCP in the ground control point database 200.

[0060] In this way, the ground control point generation device 100 generates the ground control point database 200 by repeating the processes of steps S130 to S160 for the pixel coordinates of all detected objects.

[0061] FIG. 6 is a flowchart showing details of the operation of the object detection unit 10 according to this embodiment. The operation of the object detection unit 10 will be described in detail with reference to FIG.

[0062] In step S101, the feature extraction unit 101 of the object detection unit 10 extracts features from the reference image REF, and extracts local image feature amounts of the reference image REF. In step S102, the classification unit 102 of the object detection unit 10 performs classification based on the extracted image features and outputs the object class. Furthermore, the position identification unit 103 of the object detection unit 10 performs position identification processing based on the position of the image features and outputs the pixel coordinates (i, j) of the object. In this way, the object class L of the object and the pixel coordinates (i, j) of the object are output.

[0063] In step S103, the mask image creation unit 105 of the object detection unit 10 creates a mask image using the attribute information MAP and the topographical data DEM. In step S104, the selection unit 104 of the object detection unit 10 compares the object class and pixel coordinates of the object with the mask image, selects an object, and outputs the pixel coordinates (i, j) of the selected object.

[0064] FIG. 7 is a flow diagram showing the operation of the learning device 300 used in the object detection unit 11 having a learning function according to this embodiment. FIG. 8 is a flowchart showing the details of the operation of the object detection unit 11 having a learning function according to this embodiment. 7 and 8, the operation of the object detection unit 11 configured using a learning function such as the neural network described with reference to FIG. 4 will be described.

[0065] First, the operation of the learning device 300 will be described with reference to FIG. In step S201, the data acquisition unit 121 of the learning device 300 acquires the reference image REF', the object class, and the pixel coordinates (correct answer). In FIG. 4, the reference image REF', the object class, and the pixel coordinates (correct answer) are acquired simultaneously. However, it is sufficient if the reference image REF', the object class, and the pixel coordinates (correct answer) are input in association with each other. The data for the reference image REF' and the object class and the pixel coordinates (correct answer) may be acquired at different times.

[0066] In step S202, the model generation unit 122 of the learning device 300 generates a trained model according to training data created based on a combination of the reference image REF′ and the object class and pixel coordinates (correct answer). The model generation unit 122 of the learning device 300 trains a model that infers the object class and pixel coordinates by so-called supervised learning according to the training data, and generates the trained model.

[0067] In step S203, the trained model storage unit 13 stores the trained model generated by the model generation unit 122. Note that the learning device 300 may generate the trained model at a timing separate from the operation of the ground control point generation device 100.

[0068] Next, the process by which the object detection unit 11 outputs the object class and pixel coordinates will be described with reference to FIG.

[0069] In step S301, the data acquisition unit 111 of the object detection unit 11 acquires a reference image REF. In step S302, the inference unit 112 of the object detection unit 11 inputs the reference image REF to the learned model stored in the learned model storage unit 13, and obtains the object class and pixel coordinates. In step S303, the inference unit 112 of the object detection unit 11 outputs the object class and pixel coordinates obtained by the trained model to the selection unit 104. In step S304, the mask image creation unit 105 of the object detection unit 11 creates a mask image using the attribute information MAP and the topographical data DEM. In step S305, the selection unit 104 of the object detection unit 11 compares the output object class and pixel coordinates with the mask image to select an object, and outputs the pixel coordinates (i, j) of the selected object.

[0070] ***Explanation of the effect of this embodiment*** As described above, the ground control point generation device 100 according to this embodiment extracts ground control points used to correct the correction target image from the reference image and topographical data such as DEM. When extracting ground control points, the ground control point generation device 100 according to this embodiment detects objects, such as white lines on roads, that have a small deviation from the elevation represented by topographical data such as DEM and DSM, based on the image features of the objects. The ground control point generation device 100 then registers the detected objects as ground control points. As a result, the ground control point generation device 100 according to this embodiment can obtain ground control points with little deviation in height. Furthermore, the ground control point generation device 100 according to this embodiment can obtain ground control points with high accuracy in the vertical direction over a wide range where the reference image and topographical data exist, without performing on-site GNSS surveying.

[0071] Furthermore, when detecting an object whose deviation from the elevation indicated by topographical data such as DEM is small from a reference image, the ground control point generation device 100 according to this embodiment creates a mask image by referring not only to image features but also to attribute information MAP or topographical data such as DEM. The ground control point generation device 100 then extracts, as ground control points, objects whose attributes match the features of the detected object. As a result, the ground control point generation device 100 according to this embodiment can appropriately remove false detections even when false detections occur in object detection based on image features, and can obtain ground control points with high accuracy in the height direction.

[0072] ***Explanation of hardware configuration example*** FIG. 9 is a diagram showing an example of the hardware configuration of the ground control point generation device 100 according to this embodiment.

[0073] The ground control point generation device 100 is a computer. The ground control point generation device 100 includes a processor 910, as well as other hardware such as a memory 921, an auxiliary storage device 922, an input / output interface 930, and a communication interface 950. The processor 910 is connected to the other hardware via a signal line 80 and controls the other hardware.

[0074] As described above, the ground control point generation device 100 comprises, as functional elements, the object detection unit 10, the small area image extraction unit 20, the map coordinate acquisition unit 30, the height acquisition unit 40, and the data registration unit 50. The functions of the object detection unit 10, the small area image extraction unit 20, the map coordinate acquisition unit 30, the height acquisition unit 40, and the data registration unit 50 are realized by software. The functions of the object detection unit 10, small area image extraction unit 20, map coordinate acquisition unit 30, height acquisition unit 40, and data registration unit 50 may also be referred to as functions of the ground control point generation device 100. In addition, the object detection unit 10, small area image extraction unit 20, map coordinate acquisition unit 30, height acquisition unit 40, and data registration unit 50 may also be referred to as each unit of the ground control point generation device 100.

[0075] The processor 910 is a device that executes a ground control point generation program. The ground control point generation program is a program that realizes the functions of the ground control point generation device 100. The processor 910 is an IC that performs arithmetic processing. Specific examples of the processor 910 are a CPU, a DSP, and a GPU. IC is an abbreviation for Integrated Circuit. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0076] The memory 921 is a storage device that temporarily stores data. Specific examples of the memory 921 are SRAM and DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory. The auxiliary storage device 922 is a storage device that stores data. A specific example of the auxiliary storage device 922 is a HDD. The auxiliary storage device 922 may also be a portable storage medium such as an SD (registered trademark) memory card, CF, NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. Note that HDD is an abbreviation for Hard Disk Drive. SD (registered trademark) is an abbreviation for Secure Digital. CF is an abbreviation for CompactFlash (registered trademark). DVD is an abbreviation for Digital Versatile Disk.

[0077] The input / output interface 930 is an interface for connecting an input / output device. Specific examples of the input / output interface 930 include a USB and HDMI (registered trademark) port. USB is an abbreviation for Universal Serial Bus. HDMI (registered trademark) is an abbreviation for High-Definition Multimedia Interface.

[0078] The communication interface 950 is an interface for communicating with an external device, and is specifically an Ethernet (registered trademark) port or a device for wireless communication.

[0079] The ground control point generation program is executed in the ground control point generation device 100. The ground control point generation program is read into the processor 910 and executed by the processor 910. The memory 921 stores not only the ground control point generation program but also an OS. OS is an abbreviation for Operating System. The processor 910 executes the ground control point generation program while executing the OS. The ground control point generation program and the OS may be stored in an auxiliary storage device 922. The ground control point generation program and the OS stored in the auxiliary storage device 922 are loaded into the memory 921 and executed by the processor 910. Note that part or all of the ground control point generation program may be incorporated into the OS.

[0080] The ground control point generation device 100 may include multiple processors that replace the processor 910. These multiple processors share the task of executing the ground control point generation program. Each processor is a device that executes the ground control point generation program, just like the processor 910.

[0081] The data, information, signal values ​​and variable values ​​used, processed or output by the ground control point generation program are stored in memory 921, secondary storage device 922, or in registers or cache memory within processor 910.

[0082] The "unit" of each unit of the ground control point generation device 100 may be read as a "circuit," "step," "procedure," "process," or "circuitry." The ground control point generation program causes a computer to execute each process, where the "unit" of each unit of the ground control point generation device 100 is read as a "process." The "process" of each process of the ground control point generation device 100 may be read as a "program," "program product," "computer-readable storage medium storing a program," or "computer-readable recording medium recording a program." Furthermore, the ground control point generation method is a method performed by the ground control point generation device 100 executing the ground control point generation program. The ground control point generation program may be provided by being stored in a computer-readable recording medium, or may be provided as a program product.

[0083] ***Other Configurations*** <Modification> In this embodiment, the functions of the respective units of the ground control point generation device 100 are realized by software. As a modified example, the functions of the respective units of the ground control point generation device 100 may be realized by hardware. Specifically, the ground control point generation device 100 includes an electronic circuit 909 instead of the processor 910 .

[0084] FIG. 10 is a diagram showing an example of the hardware configuration of a ground control point generating device 100 according to a modified example of this embodiment. The electronic circuit 909 is a dedicated electronic circuit that realizes the functions of each part of the ground control point generation device 100. Specifically, the electronic circuit 909 is a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, or an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array.

[0085] The functions of each unit of the ground control point generation device 100 may be realized by one electronic circuit, or may be distributed and realized by multiple electronic circuits.

[0086] As another modification, some of the functions of each unit of the ground control point generation device 100 may be realized by electronic circuits, and the remaining functions may be realized by software. Also, some or all of the functions of each unit of the ground control point generation device 100 may be realized by firmware.

[0087] Each of the processor and electronic circuits is also called a processing circuitry. In other words, the functions of each part of the ground control point generation device 100 are realized by the processing circuitry.

[0088] Embodiment 2 In this embodiment, differences from and additions to the first embodiment will be mainly described. In this embodiment, components having the same functions as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0089] In this embodiment, we will explain an image processing system 500 that corrects the position and orientation information of the correction target image SRC using the output of the ground control point generation device 100 described in embodiment 1 or the components included in the ground control point generation device 100.

[0090] ***Configuration Description*** FIG. 11 is a diagram showing an example of the configuration of an image processing system 500 according to this embodiment. The image processing system 500 includes a ground control point generation device 100, a ground control point database 200, and an image processing device 400. The ground control point database 200 may be included in the ground control point generation device 100. The image processing system 500 may also include a learning device 300. An example of the hardware configuration of the image processing device 400 according to this embodiment is the same as that of the ground control point generation device 100 described in the first embodiment.

[0091] The image processing device 400 includes an image matching unit 401 and a position and orientation information correction unit 402. The image processing device 400 is also connected to the ground control point database 200 generated by the ground control point generation device 100, and acquires ground control point data GCP from the ground control point database 200.

[0092] The image matching unit 401 receives the correction target image SRC as input. The image matching unit 401 extracts a predetermined area in the correction target image SRC and compares it with a small area image in the ground control point data GCP to search for a position where the degree of match between the two is high. The image matching unit 401 then outputs pixel coordinates in the correction target image SRC that correspond to the ground control point.

[0093] It is assumed that the correction target image SRC has position and orientation information that includes errors calculated from the trajectory and orientation information of the platform at the time of observation. By using the position and orientation information of the correction target image SRC, pixel coordinates can be obtained when the three-dimensional coordinates (X, Y, Z) of the ground control point data are projected onto the correction target image SRC. However, because the position and orientation information includes errors, there will be a discrepancy between the pixel coordinates projected from the three-dimensional coordinates of the ground control point data and the pixel coordinates where the ground control point actually appears in the correction target image SRC. To detect this deviation, the image matching unit 401 sets an area of ​​a predetermined size centered on pixel coordinates projected from the three-dimensional coordinates of the ground control point data as a search range in the correction target image SRC. Then, the image matching unit 401 obtains the pixel coordinates where the ground control point is captured by matching the small-area image of the corresponding ground control point with the search range in the correction target image SRC.

[0094] The size of the search range is determined based on the range in which the error in the position and orientation information is expected. If the correction target image SRC does not have position and orientation information, the entire image becomes the search range.

[0095] When the small-region image is simply image data, matching is performed by calculating the similarity between the small-region image and the image at each position within the search range in the correction target image SRC. Specifically, the image matching unit 401 calculates the similarity using an index such as the sum of absolute differences or cross-correlation, and performs matching by detecting the position with the highest similarity. When the small-region image has image features such as SIFT, the image matching unit 401 calculates the same image features as the small-region image at each position within the search range of the correction target image SRC, and performs matching by detecting the position with the closest distance to the image features.

[0096] Next, a position and orientation information correction unit 402 corrects the position and orientation information of the correction target image SRC based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates in the correction target image SRC detected by the image matching unit 401. Then, the position and orientation information correction unit 402 outputs a corrected image DST in which the position and orientation information of the correction target image SRC has been corrected.

[0097] A specific example of a method for correcting position and orientation information when the position and orientation information is provided by an RPC will be described. An RPC is a set of coefficients of a rational polynomial that associates a location on the Earth, indicated by latitude, longitude, and altitude, with a pixel location, which is a position in an image. Ideally, the coefficients are corrected so that the result of converting the three-dimensional coordinates (X, Y, Z) included in the ground control point data GCP generated by the ground control point generation device 100 using the RPC becomes the matching pixel coordinates (x, y) output from the image matching unit 401. In practice, the position and orientation information is corrected by optimizing the RPC coefficient set so that the error between the pixel coordinates resulting from converting the three-dimensional coordinates (X, Y, Z) using the RPC and the matching pixel coordinates (x, y) becomes small. However, if the number of ground control point data GCPs is small, a low-order transformation such as a parallel translation or affine transformation may be applied to the pixel coordinates resulting from the transformation using the RPC to find the coefficients of the transformation that bring them closer to the matching pixel coordinates (x, y).

[0098] The corrected image DST, whose position and orientation information has been corrected in this way, can then be used as an image in which the position shown on the image matches the position shown on the actual map by further performing ortho-correction (map projection).

[0099] ***Explanation of Operation*** Next, a description will be given of the operation of image processing device 400 according to this embodiment. The operation procedure of image processing device 400 corresponds to an image processing method. Furthermore, a program that realizes the operation of image processing device 400 corresponds to an image processing program.

[0100] FIG. 12 is a flow diagram showing the operation of image processing device 400 according to this embodiment. In step S401, the ground control point data GCP is generated from the reference image REF and the topographical data DEM by the operation of the ground control point generation device 100 described in FIG. The following processing in step S402 is repeated for each of the generated ground control point data GCPs.

[0101] In step S402, the image matching unit 401 performs image matching processing with the correction target image SRC for the ground control point data GCP. When the image matching unit 401 has determined the matching pixel coordinates (x, y) corresponding to all the ground control point data GCPs, it outputs them to the position and orientation information correction unit 402, and the process proceeds to step S403.

[0102] In step S403, the position and orientation information correction unit 402 corrects the position and orientation information and outputs the corrected image DST. Specifically, the position and orientation information correction unit 402 corrects the position and orientation information of the correction target image SRC based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates in the correction target image SRC. Then, the position and orientation information correction unit 402 outputs the corrected image DST in which the position and orientation information of the correction target image SRC has been corrected.

[0103] The generation of ground control points by the ground control point generation device 100 (step S401) may be performed at a timing separate from the image matching process (step S402) or the position and orientation information correction process (step S403).

[0104] ***Explanation of the effect of this embodiment*** As described above, the image processing device 400 according to this embodiment can correct the position and orientation information of the image to be corrected using ground control point data generated by the ground control point generation device. This makes it possible to correct the position information of the image to be corrected with high accuracy even in areas where ground control points measured by GNSS surveying have not been established.

[0105] In the above first and second embodiments, each unit of the ground control point generation device and the image processing device has been described as an independent functional block. However, the configuration of each device of the ground control point generation device and the image processing device does not have to be the same as that of the above-mentioned embodiments. The functional blocks of each device of the ground control point generation device and the image processing device may have any configuration as long as they can realize the functions described in the above-mentioned embodiments. Furthermore, each device of the ground control point generation device and the image processing device may not be a single device, but may be a system composed of multiple devices. Furthermore, it is possible to combine multiple parts of the first and second embodiments. Alternatively, it is possible to implement only one part of these embodiments. In addition, it is possible to combine these embodiments in any way, either as a whole or in part. That is, in the first and second embodiments, the respective embodiments can be freely combined, or any of the components in the respective embodiments can be modified, or any of the components in the respective embodiments can be omitted.

[0106] The above-described embodiments are essentially preferred examples and are not intended to limit the scope of the present disclosure, the scope of application of the present disclosure, or the scope of use of the present disclosure. The above-described embodiments can be modified in various ways as needed. For example, the procedures described using flow charts or sequence diagrams may be modified as appropriate.

[0107] Various aspects of the present disclosure are summarized below as appendices.

[0108] (Appendix 1) A ground control point generation device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, an object detection unit that outputs an object class representing the type of object and pixel coordinates of the object for an object installed at a height approximately equal to the average elevation of the surrounding land; a small-area image extraction unit that extracts an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition unit that acquires map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition unit that acquires a height of the object by referring to the topographical data corresponding to pixel coordinates of the object; a data registration unit that registers data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; A ground control point generating device comprising: (Appendix 2) The object detection unit 2. A ground control point generating device according to claim 1, which identifies and positions the object based on local image features of the reference image. (Appendix 3) The object detection unit a feature extraction unit that extracts local image features of the reference image; a classification unit that classifies at least one object class having predetermined image features based on the image feature amount; a position specifying unit that detects the position of a representative point of an image region having the predetermined image feature; 3. A ground control point generating device according to claim 2, comprising: (Appendix 4) The object detection unit 2. The ground control point generation device according to claim 1, further comprising an inference unit that outputs an object class and pixel coordinates from the reference image using a trained model generated from training data including a training image corresponding to the reference image and an object class and pixel coordinates corresponding to the training image. (Appendix 5) The object detection unit a mask image creation unit that creates a mask image based on attribute information indicating attributes of features corresponding to map coordinates or the topographical data; a selection unit that selects a detection result of the object by comparing the object class with the attribute indicated in the mask image; 5. The ground control point generating device according to claim 1, comprising: (Appendix 6) A ground control point generation device according to any one of Supplementary Note 1 to Supplementary Note 5; an image processing device including an image matching unit that extracts a predetermined area of ​​the image to be corrected and compares it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputs pixel coordinates of the image to be corrected that correspond to the ground control points; and a position and orientation information correction unit that corrects position and orientation information of the image to be corrected based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates of the image to be corrected; An image processing system comprising: (Appendix 7) A ground control point generating method used in a ground control point generating device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, comprising: The computer outputs, for an object installed at a height approximately equal to the average elevation of the surrounding land, an object class representing the type of the object and pixel coordinates of the object; a computer extracting an image or image feature of a small region of the reference image including the object from pixel coordinates of the object as a small region image; a computer obtains map coordinates of the object from the pixel coordinates of the object using the correspondence between the reference image and map coordinates; the computer obtains the height of the object by referencing the topographical data corresponding to pixel coordinates of the object; A ground control point generating method in which a computer registers data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database. (Appendix 8) An image processing method for an image processing system that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, and corrects the image to be corrected using the ground control points, The computer outputs, for an object installed at a height approximately equal to the average elevation of the surrounding land, an object class representing the type of the object and pixel coordinates of the object; a computer extracting an image or image feature of a small region of the reference image including the object from pixel coordinates of the object as a small region image; a computer obtains map coordinates of the object from the pixel coordinates of the object using the correspondence between the reference image and map coordinates; the computer obtains the height of the object by referencing the topographical data corresponding to pixel coordinates of the object; The computer registers data linking the small-area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; a computer extracts a predetermined area of ​​the image to be corrected and compares it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputs pixel coordinates of the image to be corrected that correspond to the ground control point; An image processing method in which a computer corrects position and orientation information of the image to be corrected based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates of the image to be corrected. (Appendix 9) A ground control point generation program used in a ground control point generation device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, comprising: an object detection step for outputting an object class representing the type of object and pixel coordinates of the object for an object installed at a height approximately equal to the average elevation of the surrounding land; a small-area image extraction process for extracting an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition process for acquiring map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition process for acquiring a height of the object by referencing the topographical data corresponding to pixel coordinates of the object; a data registration process for registering data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; A ground control point generation program that causes a computer to execute the above. (Appendix 10) An image processing program used in an image processing system that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, and corrects the image to be corrected using the ground control points, an object detection step for outputting an object class representing the type of object and pixel coordinates of the object for an object installed at a height approximately equal to the average elevation of the surrounding land; a small-area image extraction process for extracting an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition process for acquiring map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition process for acquiring a height of the object by referencing the topographical data corresponding to pixel coordinates of the object; a data registration process for registering data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; an image matching process for extracting a predetermined area of ​​the image to be corrected and comparing it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputting pixel coordinates of the image to be corrected that correspond to the ground control point; a position and orientation information correction processing unit that corrects position and orientation information of the correction target image based on a correspondence relationship between the three-dimensional coordinates of the ground control point data and the pixel coordinates of the correction target image; An image processing program that causes a computer to execute the following. [Explanation of symbols]

[0109] 100 ground control point generation device, 10,11 object detection unit, 20 small area image extraction unit, 30 map coordinate acquisition unit, 40 height acquisition unit, 50 data registration unit, 101 feature extraction unit, 102 identification unit, 103 position identification unit, 104 selection unit, 105 mask image creation unit, 111 data acquisition unit, 112 inference unit, 121 data acquisition unit, 122 model generation unit, 13 trained model storage unit, 401 image matching unit, 402 position and orientation information correction unit, 200 ground control point database, 300 learning device, 400 image processing device, 500 image processing system, 80 signal line, 909 electronic circuit, 910 processor, 921 memory, 922 auxiliary storage device, 930 input / output interface, 950 communication interface.

Claims

1. A ground control point generation device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, an object detection unit that identifies an object located at a height approximately equal to the average elevation of the surrounding land based on local image features of the reference image, outputs an object class representing the type of the object, and identifies the position of the object and outputs pixel coordinates of the object; a small-area image extraction unit that extracts an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition unit that acquires map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition unit that acquires a height of the object by referring to the topographical data corresponding to pixel coordinates of the object; a data registration unit that registers data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; Equipped with The object detection unit a mask image creation unit that acquires a land cover classification map as an attribute information map, acquires attribute information indicating attributes of features corresponding to map coordinates based on the attribute information map, and creates a mask image based on the attribute information or the topographical data; a selection unit that selects a detection result of the object by comparing the object class with the attribute indicated in the mask image; A ground control point generating device comprising:

2. The object detection unit a feature extraction unit that extracts local image features of the reference image; a classification unit that classifies at least one object class having predetermined image features based on the image feature amount; a position specifying unit that detects the position of a representative point of an image region having the predetermined image feature; The ground control point generating device according to claim 1 , comprising:

3. The object detection unit 2. The ground control point generation device according to claim 1, further comprising an inference unit that outputs an object class and pixel coordinates from the reference image using a trained model generated from training data including a training image corresponding to the reference image and an object class and pixel coordinates corresponding to the training image.

4. The ground control point generating device according to claim 1 ; an image processing device including an image matching unit that extracts a predetermined area of ​​the image to be corrected and compares it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputs pixel coordinates of the image to be corrected that correspond to the ground control points; and a position and orientation information correction unit that corrects position and orientation information of the image to be corrected based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates of the image to be corrected; An image processing system comprising:

5. A ground control point generating method used in a ground control point generating device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, comprising: a computer identifies an object located at a height approximately equal to the average elevation of the surrounding land based on local image features of the reference image, outputs an object class representing the type of the object, and identifies the position of the object and outputs pixel coordinates of the object; a computer extracting an image or image feature of a small region of the reference image including the object from pixel coordinates of the object as a small region image; a computer obtains map coordinates of the object from the pixel coordinates of the object using the correspondence between the reference image and map coordinates; the computer obtains the height of the object by referencing the topographical data corresponding to pixel coordinates of the object; a computer registering data linking the small-area image with three-dimensional coordinates consisting of map coordinates and heights of the object as ground control point data in a ground control point database, the computer comprising: A ground control point generation method in which a computer acquires a land cover classification map as an attribute information map, acquires attribute information indicating the attributes of features corresponding to map coordinates based on the attribute information map, generates a mask image based on the attribute information or the topographical data, and selects the detection result of the object by comparing the object class with the attributes indicated in the mask image.

6. An image processing method for an image processing system that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, and corrects the image to be corrected using the ground control points, a computer identifies an object located at a height approximately equal to the average elevation of the surrounding land based on local image features of the reference image, outputs an object class representing the type of the object, and identifies the position of the object and outputs pixel coordinates of the object; a computer extracting an image or image feature of a small region of the reference image including the object from pixel coordinates of the object as a small region image; a computer obtains map coordinates of the object from the pixel coordinates of the object using the correspondence between the reference image and map coordinates; the computer obtains the height of the object by referencing the topographical data corresponding to pixel coordinates of the object; The computer registers data linking the small-area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; a computer extracts a predetermined area of ​​the image to be corrected and compares it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputs pixel coordinates of the image to be corrected that correspond to the ground control point; an image processing method in which a computer corrects position and orientation information of the correction target image based on a correspondence relationship between three-dimensional coordinates of the ground control point data and pixel coordinates of the correction target image, An image processing method in which a computer acquires a land cover classification map as an attribute information map, acquires attribute information indicating the attributes of features corresponding to map coordinates based on the attribute information map, generates a mask image based on the attribute information or the topographical data, and selects the detection result of the object by comparing the object class with the attributes indicated in the mask image.

7. A ground control point generation program used in a ground control point generation device that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, comprising: an object detection process for identifying an object located at a height approximately equal to the average elevation of the surrounding land based on local image features of the reference image, outputting an object class representing the type of the object, and locating the object and outputting pixel coordinates of the object; a small-area image extraction process for extracting an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition process for acquiring map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition process for acquiring a height of the object by referencing the topographical data corresponding to pixel coordinates of the object; a data registration process for registering data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database, The object detection process includes: a mask image creation process for acquiring a land cover classification map as an attribute information map, acquiring attribute information indicating attributes of features corresponding to map coordinates based on the attribute information map, and generating a mask image based on the attribute information or the topographical data; a selection process for selecting a detection result of the object by matching the object class with the attributes shown in the mask image; A ground control point generation program comprising:

8. An image processing program used in an image processing system that generates ground control points using a reference image that includes at least the same field of view as an image to be corrected and that is associated with map coordinates, and topographical data that includes at least the same map range as the reference image, and corrects the image to be corrected using the ground control points, an object detection process for identifying an object located at a height approximately equal to the average elevation of the surrounding land based on local image features of the reference image, outputting an object class representing the type of the object, and locating the object and outputting pixel coordinates of the object; a small-area image extraction process for extracting an image or image feature of a small area of ​​the reference image including the object as a small-area image from pixel coordinates of the object; a map coordinate acquisition process for acquiring map coordinates of the object from pixel coordinates of the object using a correspondence relationship between the reference image and map coordinates; a height acquisition process for acquiring a height of the object by referencing the topographical data corresponding to pixel coordinates of the object; a data registration process for registering data linking the small area image with three-dimensional coordinates consisting of the map coordinates and height of the object as ground control point data in a ground control point database; an image matching process for extracting a predetermined area of ​​the image to be corrected and comparing it with the small area image in the ground control point data to search for a position where the degree of match between the two is high, and outputting pixel coordinates of the image to be corrected that correspond to the ground control point; a position and orientation information correction process for correcting the position and orientation information of the correction target image based on the correspondence between the three-dimensional coordinates of the ground control point data and the pixel coordinates of the correction target image; An image processing program that causes a computer to execute the following: The object detection process includes: a mask image creation process for acquiring a land cover classification map as an attribute information map, acquiring attribute information indicating attributes of features corresponding to map coordinates based on the attribute information map, and generating a mask image based on the attribute information or the topographical data; a selection process for selecting a detection result of the object by matching the object class with the attributes shown in the mask image; An image processing program comprising:

Images