Image processing device, image processing method, and program

Abstract

Provided are an image processing device, an image processing method, and a program for setting a GCP in an image without work for installing an air sign and GCP setting work performed by a person. The present invention comprises: one or more processors; and one or more memories in which programs to be executed by the one or more processors are stored. The processor executes instructions of the programs, thereby: acquiring an image group in which a fixed imaging region is captured using a camera; selecting, from the image group, a setting image for setting a ground reference point; identifying a map corresponding to the imaging region of the setting image; identifying, from the map, an object for setting the ground reference point; searching for a candidate position corresponding to the position of the object from the setting image; and setting the candidate position to the ground reference point.

Description

Image processing device, image processing method and program

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to a technique for setting ground control points in an image.

[0002] In recent years, three-dimensional models generated using Structure from Motion (SfM) based on images taken by drones have come to be used to assess damage situations during disasters and for surveying during normal times.

[0003] Patent Literature 1 discloses an image processing technology that includes a process for associating an image captured by a camera mounted on a drone with the spatial position of the target range. According to the technology described in Patent Literature 1, the parameter values ​​of the camera matrix are automatically searched for and optimal parameter values ​​are selected based on sensor data obtained from the drone, which enables highly accurate alignment of the map data of the target range with the captured image without the need for a human to specify corresponding points.

[0004] International Publication No. 2023 / 047799

[0005] Relationship between the number of GCPs and drone map accuracy [Retrieved June 26, 2023], Internet <URL: https: / / www.pix4d.com / jp / blog / GCP-accuracy-drone-maps / >

[0006] When a 3D model and orthoimages are generated from aerial images using SfM processing without using GCPs (Ground Control Points) whose latitude and longitude are known, a misalignment of approximately 5 to 10 meters occurs when the 3D model and orthoimages are overlaid on a map due to factors such as positional deviations in the GPS (Global Positioning System) information when the images were captured.

[0007] On the other hand, by installing aerial markers on ground GCPs and extracting the aerial markers from aerial images, it is possible to generate 3D models and orthoimages that can be accurately overlaid on maps. However, installing aerial markers over a wide area of ​​a photograph is time-consuming. Furthermore, while it is possible for a person to visually check the image and map and manually set corresponding points on the GCP, this process is also time-consuming.

[0008] The present invention has been made in consideration of the above circumstances, and aims to provide an image processing device, an image processing method, and a program that set a GCP in an image without the need to install anti-aircraft markers or manually set a GCP.

[0009] In order to achieve the above object, an image processing device according to a first aspect of the present disclosure is an image processing device that includes one or more processors and one or more memories that store programs to be executed by the one or more processors, and the processor executes instructions of the program to acquire a group of images of a certain shooting area taken using a camera, select a setting image for setting a ground control point from the group of images, identify a map corresponding to the shooting area of ​​the setting image, identify an object for setting a ground control point from the map, search for a candidate position corresponding to the position of the object from the setting image, and set the candidate position as a ground control point.

[0010] According to the first aspect, it is possible to set a GCP in an image without the need to install an anti-aircraft marker or manually set a GCP.

[0011] An image processing device according to a second aspect of the present disclosure is the image processing device according to the first aspect, wherein the processor preferably selects a plurality of setting images.

[0012] In the image processing device according to the third aspect of the present disclosure, in the image processing device according to the first or second aspect, it is preferable that the processor divides the shooting area into a plurality of setting image selection areas, each containing a plurality of images, calculates the number of features in each image in the group of images, and for each setting image selection area of ​​the plurality of setting image selection areas, selects an image with a relatively large number of features from the plurality of images included in the setting image selection area as the setting image.

[0013] An image processing device according to a fourth aspect of the present disclosure is the image processing device according to any one of the first to third aspects, wherein the object is preferably a bending point of a road.

[0014] In an image processing device according to a fifth aspect of the present disclosure, in the image processing device according to the fourth aspect, it is preferable that the processor extracts road bend points from the map, converts the setting image into a line segment image, extracts bend points from the line segment image as candidate positions, and sets the bend points of the line segment image corresponding to the road bend points as ground reference points.

[0015] In an image processing device according to a sixth aspect of the present disclosure, in an image processing device according to any one of the first to fifth aspects, it is preferable that the processor displays on a display device an extraction result image in which a figure is superimposed at the position of the ground reference point of the setting image.

[0016] An image processing device according to a seventh aspect of the present disclosure is preferably the image processing device according to the sixth aspect, wherein the processor causes the display device to display an enlarged image of the positions of the ground reference points of the setting image.

[0017] In an image processing device according to an eighth aspect of the present disclosure, in the image processing device according to the sixth or seventh aspect, it is preferable that the processor displays the extraction result image and a confirmation image based on the setting image side by side on the display device, and further displays a decision button on the display device that allows the user to decide whether or not to adopt the ground reference point.

[0018] In an image processing device according to a ninth aspect of the present disclosure, in the image processing device according to the eighth aspect, it is preferable that the processor calculates a reliability indicating how reliable the set ground control point is as a ground control point, and displays the reliability on a display device.

[0019] In an image processing device according to a tenth aspect of the present disclosure, in an image processing device according to any of the first to ninth aspects, it is preferable that the processor calculates a reliability indicating how reliable the candidate position is as a ground control point, and sets the candidate position as a ground control point according to the reliability.

[0020] An image processing device according to an eleventh aspect of the present disclosure is an image processing device according to any one of the first to tenth aspects, wherein the group of images is captured with an area that overlaps with adjacent images, and the processor preferably sets ground control points set in the overlapping areas as tie points, and calculates the position and attitude of the camera when it captured each image in the group of images based on the tie points.

[0021] In the image processing device according to the twelfth aspect of the present disclosure, in the image processing device according to any of the first to eleventh aspects, it is preferable to acquire an overall map corresponding to a certain shooting area and align each image in the image group with the overall map using the set ground control points.

[0022] In order to achieve the above object, an image processing method according to a thirteenth aspect of the present disclosure is an image processing method executed by one or more processors, the image processing method including the processor acquiring a group of images of a certain shooting area using a camera, selecting a setting image for setting a ground control point from the group of images, identifying a map corresponding to the shooting area of ​​the setting image, identifying an object for setting a ground control point from the map, searching for a candidate position corresponding to the position of the object from the setting image, and setting the candidate position as a ground control point.

[0023] According to the thirteenth aspect, a GCP can be set in an image without the need for installing an anti-aircraft marker or manually setting a GCP.

[0024] In order to achieve the above object, the program according to the fourteenth aspect of the present disclosure is a program that enables a computer to perform the following functions: acquire a group of images of a certain shooting area taken using a camera; select a setting image for setting a ground reference point from the group of images; identify a map corresponding to the shooting area of ​​the setting image; identify an object for setting a ground reference point from the map; search for a candidate position corresponding to the position of the object from the setting image; and set the candidate position as a ground reference point.

[0025] According to the fourteenth aspect, it is possible to set a GCP in an image without the need for the installation of an anti-aircraft beacon or the need for manual GCP setting work.

[0026] The present disclosure also includes a non-transitory computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) that stores the program according to the fourteenth aspect.

[0027] According to the present invention, it is possible to set a GCP in an image without the need for the installation of anti-aircraft markers or the need for manual GCP setting work.

[0028] FIG. 1 is a schematic diagram showing an example of the configuration of a captured image processing system. FIG. 2 is a block diagram showing an example of the electrical configuration of a drone equipped with a camera. FIG. 3 is a block diagram showing an example of the hardware configuration of an image processing device. FIG. 4 is a block diagram showing an example of the functional configuration of the image processing device. FIG. 5 is a flowchart showing each step of an image processing method according to a first embodiment. FIG. 6 is an example of multiple images acquired by a captured image acquisition unit. FIG. 7 is a diagram showing an image included in the multiple images and a Geospatial Information Authority of Japan map corresponding to the image. FIG. 8 is a diagram showing multiple images and a calculation result of the number of houses in each of the multiple images. FIG. 9 is a diagram for explaining the process of selecting an image for setting a GCP. FIG. 10 is a diagram showing the result of applying geocoding to an image for setting a GCP. FIG. 11 is a diagram for explaining the extraction of a GCP. FIG. 12 is a diagram showing an example of a GCP confirmation / correction screen. FIG. 13 is a diagram showing another example of a GCP confirmation / correction screen. FIG. 14 is a flowchart showing each step of an image processing method according to a second embodiment. FIG. 15 is a diagram for explaining automatic GCP extraction and tie point setting. FIG. 16 is a diagram for explaining automatic extraction of GCPs and setting of tie points.

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In this specification, the same components are designated by the same reference numerals, and redundant explanations will be omitted where appropriate.

[0030] 1 is a schematic diagram showing an example configuration of a photographing and image processing system 10. The photographing and image processing system 10 includes a drone 12 for aerial photography, a camera 14 mounted on the drone 12, a remote controller 16, and an image processing device 20. The drone 12 is an unmanned aerial vehicle that is remotely controlled using the remote controller 16. The drone 12 may have an autopilot function that flies according to a program.

[0031] The camera 14 is mounted on the drone 12 via a gimbal head 13. The camera 14 includes an optical system, an image sensor, and a signal processing circuit (not shown). The optical system includes one or more lenses such as a focus lens. The image sensor may be, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor.

[0032] The camera 14 generates digital image data of the photographed subject by processing signals obtained from the image sensor using a signal processing circuit. The digital image data generated by the camera 14 can be considered an "image." Images captured using the camera 14 are stored in an internal storage device built into the drone 12 and / or a storage device such as a memory card removably attached to the drone 12. In addition, images captured using the camera 14 may be transferred to the remote controller 16 or the image processing device 20 using wireless communication.

[0033] The remote controller 16 is a transmitter that controls the operation of the camera 14 and the drone 12 via wireless communication. The wireless communication may be in the form of a wireless local area network (LAN). The wireless communication may be in the form of a communication using radio waves in the 2.4 GHz band or 5.7 GHz band. The wireless communication may be in the form of a mobile communication network. The communication format for the control signal for operating the drone 12 and the communication format for transferring images captured using the camera 14 may be different or the same.

[0034] The remote controller 16 includes a display 16A, left and right sticks (not shown) for controlling the flight operations of the drone 12, a lever (not shown) for operating the gimbal head 13, a shooting button (not shown) for instructing the camera 14 to take a photograph, and a shooting mode button (not shown) for switching between video shooting and still image shooting.

[0035] The display 16A may be a touch panel display. Various operations on the drone 12, the gimbal head 13, and the camera 14 may be performed by touching the touch panel display. Touch operations include a tap operation, a double tap operation, a flick operation, a swipe operation, a drag operation, a pinch-in operation, and a pinch-out operation.

[0036] The live video captured by the camera 14 is displayed on the display 16A of the remote controller 16. The remote controller 16 also grasps the status of the drone 12, such as its flight position and flight speed, in real time based on data from various sensors provided on the drone 12. Flight information indicating the status of the drone may be displayed on the display 16A.

[0037] The photographed image processing system 10 photographs a plurality of still images (photographed images) from the air using a camera 14 and processes the photographed images in an image processing device 20 .

[0038] The image processing device 20 automatically sets GCPs (Ground Control Points) required for generating a highly accurate 3D model and an orthoimage from a group of aerial photographic images in which adjacent images have overlapping regions. A GCP is a point on the ground whose latitude, longitude, and altitude are known and which is a characteristic topographical point visible in the image. The image processing device 20 may generate a highly accurate 3D model and an orthoimage from a group of aerial photographic images.

[0039] The image processing device 20 is configured by a computer. The computer applied to the image processing device 20 may be a server, a personal computer, or a workstation.

[0040] The image processing device 20 performs data communication with the remote controller 16 via a network 22. The network 22 may be a local area network or a wide area network. The image processing device 20 acquires various information from the drone 12 and the camera 14. The image processing device 20 also acquires map data of the imaging target range from a geographic information system (not shown) via the network 22. The image processing device 20 may acquire the map data in advance before capturing an image with the camera 14, or may acquire the map data after capturing an image with the camera 14.

[0041] 2 is a block diagram showing an example of the electrical configuration of a drone 12 equipped with a camera 14. The drone 12 includes a GPS (Global Positioning System) receiver 30, an air pressure sensor 32, a direction sensor 34, a gyro sensor 36, a motor 38, a processor 40, a storage device 42, a communication interface 44, a battery (not shown), and a battery charging terminal.

[0042] The GPS receiver 30 acquires location information including the latitude and longitude of the drone 12. The barometric pressure sensor 32 detects the barometric pressure at the drone 12's location. The drone 12 acquires the altitude of the drone 12's location based on the barometric pressure detected using the barometric pressure sensor 32. The term "acquire" includes the concept of generating information by data processing such as calculation. The latitude, longitude, and altitude of the drone 12 constitute the location information of the drone 12 and the camera 14.

[0043] The orientation sensor 34 may be, for example, a geomagnetic sensor. The drone 12 uses the orientation sensor 34 to detect the azimuth angle at which the lens of the camera 14 is facing.

[0044] The gyro sensor 36 detects a roll angle representing the angle of rotation about the roll axis, a pitch angle representing the angle of rotation about the pitch axis, and a yaw angle representing the angle of rotation about the yaw axis. The drone 12 acquires attitude information of the camera 14 based on the rotation angles acquired using the gyro sensor 36. Note that some or all of the sensors, such as the GPS receiver 30, the air pressure sensor 32, the orientation sensor 34, and the gyro sensor 36, may be located on the camera 14 side.

[0045] The motor 38 is a power source that rotates rotors (not shown). The drone 12 includes a plurality of motors 38 that drive a plurality of rotors.

[0046] The storage device 42 may be a memory, an internal storage, an external storage device, or a combination thereof. The processor 40 serves as a flight controller and performs various calculations necessary for flight control of the drone 12 based on sensor data obtained from various sensors.

[0047] The communication interface 44 is a communication unit that performs wireless communication with the remote controller 16, etc. The communication interface 44 may include a communication terminal that supports wired communication.

[0048] 3 is a block diagram showing an example of the hardware configuration of the image processing device 20. The image processing device 20 includes one or more processors 202, one or more computer-readable media 204, a communication interface 206, an input / output interface 208, and a bus 210.

[0049] The hardware structure of the processor 202 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various functional units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electrical circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0050] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple functional units may also be configured with a single processor. Examples of multiple functional units configured with a single processor include: a first configuration, as typified by a client or server computer, in which a single processor is configured with a combination of one or more CPUs and software, and this processor operates as multiple functional units; and a second configuration, as typified by a SoC (System on Chip), in which a processor is used to realize the functions of an entire system including multiple functional units on a single IC (Integrated Circuit) chip. In this way, the various functional units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0051] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0052] Processor 202 is connected to computer-readable medium 204 , communication interface 206 , and input / output interface 208 via bus 210 .

[0053] The computer-readable medium 204 stores instructions to be executed by the processor 202. The computer-readable medium 204 includes a memory serving as a main storage device and a storage serving as an auxiliary storage device. The computer-readable medium 204 may be, for example, a semiconductor memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these. The computer-readable medium 204 stores various programs, including an image processing program, and data.

[0054] The communication interface 206 controls communications over the network 22 .

[0055] The input / output interface 208 is connected to an input device 214 and a display device 216 and controls input and output to and from the image processing device 20 .

[0056] The input device 214 is configured by, for example, a keyboard, a mouse, a multi-touch panel, other pointing devices, a voice input device, or an appropriate combination of these.

[0057] The display device 216 is configured by, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination of these.

[0058] The image processing device 20 may include an input device 214 and a display device 216 .

[0059] 4 is a block diagram showing an example of the functional configuration of the image processing device 20. The image processing device 20 includes a captured image acquisition unit 100, a shooting condition acquisition unit 102, a map information acquisition unit 104, a GCP setting image selection unit 106, a geocoding application unit 108, a GCP setting unit 110, a tie point setting unit 112, an SfM processing unit 114, an input reception unit 116, and a display control unit 118. Each function of the image processing device 20 is realized by the processor 202 executing a program stored in a computer-readable medium 204.

[0060] The captured image acquisition unit 100 acquires a plurality of images (an example of an "image group") of a certain captured area captured using the camera 14 from the camera 14. The captured image acquisition unit 100 acquires a plurality of images, each of which is a capture of a portion of the captured area, and each of which has an overlapping area with adjacent images (an example of an "overlapping area").

[0061] The photographing condition acquisition unit 102 acquires the photographing conditions for each of the multiple images from the drone 12. The photographing conditions include at least one of location information including the latitude, longitude, and altitude of the camera 14 at the time of photographing, azimuth angle information to which the lens of the camera 14 is directed, and attitude information of the camera 14.

[0062] The map information acquisition unit 104 acquires map information including the image capture areas of the multiple images acquired by the captured image acquisition unit 100. The map information acquisition unit 104 may acquire the map information from the computer-readable medium 204, from the input device 214, or via the network 22.

[0063] The GCP setting image selection unit 106 selects a GCP setting image for setting a GCP from the plurality of images acquired by the photographed image acquisition unit 100. It is preferable that the GCP setting image selection unit 106 selects a plurality of GCP setting images evenly across the photographed areas of the plurality of images. It is also preferable that the GCP setting image selection unit 106 selects a plurality of GCP setting images from images in which a relatively large number of houses are photographed.

[0064] It is known that 5 to 10 GCPs are sufficient for mapping an image and a map (see Non-Patent Document 1). Therefore, the GCP setting image selection unit 106 selects 5 to 10 GCP setting images from among a plurality of images.

[0065] The GCP setting image selection unit 106 includes an area division unit 106A and a feature number calculation unit 106B. The area division unit 106A divides the photographed area captured in the group of images acquired by the photographed image acquisition unit 100 into a plurality of setting image selection areas, each including a plurality of images. The feature number calculation unit 106B calculates the number of features captured in each image in the group of images acquired by the photographed image acquisition unit 100. The features are, for example, houses. Houses are not limited to residential buildings such as "detached houses" and "apartment buildings," but may also include buildings in general, such as "stores," "offices," "schools," and "factories."

[0066] The GCP setting image selection unit 106 may select, for each of the multiple setting image selection areas, an image with a relatively large number of features from among the multiple images included in the setting image selection area as the GCP setting image.

[0067] The geocoding application unit 108 applies geocoding to the GCP setting image selected by the GCP setting image selection unit 106, matches the line segments extracted from the image with the line segments extracted from the map, and projects the road line segments of the map onto the GCP setting image with an accuracy of within about 3 to 5 meters. Note that in this specification, "geocoding" refers to the alignment technology described in Patent Document 1.

[0068] The GCP setting unit 110 identifies a map corresponding to the shooting area of ​​the GCP setting image selected by the GCP setting image selection unit 106, and identifies an object for setting a ground control point from the map. Furthermore, the GCP setting unit 110 searches for a candidate position corresponding to the position of the identified object from the setting image, and sets the candidate position to the GCP. The GCP setting unit 110 may display an extraction result image in which a figure is superimposed on the position of the GCP set in the GCP setting image on the display device 216. The GCP setting unit 110 may display an enlarged image obtained by enlarging and cutting out the position of the GCP set in the GCP setting image on the display device 216.

[0069] The GCP setting unit 110 may display an extraction result image in which a graphic is superimposed at the position of the set GCP in the GCP setting image and a confirmation image based on the GCP setting image side by side on the display device 216. The GCP setting unit 110 may display a decision button on the display device 216 for the user to decide whether or not to adopt the GCP.

[0070] The GCP setting unit 110 includes a line segment image conversion unit 110A, a singular point extraction unit 110B, and a reliability calculation unit 110C.

[0071] The line segment image conversion unit 110A extracts line segments from the GCP setting image and converts the GCP setting image into a line segment image. The line segments are extracted, for example, by performing a differential process on the GCP setting image. The line segment image is, for example, an image in which the outlines of roads are extracted as line segments from the GCP setting image.

[0072] The singular point extraction unit 110B extracts singular points, which are objects to be set in the GCP, from the map corresponding to the shooting area of ​​the GCP setting image, among the map information acquired by the map information acquisition unit 104. A singular point is a point that is easy to distinguish compared to the shapes of other parts of the map, and is preferably a point with a unique shape. Examples of singular points include points where the direction of the road outline changes, points where road outlines intersect, and points where the width of the road changes. Here, the singular point extraction unit 110B extracts road bending points from the map. A road bending point is a point where the direction of the road outline changes, such as a corner of an intersection.

[0073] Furthermore, the singular point extracting unit 110B extracts bending points from the line segment image converted by the line segment image converting unit 110A.

[0074] The GCP setting unit 110 sets, for example, a bending point of a line segment image corresponding to a bending point of a road extracted from a map as a GCP.

[0075] The reliability calculation unit 110C calculates the reliability indicating how reliable the set GCP is as a GCP. The reliability calculation unit 110C may use road information extracted from the image for GCP setting by AI (artificial intelligence) to calculate how much two line segments that constitute the bending point that is the basis of the GCP and that are projected onto the image for GCP setting from map information match with the road edges in the image for GCP setting, and calculate a relatively higher reliability as the match increases.

[0076] The reliability calculation unit 110C may calculate a reliability indicating how reliable a candidate position that is a candidate for a GCP is as a GCP. The GCP setting unit 110 may set the candidate position as a GCP according to the reliability calculated by the reliability calculation unit 110C. The GCP setting unit 110 may display the reliability calculated by the reliability calculation unit 110C on the display device 216.

[0077] The tie point setting unit 112 sets tie points in the images. A tie point is a point at the same location that appears in two or more images. The tie point setting unit 112 normally sets tie points automatically by the following processes (1) and (2).

[0078] (1) Feature points are extracted from all target images. All target images are, for example, multiple images acquired by the captured image acquisition unit 100. For feature point extraction, an existing feature point extraction method that is robust against image rotation, enlargement, and reduction is used. Examples of existing feature point extraction methods include SURF (Speed ​​Up Robust Features) and AKAZE (Accelerated KAZE).

[0079] (2) For all combinations of two images among all target images, feature point matching is performed between the two images. Feature point matching is performed by calculating the similarity between the feature amounts corresponding to each of the two feature points using an existing method.

[0080] This provides points at the same location that appear in two or more images, that is, tie points.

[0081] In this embodiment, the tie point setting unit 112 sets, as the tie point of an image, particularly an image having an overlapping area with the GCP setting image selected by the GCP setting image selection unit 106, among the multiple images acquired by the photographed image acquisition unit 100. That is, in this embodiment, setting of a tie point means setting that a GCP set in one image is the same point when it appears in another image.

[0082] That is, the image processing device 20 does not set GCPs for all tie points that may exist in one image, but sets GCPs only for tie points that are certain to be the same as the objects in the image as the objects in the map information. This makes it possible to associate accurate coordinates, including latitude, longitude, and altitude, with the obtained tie points.

[0083] The SfM processing unit 114 performs SfM processing on the multiple images acquired by the captured image acquisition unit 100 based on the tie points set by the tie point setting unit 112, and calculates position information and attitude information of the camera 14 at the time of capturing each image. The SfM processing unit 114 may align the multiple images acquired by the captured image acquisition unit 100 with an overall map corresponding to the capturing area, or may generate a 3D model and orthoimage of the capturing area.

[0084] The input receiving unit 116 receives input from the input device 214. The display control unit 118 controls the display on the display device 216.

[0085] Image Processing Method: First Embodiment Figure 5 is a flowchart showing the steps of an image processing method according to a first embodiment. The image processing method automatically calculates the latitude, longitude, and altitude of points in a plurality of aerial images taken using a drone 12, thereby enabling the generation of highly accurate 3D models and orthoimages without the need for the installation of anti-aircraft markers or manual GCP setting. The image processing method is realized by the processor 202 executing an image processing program stored in the computer-readable medium 204. The image processing program may be provided by a computer-readable non-transitory storage medium or via the network 22.

[0086] In step S1, the image processing device 20 acquires a plurality of images.

[0087] Here, the captured image acquisition unit 100 acquires a plurality of overlapping still images of a certain capture area during a single flight of the drone 12, each of which captures a portion of the capture area. FIG. 6 is an example of a plurality of images IA acquired by the captured image acquisition unit 100. FIG. 6 shows each of the plurality of images IA arranged in accordance with the position of the capture area of ​​each image. Here, the certain capture area is divided into 35 images, five vertically and seven horizontally, but the number of images is arbitrary.

[0088] In step S2, the image processing device 20 calculates the number of features appearing in each of the multiple images acquired in step S1.

[0089] To calculate the number of features, the shooting condition acquisition unit 102 first acquires the shooting conditions for each of the multiple images. The GCP setting image selection unit 106 then calculates the shooting range of each image based on the shooting conditions for each image. The map information acquisition unit 104 then acquires a Geospatial Information Authority of Japan map corresponding to the calculated shooting range for each image. The feature number calculation unit 106B then calculates the approximate number of houses appearing in each image based on the acquired Geospatial Information Authority of Japan map.

[0090] FIG. 7 shows images included in multiple images IA and Geospatial Information Authority of Japan maps corresponding to the images. F7A in FIG. 7 shows image I1 from the multiple images IA and map M1 corresponding to image I1. There are no houses on map M1. Therefore, the number of houses shown in image I1 is calculated to be "0." F7B in FIG. 7 shows image I2 from the multiple images IA and map M2 corresponding to image I2. There are 27 houses on map M2. Therefore, the number of houses shown in image I2 is calculated to be "27."

[0091] Fig. 8 is a diagram showing multiple images IA and calculation results RC of the number of houses. Fig. 8 shows each of the multiple images IA arranged in correspondence with the position of the shooting area of ​​each image. Fig. 8 also shows the calculation results RC of each image arranged in correspondence with the position of each image. That is, although the calculation results RC shown in Fig. 8 are calculated in overlapping areas, they generally represent a two-dimensional distribution of the number of houses in the shooting areas of the multiple images IA.

[0092] In step S3, the image processing device 20 selects an image for GCP setting.

[0093] Although the user may manually select a GCP setting image, in this example, the GCP setting image selection unit 106 selects a GCP setting image from the multiple images acquired in step S1. The GCP setting image selection unit 106 uses the calculation result RC calculated in step S2 to select a GCP setting image from images that include a certain number of houses or more. Here, the GCP setting image selection unit 106 selects GCP setting images as evenly as possible from the shooting area (so as not to be biased in location). For example, the GCP setting image selection unit 106 divides the shooting area into multiple setting image selection areas and selects one GCP setting image from each setting image selection area, thereby selecting GCP setting images evenly.

[0094] 9 is a diagram for explaining the process of selecting an image for setting a GCP, which will be explained using a calculation result RC that indicates the distribution of the number of houses in the shooting areas of a plurality of images IA.

[0095] First, the area dividing unit 106A divides the shooting area of ​​the multiple images IA into 10 setting image selection areas (hereinafter referred to as areas). F9A in FIG. 9 shows an example of 10 areas A1, A2, A3, A4, ..., A9, A10 into which the entire calculation result RC corresponding to the shooting area is divided. Here, each area is a vertically elongated area, but the shape of each of the areas A1 to A10 is not limited. It is preferable that each of the areas A1 to A10 is an area into which the shooting area is equally divided.

[0096] The feature number calculation unit 106B selects, for each of the areas A1 to A10, the image with the most houses (the image with the most houses), the image with the second most houses (the image with the second most houses), and the image with the third most houses (the image with the third most houses). F9B in FIG. 9 shows the images selected by the GCP setting image selection unit 106 with hatching. In F9B, the image with the most houses is shown with the darkest hatching, the image with the second most houses is shown with the second darkest hatching, and the image with the third most houses is shown with the third darkest hatching.

[0097] Next, the GCP setting image selection unit 106 selects one GCP setting image from the three images with the first to third largest number of houses for each of the areas A1 to A10. By selecting one image from the three images for each of the ten areas, 3 to the power of 10 combinations are generated. The image processing device 20 selects the most uniform combination based on the index of "uniformity of image selection within the shooting area."

[0098] F9C in Fig. 9 shows the finally selected GCP setting image. In F9C, the selected GCP setting image is indicated by a bold frame. In this way, the GCP setting image of the area with a relatively large number of houses is selected so as not to be biased in location.

[0099] Here, ten GCP setting images are selected by dividing the photographic regions of the plurality of images IA into ten areas, but the number of GCP setting images is not limited to ten.

[0100] Next, in step S4, the image processing device 20 applies geocoding to the GCP setting image selected in step S3, and aligns the GCP setting image with the map information acquired in step S2. Here, the geocoding application unit 108 projects (superimposes) road line segments in the map information onto the GCP setting image.

[0101] Figure 10 is a diagram showing the results of applying geocoding to the GCP setting image I3. In Figure 10, the solid lines projected onto the GCP setting image I3 are road segments from the map information. Also, in Figure 10, the dashed lines projected onto the GCP setting image I3 are the outlines of the ground surfaces of houses from the map information, and the dotted lines projected onto the GCP setting image I3 are the outlines of the roofs of houses at a height equivalent to 6 meters above ground level. Here, the solid lines, dashed lines, and dotted lines are projected onto the GCP setting image I3 with an accuracy of approximately 3 to 5 meters.

[0102] In step S5, the image processing device 20 automatically extracts the GCP from the GCP setting image.

[0103] To automatically extract GCPs, the singular point extraction unit 110B first identifies map information corresponding to the GCP setting image and extracts road bending points from the map. The GCP setting image is selected from an image of an area with a relatively large number of houses. Therefore, it is assumed that the GCP setting image contains a relatively large number of roads, and also a relatively large number of road bending points. FIG. 11 is a diagram for explaining the extraction of GCPs. F11A in FIG. 11 shows a map M4 corresponding to the GCP setting image I4. As shown in F11A, a road bending point P1A has been extracted from the map M4.

[0104] Next, the singular point extraction unit 110B searches for a point at a position corresponding to the position of the bending point P1A in the GCP setting image I4. F11B in FIG. 11 is a diagram showing the search range of the GCP setting image I4. As shown in F11B, a search range R1 for the point corresponding to the bending point P1A is set in the GCP setting image I4. The search range R1 is a rectangular range approximately 5 meters in length and width, centered on the point onto which the bending point is projected by geocoding, for example.

[0105] Here, the line segment image conversion unit 110A converts the GCP setting image I4 into a line segment image. The singular point extraction unit 110B extracts bending points from this line segment image and finds the bending point corresponding to the bending point P1A extracted from the map M4. F11C in FIG. 11 is a diagram showing point P1B on the GCP setting image I4 that corresponds to bending point P1A on the map M4. The image coordinates of point P1B correspond to the map coordinates of bending point P1A. The map coordinates include latitude, longitude, and altitude.

[0106] When a plurality of GCPs are obtained from one image, the GCP setting unit 110 may select one GCP having the highest reliability calculated by the reliability calculation unit 110C from among the plurality of GCPs.

[0107] The image processing device 20 performs the processes of steps S4 and S5 on each of the 10 GCP setting images selected in step S3.

[0108] Finally, in step S6, the image processing device 20 displays a GCP confirmation / correction screen on the display device 216. The user can confirm and correct the automatically extracted GCP on the GCP confirmation / correction screen.

[0109] According to the image processing method of the first embodiment, GCPs can be set in images without the need to install anti-aircraft markers or manually set GCPs, and highly accurate 3D models and orthoimages can be generated.

[0110] In the first embodiment, images for setting a GCP are selected evenly from the photographed area, and singular points, which are objects relatively suitable as GCPs, are extracted from the setting images and set as GCPs. In the case of damage certification surveys for natural disaster damage certificates, homes are the subject of the survey, so residential areas are often the subject of photography. In this case, it is expected that all images will contain sufficient objects suitable as GCPs. Therefore, the selection of images for setting a GCP relatively becomes important.

[0111] On the other hand, when photographing a forest area or an area with many fields, the rarity of objects suitable as GCPs becomes relatively high. In this case, it is possible to first extract singular points that are objects suitable as GCPs, and then obtain an image containing those objects as an image for setting a GCP. In other words, it is preferable to prioritize setting objects suitable as GCPs as GCPs, and to distribute GCPs as evenly as possible.

[0112] [Example of GCP Confirmation / Modification Screen] Fig. 12 is a diagram showing an example of a GCP confirmation / modification screen. Here, an example is shown in which three GCPs, from the first to the third, are confirmed or modified. The remaining GCPs, from the fourth onwards, can be confirmed or modified by scrolling the screen using the input device 214 or by transitioning between screens. The number of GCPs displayed at one time and the display order of the GCPs are not particularly limited.

[0113] As shown in FIG. 12, the GCP confirmation / correction screen displays the automatically extracted results, a confirmation image, and a judgment button for each GCP.

[0114] In the upper part of the display screen shown in FIG. 12, an image I11A is displayed as the result of automatic extraction of the first GCP, and an image I11B is displayed as a confirmation image of the first GCP, both of the same size, side by side.

[0115] Image I11B is an image obtained by enlarging and cropping the GCP setting image centered on the position of the automatically extracted GCP. Image I11B may be the GCP setting image itself. Image I11A is the same image as image I11B, with a point figure FP11 and a line figure FL11 superimposed on it. The point figure FP11 is superimposed on the position of the automatically extracted GCP, and the line figure FL11 is superimposed on the position of the road outline extracted from the map information. It is preferable that the point figure FP11 and the line figure FL11 are displayed in colors that are easily visible when superimposed on the image. For example, the point figure FP11 is displayed in green, and the line figure FL11 is displayed in blue.

[0116] By comparing the displayed images I11A and I11B, the user can confirm whether the position of the point figure FP11 superimposed on image I11A is the position of the bend point of the road in image I11B, i.e., whether the position of the automatically extracted GCP is appropriate.

[0117] 12, an accept button BA11, a reject button BB11, and a modify button BC11, which are GUI (Graphical User Interface) buttons, are arranged as judgment buttons for the first GCP. The accept button BA11, the reject button BB11, and the modify button BC11 can be selected by the user using the input device 214. The selection operation is, for example, a click operation after moving the mouse cursor to a desired setting area.

[0118] When the adopt button BA11 is selected, the automatically extracted GCP is adopted. When the reject button BB11 is selected, the automatically extracted GCP is not adopted. When the edit button BC11 is selected, the position of the automatically extracted GCP can be edited.

[0119] For example, the position of the point graphic FP11 is acceptable as a GCP. Therefore, the user can adopt the first GCP by selecting the adopt button BA11. That is, the GCP is set at the position of the point graphic FP11 superimposed on the image I11A.

[0120] In addition, in the middle of the display screen shown in Figure 12, an image I12A is displayed as the automatic extraction result of the second GCP, and an image I12B is displayed as a confirmation image of the second GCP, side by side, with the same size.

[0121] Image I12B is an image obtained by enlarging and cutting out the GCP setting image with the position of the automatically extracted GCP at the center. Image I12A is the same image as image I12B, with a point figure FP12 and a line figure FL12 superimposed on it. The point figure FP12 is superimposed on the position of the automatically extracted GCP, and the line figure FL12 is superimposed on the position of the road outline in the map information.

[0122] In addition, in the middle of the display screen shown in FIG. 12, an adopt button BA12, a reject button BB12, and a correct button BC12 are arranged as judgment buttons for the second GCP.

[0123] Although the position of the point figure FP12 appears to be suitable, it is on a road hidden in the shadow of a house, making it unsuitable as a GCP. Therefore, the user can reject the second GCP by selecting the reject button BB12. In other words, no GCP is set at the position of the point figure FP12 superimposed on the image I12A.

[0124] Similarly, in the lower part of the display screen shown in Figure 12, image I13A is displayed as the automatic extraction result of the third GCP, and image I13B is displayed as a confirmation image of the third GCP, side by side, each of the same size.

[0125] Image I13B is an image obtained by enlarging and cutting out the GCP setting image with the position of the automatically extracted GCP at the center. Image I13A is the same image as image I13B, with a point figure FP13 and a line figure FL13 superimposed on it. The point figure FP13 is superimposed on the position of the automatically extracted GCP, and the line figure FL13 is superimposed on the position of the road outline in the map information.

[0126] Further, in the lower part of the display screen shown in FIG. 12, an accept button BA13, a reject button BB13, and a correct button BC13 are arranged as judgment buttons for the third GCP.

[0127] The point figure FP13 is located at a position that is offset from the road position, and is therefore undesirable as a GCP. In such a case, the user can correct the position of the point figure FP13 by selecting the correction button BC13. The user corrects the position of the point figure FP13 by specifying the correct position on the confirmation image using the input device 214. In the example shown in FIG. 12, the corrected position is specified by a circle figure FP13N in image I13B. The circle figure FP13N is preferably displayed in a color that is easily visible when superimposed on the image; for example, the circle figure FP13N is displayed in yellow.

[0128] Fig. 13 is a diagram showing another example of the GCP confirmation / correction screen. Similar to the example shown in Fig. 12, this shows an example in which three GCPs are confirmed or corrected. The GCP confirmation / correction screen displays the automatically extracted results, a confirmation image, and a judgment result for each GCP.

[0129] In the upper part of the display screen shown in FIG. 13, an image I21A as the result of automatic extraction of the first GCP and an image I21B as a confirmation image of the first GCP are displayed side by side in the same size.

[0130] Image I21B is an image obtained by enlarging and cutting out the GCP setting image with the automatically extracted GCP position at the center, and the road area extracted from the image is displayed in yellow, for example. Furthermore, image I21B has a point figure FP31 superimposed on the position of the automatically extracted GCP, and line figures FL31A and FL31B superimposed on the two line segments that form the bending point of the GCP, respectively.

[0131] The colors of the line figures FL31A and FL31B are determined depending on whether they overlap the edges of the road area. In image I21B, the line figures FL31A and FL31B overlap the edges of the road area, so they are displayed in green, for example. Accordingly, the point figure FP31 is also displayed in green.

[0132] Image I21A is an image obtained by enlarging and cutting out the GCP setting image around the position of the automatically extracted GCP, and superimposing a point figure FP21, a line figure FL21, a line figure FL21A, and a line figure FL21B on the image. The line figure FL21 is superimposed on the position of the road contour extracted from the map information. The positions at which the point figure FP21, the line figure FL21A, and the line figure FL21B are superimposed are the same as those of the point figure FP31, the line figure FL31A, and the line figure FL31B in image I21B, respectively.

[0133] The line figure FL21 is displayed in blue, for example. The colors of the point figure FP21, line figure FL21A, and line figure FL21B are determined according to the reliability of the GCP, and are the same as those of the point figure FP31, line figure FL31A, and line figure FL31B in the image I21B, respectively.

[0134] By comparing the displayed image I21A and image I21B, the user can confirm whether the position of the point figure FP21 superimposed on image I21A is appropriate as a GCP.

[0135] 13, an accept button BA21, a reject button BB21, and a correct button BC21 are arranged as judgment buttons for the first GCP. One of the accept button BA21, the reject button BB21, and the correct button BC21 is pre-selected depending on the reliability of the automatically extracted GCP. Here, the GCP of image I21A has a relatively high reliability. Therefore, the accept button BA21 is pre-selected. Pre-selected buttons are displayed in a color different from unselected buttons. Here, the unselected reject button BB21 and the correct button BC21h are displayed in white, for example, and the pre-selected accept button BA21 is displayed in red, for example.

[0136] Furthermore, a reason R21 for the pre-selection of the first GCP is displayed below the Accept button BA21, the Reject button BB21, and the Correct button BC21. Here, the reason R21 is displayed as "Both of the two line segments overlap the edges of the road area on the image, and it is highly likely that they are the correct GCP." The reason R21 allows the user to know the reason why the Accept button BA21 was pre-selected.

[0137] In the middle of the display screen shown in FIG. 13, an image I22A is displayed as the automatic extraction result of the second GCP, and an image I22B is displayed as a confirmation image of the second GCP, both of the same size, side by side.

[0138] Image I22B is an image obtained by enlarging and cropping the GCP setting image with the position of the automatically extracted GCP at the center, and the extracted road area is displayed in yellow, for example. Furthermore, image I22B has a point figure FP32 superimposed on the position of the automatically extracted GCP, and line figures FL32A and FL32B superimposed on the two line segments that form the bending point of the GCP, respectively.

[0139] In the image I22B, the line figures FL32A and FL32B do not overlap the edges of the road area, and are therefore displayed in red, for example. Accordingly, the point figure FP32 is also displayed in red.

[0140] Image I22A is an image obtained by enlarging and cutting out the GCP setting image around the position of the automatically extracted GCP, and superimposing a point figure FP22, a line figure FL22, a line figure FL22A, and a line figure FL22B on the image. The line figure FL22 is superimposed on the position of the road outline in the map information. The positions at which the point figure FP22, line figure FL22A, and line figure FL22B are superimposed are the same as those of the point figure FP31, line figure FL31A, and line figure FL31B in image I22B, respectively.

[0141] The line figure FL22 is displayed in blue, for example. The colors of the point figure FP22, line figure FL22A, and line figure FL22B are determined according to the reliability of the GCP, and are the same as those of the point figure FP32, line figure FL32A, and line figure FL32B in the image I22B, respectively.

[0142] 13, there are disposed, as judgment buttons for the second GCP, an accept button BA22, a reject button BB22, and a correct button BC22. In this example, the GCP of image I22A has a relatively low reliability, so the reject button BB22 is pre-selected. The pre-selected reject button BB22 is displayed in red, for example.

[0143] Furthermore, a reason R22 for the pre-selection of the second GCP is displayed below the accept button BA22, the reject button BB22, and the correct button BC22. Here, the reason R22 reads, "Neither of the two line segments overlaps the edge of the road area on the image, so the GCP is likely to be incorrect." The reason R22 allows the user to know the reason why the reject button BB22 was pre-selected.

[0144] In the lower part of the display screen shown in FIG. 13, an image I23A is displayed as the automatic extraction result of the third GCP, and an image I23B is displayed as a confirmation image of the third GCP, both of the same size and arranged side by side.

[0145] Image I23B is an image obtained by enlarging and cropping the GCP setting image with the automatically extracted GCP position at the center, and the extracted road area is displayed in yellow, for example. Furthermore, image I23B has a point figure FP33 superimposed on the position of the automatically extracted GCP, and line figures FL33A and FL33B superimposed on the two line segments that form the bending point of the GCP, respectively.

[0146] In image I23B, the linear figure FL33A overlaps with the edge of the road area, so the linear figure FL33A is displayed in green. On the other hand, the linear figure FL33B does not overlap with the edge of the road area, so the linear figure FL33B is displayed in red. Accordingly, the point figure FP33 is displayed in orange, for example.

[0147] Image I23A is an image obtained by enlarging and cutting out the GCP setting image around the position of the automatically extracted GCP, and superimposing a point figure FP23, a line figure FL23, a line figure FL23A, and a line figure FL23B on the image. The line figure FL23 is superimposed on the position of the road outline in the map information. The positions at which the point figure FP23, line figure FL23A, and line figure FL23B are superimposed are the same as those of the point figure FP33, line figure FL33A, and line figure FL33B in image I23B, respectively.

[0148] The line figure FL23 is displayed in blue, for example. The colors of the point figure FP23, line figure FL23A, and line figure FL23B are determined according to the reliability of the GCP, and are the same as those of the point figure FP33, line figure FL33A, and line figure FL33B in the image I23B, respectively.

[0149] 13, there are disposed, as judgment buttons for the third GCP, an accept button BA23, a reject button BB23, and a correct button BC23. In this example, the GCP of image I23A has a relatively medium reliability, so the correct button BC23 is pre-selected. The pre-selected correct button BC23 is displayed in red, for example.

[0150] Furthermore, a reason R23 for the pre-selection of the second GCP is displayed below the accept button BA23, the reject button BB23, and the correct button BC23. Here, the reason R23 reads, "One of the two line segments does not overlap with the road area on the image, and the GCP may be misaligned from its original location." The reason R23 allows the user to know the reason why the correct button BC23 was pre-selected.

[0151] Here, the image processing device 20 determines the color of the dot figure and the color of the line figure according to the reliability of the GCP, but the shape of the dot figure and the line type of the line figure may be changed according to the reliability of the GCP. The image processing device 20 may blink the dot figure and the line figure. When the user moves the cursor close to the dot figure indicating the position of the GCP, the image processing device 20 may pop up an enlarged image of the position of the GCP. The image processing device 20 may display the reliability of the GCP on the GCP confirmation / correction screen.

[0152] The image processing device 20 may display a map corresponding to the GCP setting image in accordance with the orientation and angle of the GCP setting image. In this case, the map may be a Geospatial Information Authority map or any other map.

[0153] Image Processing Method: Second Embodiment FIG. 14 is a flowchart showing the steps of an image processing method according to the second embodiment.

[0154] In step S11, the image processing device 20 acquires a plurality of overlapping images captured over a certain shooting range during a single flight of the drone 12. This process is similar to step S1 in FIG. 5 .

[0155] Next, in step S12, the image processing device 20 selects images for setting a GCP. This process is similar to steps S2 and S3 in Fig. 5. For example, 10 images for setting a GCP are selected so as not to be biased in location.

[0156] In step S13, the image processing device 20 applies geocoding to the GCP setting image selected in step S12, and aligns the GCP setting image with the map. This process is the same as step S4 in Fig. 5. Note that the alignment of the GCP setting image with the map may be performed using a means other than geocoding.

[0157] In step S14, the image processing device 20 automatically extracts GCPs from the GCP setting image and sets tie points. The process of automatically extracting GCPs is the same as step S5 in FIG.

[0158] The tie points are set by the tie point setting unit 112. The tie point setting unit 112 sets, as tie points, images that have overlapping areas with the GCP setting image selected in step S12 from among the multiple images acquired in step S11, and associates accurate coordinates with the set tie points.

[0159] 15 and 16 are diagrams for explaining the automatic extraction of GCPs and the setting of tie points. F15A in Fig. 15 shows an enlarged image 131 of the GCP setting image. F15A shows a road contour C1 extracted from map information, and bend points P31A, P31B, P31C, P31D, P31E, P31F, P31G, and P31H of the road. Here, it is assumed that bend point P31D has been automatically extracted as a GCP.

[0160] 15B shows an enlarged image I32 of an image other than the GCP setting image that contains the bending point P31D among the multiple images acquired in step S11. A tie point TP32 is set in the enlarged image I32.

[0161] 15A and 15B show enlarged images I33 and I34, respectively, of the images acquired in step S11 that include the bending point P31D other than the GCP setting image. Tie points TP33 and TP34 are set in the enlarged images I33 and I34, respectively.

[0162] Coordinates are associated with each of the tie points TP32, TP33, and TP34. However, these coordinates are calculated based on the camera position and orientation obtained from the sensor data of the camera 14 at the time of shooting, and are not necessarily accurate.

[0163] On the other hand, because the contour C1 of the road F15A has been extracted from map information, the exact coordinates (latitude, longitude, and altitude) of bend points P31A to P31H on the road contour C1 are known. Therefore, points on the road contour C1 corresponding to tie points TP32, TP33, and TP34 are found, and the exact coordinates are linked to tie points TP32, TP33, and TP34. In the example of Figure 15, the point corresponding to tie points TP32, TP33, and TP34 is bend point P31D, and the exact coordinates of bend point P31D are linked to tie points TP32, TP33, and TP34.

[0164] 16A shows an enlarged image 141 of the GCP setting image. F16A shows a road contour C2 extracted from map information, and bend points P41A, P41B, and P41C of the road. Here, it is assumed that bend point P41C has been automatically extracted as a GCP.

[0165] 16B shows an enlarged image I42 of an image other than the GCP setting image that contains the bending point P41C among the multiple images acquired in step S11. A tie point TP42 is set in the enlarged image I42.

[0166] 16A and 16B show enlarged images I43 and I44, respectively, of the images acquired in step S11 that include the bending point P41C other than the GCP setting image. Tie points TP43 and TP44 are set in the enlarged images I43 and I44, respectively.

[0167] Low-precision coordinates are associated with tie points TP42, TP43, and TP44. Therefore, points on the road contour C2 corresponding to tie points TP42, TP43, and TP44 are found, and accurate coordinates are associated with tie points TP42, TP43, and TP44. In the example of Figure 16, the point corresponding to tie points TP42, TP43, and TP44 is bend point P41C, and the accurate coordinates of bend point P41C are associated with tie points TP42, TP43, and TP44.

[0168] Finally, in step S15, the image processing device 20 outputs the camera position and attitude of each image for one flight by SfM processing. Here, the SfM processing unit 114 performs SfM processing on each image acquired in step S11 based on the tie points set in step S14, and calculates position information and attitude information of the camera 14 at the time of capturing each image.

[0169] The GCPs automatically extracted from 445 images taken in a 500-meter square area were input into commercially available software Pix4D (registered trademark), tie points were set semi-automatically, and the processing of step S15 was performed, which took approximately 44 minutes.

[0170] We evaluated the accuracy based on the obtained position information and attitude information of the camera 14, and found that the accuracy of geocoding was significantly improved from 92.7% to 99.6%. In this way, we confirmed that the accuracy was equivalent to that obtained when SfM processing was performed after GCP setting by a human.

[0171] The above "accuracy" was calculated as follows.

[0172] First, the image and the map are superimposed by geocoding and this embodiment. To align the image and the map, it is necessary to obtain the position and orientation of the camera.

[0173] In the case of geocoding, the position and orientation of the camera are determined by matching line segments extracted from the image with line segments extracted from the map. Geocoding is performed on a per-image basis.

[0174] In this embodiment, after the GCP is determined, SfM processing is performed. SfM processing generally uses several hundred images with many overlapping regions. Tie points are determined by extracting features from the images and matching the features, and the position and orientation of the camera are determined based on the tie points. By setting the GCP, i.e., the correct coordinates of latitude, longitude, and altitude, for several tie points, the position and orientation of the camera can be accurately determined.

[0175] Next, for each house in the image, the registration is judged as "successful" or "failed."

[0176] To make this determination, a correct answer box surrounding the house to be evaluated in the image is first set in advance in the image. Next, based on the results of overlaying the image and the map using geocoding, a first box around the house on the map is set in the image. Houses where the correct answer box overlaps with the first box by 80% or more are deemed to have been "successful" in geocoding.

[0177] Similarly, based on the result of superimposing the image and the map using this embodiment, a second frame around the house on the map is set on the image. A house where the correct frame and the second frame overlap by 80% or more is considered a "success" in this embodiment.

[0178] If the number of houses in all images is an integer N and the number of "successful" houses is an integer M, then M / N is the success rate, or "accuracy."

[0179] Note that when the same house appears in multiple images, it is treated as a separate house. Also, since there are houses that appear on the map but do not exist in the actual location, the number of houses in all images, N, is more accurately the number of houses that appear on the map among all the houses in all images.

[0180] According to the image processing method of the second embodiment, GCPs and tie points are automatically set on the image, so that highly accurate 3D models and orthoimages can be generated without the need to install anti-aircraft markers or manually set GCPs.

[0181] [Others] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations of the respective embodiments can be appropriately combined with each other within the scope that does not deviate from the spirit of the present invention.

[0182] DESCRIPTION OF SYMBOLS 10... Photographed image processing system 12... Drone 13... Gimbal head 14... Camera 16... Remote controller 16A... Display 20... Image processing device 22... Network 30... GPS receiver 32... Barometric pressure sensor 34... Orientation sensor 36... Gyro sensor 38... Motor 40... Processor 42... Storage device 44... Communication interface 100... Photographed image acquisition unit 102... Photographing condition acquisition unit 104... Map information acquisition unit 106... GCP setting image selection unit 106A... Area division unit 106B... Feature number calculation unit 108... Geocoding application unit 110... GCP setting unit 110A... Line segment image conversion unit 110B... Singular point extraction unit 110C... Reliability calculation unit 112... Tie point setting unit 114... SfM processing unit 116... Input reception unit 118... Display control unit 202...Processor 204...Computer readable medium 206...Communication interface 208...Input / output interface 210...Bus 214...Input device 216...Display device A1...Area A2...Area A3...Area A4...Area A5...Area A6...Area A7...Area A8...Area A9...Area A10...Area BA11...Accept button BA12...Accept button BA13...Accept button BA21...Accept button BA22...Accept button BA23...Accept button BB11...Reject button BB12...Reject button BB13...Reject button BB21...Reject button BB22...Reject button BB23...Reject button BC11...Correction button BC12...Correction button BC13...Correction button BC21...Correction button BC22...Correction button BC23...Edit button C1...Road outline C2...Road outline FL11...Linear figure FL12...Linear figure FL13...Linear figure FL21...Linear figure FL21A...Linear figure FL21B...Linear figure FL22...Linear figure FL22A...Linear figure FL22B...Linear figure FL23...Linear figure FL23A...Linear figure FL23B...Linear figure FL31A...Linear figure FL31B...Linear figure FL32A...Linear figure FL32B...Linear figure FL33A...Linear figure FL34B...Linear figure FP11...Point figure FP12...Point figure FP13...Point figure FP13N...Circle figure FP21...Point figure FP22...Point figure FP23...Point figure FP31...Point figure FP32...Point figure FP33...Point figure I1...Image I2...ImageI3...Image for GCP setting I4...Image for GCP setting I11A...Image I11B...Image I12A...Image I12B...Image I13A...Image I13B...Image I21A...Image I21B...Image I22A...Image I22B...Image I23A...Image I23B...Image I31...Enlarged image I32...Enlarged image I33...Enlarged image I34...Enlarged image I41...Enlarged image I42...Enlarged image I43...Enlarged image I44...Enlarged image IA...Multiple images M1...Map M2...Map M4...Map P1A...Bending point P1B...Point P31A...Bending point P31B...Bending point P31C...Bending point P31D...Bending point P31E...Bending point P31F...Bending point P31G...Bending point P31H...Bending point P41A...Bending point P41B...Bending point P41C...Bending point R1...Search range R21...Reason R22...Reason R23...Reason RC...Calculation result S1 to S6...Steps of image processing method S11 to S15...Steps of image processing method TP32...Tie point TP33...Tie point TP34...Tie point TP42...Tie point TP43...Tie point TP44...Tie point

Claims

1. An image processing device comprising: one or more processors; and one or more memories in which programs to be executed by the one or more processors are stored, wherein the processor executes instructions of the program to obtain a group of images in which a certain shooting area is photographed using a camera, select a setting image for setting a ground control point from the group of images, identify a map corresponding to the shooting area of ​​the setting image, identify an object for setting the ground control point from the map, search for a candidate position corresponding to the position of the object from the setting image, and set the candidate position to the ground control point.

2. The image processing device according to claim 1, wherein the processor selects a plurality of the setting images.

3. The image processing device according to claim 2, wherein the processor divides the shooting area into a plurality of setting image selection areas each including a plurality of images, calculates the number of features in each image of the group of images, and selects, for each of the plurality of setting image selection areas, an image having a relatively large number of features from among the plurality of images included in the setting image selection area as the setting image.

4. The image processing device according to claim 1, wherein the object is a bend in a road.

5. The image processing device according to claim 4, wherein the processor extracts bend points of the road from the map, converts the setting image into a line segment image, extracts bend points from the line segment image as candidate positions, and sets the bend points of the line segment image corresponding to the bend points of the road as the ground reference points.

6. The image processing device according to claim 1, wherein the processor causes a display device to display an extraction result image in which a figure is superimposed on the position of the ground reference point of the setting image.

7. The image processing device according to claim 6, wherein the processor causes the display device to display an enlarged image in which the position of the ground reference point in the setting image is enlarged.

8. The image processing device according to claim 6, wherein the processor causes the display device to display the extraction result image and a confirmation image based on the setting image side by side, and further causes the display device to display a decision button that allows the user to decide whether or not to adopt the ground reference point.

9. The image processing device according to claim 8, wherein the processor calculates a reliability indicating how reliable the set ground control point is as a ground control point, and causes the display device to display the reliability.

10. The image processing device according to claim 1, wherein the processor calculates a reliability indicating how reliable the candidate position is as a ground control point, and sets the candidate position to the ground control point according to the reliability.

11. The image processing device described in claim 1, wherein the group of images are each captured with an overlapping area with adjacent images, and the processor sets the ground control points set in the overlapping areas as tie points, and calculates the position and orientation of the camera when the camera captured each image of the group of images based on the tie points.

12. An image processing device as described in any one of claims 1 to 11, which obtains an overall map corresponding to the certain shooting area, and aligns each image of the image group with the overall map using the set ground reference points.

13. An image processing method executed by one or more processors, comprising: the processors acquiring a group of images of a certain shooting area captured using a camera; selecting a setting image for setting a ground control point from the group of images; identifying a map corresponding to the shooting area of ​​the setting image; identifying an object for setting the ground control point from the map; searching for a candidate position corresponding to the position of the object from the setting image; and setting the candidate position to the ground control point.

14. A program that enables a computer to realize the following functions: acquiring a group of images of a certain shooting area taken using a camera; selecting a setting image for setting a ground control point from the group of images; identifying a map corresponding to the shooting area of ​​the setting image; identifying an object for setting the ground control point from the map; searching for a candidate position corresponding to the position of the object from the setting image; and setting the candidate position to the ground control point.

15. A non-transitory computer-readable recording medium having the program according to claim 14 recorded thereon.

Images