Orthophoto creation method, orthophoto creation system, three-dimensional model creation method, three-dimensional model creation system, and marker used therefor

Abstract

To provide an ortho-image creation method and system as well as a three-dimensional model creation method and system which easily examine a road state on the basis of a photographed image captured from the sky even when a part of a road surface is covered by an obstacle when viewed from the sky.SOLUTION: An ortho-image creation method includes the steps of: acquiring a plurality of first photographed images by a first imaging device from the altitude higher than an obstacle; acquiring a plurality of second photographed images by a second imaging device from the altitude lower than the obstacle; acquiring three-dimensional coordinates for a first feature point that is outside a region covered by the obstacle but is included in the plurality of first photographed images; acquiring three-dimensional coordinates for a second feature point that is in the region covered by the obstacle and is included in the plurality of second photographed images; and creating a corrected ortho-image obtained by correcting the region covered by the obstacle on a road surface to the region not covered by the obstacle on the basis of the plurality of first photographed images, the plurality of second photographed images and the three-dimensional coordinates of the first feature point and the second feature point.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an ortho-image creation method for creating an ortho-image based on a captured image captured from above by an unmanned aircraft, an ortho-image creation system, a three-dimensional model creation method, a three-dimensional model creation system, and a marker used therefor.

Background Art

[0002] Conventionally, when damage such as cracks occurs on the surface of the asphalt pavement that constitutes the surface layer of a road, it is necessary to repair the road.

[0003] In order to repair a road, various investigations are carried out, such as an investigation of the road condition (for example, the crack state) at the start time of repair work and the position of planar elements including section lines such as the end of the road and the lane marking lines. Conventionally, for example, the investigation of the crack state has been carried out by visual inspection by an inspector to check the locations where cracks have occurred on the road and the amount of cracks. Instead of an inspector detecting cracks, the road condition may be investigated using a dedicated road surface property measurement vehicle (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The work of inspecting a road by an inspector to detect cracks is very cumbersome. When investigating the road condition using a dedicated road surface property measurement vehicle, it is necessary to drive the road surface property measurement vehicle, but for a narrow road, it is impossible for the road surface property measurement vehicle to drive, and it is impossible to investigate the road condition.

[0006] In order to solve the above technical problems, it is conceivable to photograph a road with an unmanned aircraft flying in the air during repair work start time, create an orthoimage of the road during repair work start time based on the captured image, and detect cracks on the road surface from the orthoimage.

[0007] However, as an obstacle, for example, if there are street trees around the road and the obstacle covers the widthwise ends of the road surface, even if the road surface is photographed by an unmanned aircraft flying in the air, it is impossible to photograph the portion covered by the obstacle on the road surface. Therefore, it is impossible to create an orthoimage including the entire area of the road surface, and it is impossible to investigate cracks in the road surface covered by obstacles. Also, when the obstacle covers the widthwise ends of the road surface, even if the road surface is photographed by an unmanned aircraft flying in the air, it is impossible to photograph the portion covered by the obstacle on the road surface, and it is impossible to conduct an investigation on the width of the road (the position of the road ends) and the position of planar elements including lane marking lines and the like. Note that when a road is photographed by an unmanned aircraft flying in the air, obstacles similar to the above-mentioned obstacles include, for example, pedestrian bridges and traffic signals arranged above the road surface.

[0008] The present invention has been made paying attention to such problems, and an orthoimage creation method, an orthoimage creation system and a marker used therefor, a three-dimensional model creation method, a three-dimensional model creation system and a marker used therefor that enable easy investigation of the road condition during repair work start time based on a captured image taken from the air even when there are obstacles covering a part of the road surface are provided.

Means for Solving the Problems

[0009] In order to solve such problems, the present invention has taken the following means.

[0016] The method for creating an orthoimage according to the present invention is applicable to a road including a region where a part of the road surface is covered by an obstacle when viewed from above. Based on a plurality of first captured images taken such that a part of the road surface becomes a region covered by the obstacle, a plurality of second captured images in which the road surface in the region covered by the obstacle is captured, first feature points included in at least two first captured images, and second feature points included in at least two second captured images, an orthoimage is created in which at least a part of the region of the road surface covered by the obstacle is replaced with a region not covered by the obstacle. The orthoimage creation system according to the present invention includes: a first captured image storage means for storing a plurality of first captured images taken such that a part of the road surface becomes a region covered by an obstacle when viewed from above for a road including a region where a part of the road surface is covered by an obstacle; a second captured image storage means for storing a plurality of second captured images in which the road surface in the region covered by the obstacle is captured; based on the plurality of first captured images stored in the first captured image storage means, the plurality of second captured images stored in the second captured image storage means, first feature points included in at least two first captured images, and second feature points included in at least two second captured images, an orthoimage creation means for creating an orthoimage in which at least a part of the region of the road surface covered by the obstacle is replaced with a region not covered by the obstacle.

[0017] Thereby, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected orthoimage in which a part of the road surface with a region covered by the obstacle is corrected to a region not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road state at the time of repair start based on the captured images taken from above.

[0018] In the method for creating an orthoimage according to the present invention, three-dimensional coordinates are respectively assigned to the first feature points and the second feature points. In the ortho-image creation system according to the present invention, the ortho-image creation means creates an ortho-image in which at least a part of the area of the road surface covered by the obstacle is replaced with an area not covered by the obstacle, based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, three-dimensional coordinates of first feature points included in at least two first captured images, and three-dimensional coordinates of second feature points included in at least two second captured images.

[0019] Thereby, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected ortho-image in which a part of the road surface where the area covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road condition at the time of repair start based on the captured images taken from above.

[0020] A method for creating an ortho-image according to the present invention creates an ortho-image in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle, based on a plurality of first captured images taken so that a part of the road surface is an area covered by the obstacle for a road having an obstacle above the road surface, a plurality of second captured images in which the road surface in the area covered by the obstacle is captured, first feature points included in at least two first captured images, and second feature points included in at least two second captured images. The ortho-image creation system according to the present invention includes: a first captured image storage means for storing a plurality of first captured images taken such that a part of the road surface is an area covered by the obstacle for a road having an obstacle above the road surface; a second captured image storage means for storing a plurality of second captured images in which the road surface in the area covered by the obstacle is captured; a plurality of first captured images stored in the first captured image storage means; a plurality of second captured images stored in the second captured image storage means; first feature points included in at least two first captured images; and second feature points included in at least two second captured images, and an ortho-image creation means for creating an ortho-image in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle.

[0021] Thereby, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected ortho-image in which the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road condition at the time of repair work start based on the captured images taken from above.

[0022] In the method for creating an ortho-image according to the present invention, 3D coordinates are respectively assigned to the first feature point and the second feature point. In the ortho-image creation system according to the present invention, the ortho-image creation means creates an ortho-image in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, 3D coordinates of the first feature points included in at least two first captured images, and 3D coordinates of the second feature points included in at least two second captured images.

[0023] Thereby, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected ortho-image in which the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road condition at the time of repair work start based on the captured images taken from above.

[0030] The method for creating a three-dimensional model according to the present invention is for a road including a region where a part of the road surface is covered by an obstacle when viewed from above. A plurality of first captured images captured such that a part of the road surface becomes a region covered by the obstacle, a plurality of second captured images in which the road surface in the region covered by the obstacle is captured, a first feature point included in at least two first captured images, and a second feature point included in at least two second captured images. Based on these, a three-dimensional model is created in which at least a part of the region of the road surface covered by the obstacle is replaced with a region not covered by the obstacle. The 3D model creation system according to the present invention, for a road including an area where a part of the road surface is covered by an obstacle when viewed from above, a first captured image storage means for storing a plurality of first captured images captured so that an area where a part of the road surface is covered by the obstacle becomes an area, a second captured image storage means for storing a plurality of second captured images in which the road surface in the area covered by the obstacle is captured, a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, a first feature point included in at least two first captured images, and a second feature point included in at least two second captured images, based on at least a part of the area of the road surface covered by the obstacle is replaced with an area not covered by the obstacle. A 3D model creating means for creating a 3D model Create It is characterized by comprising means.

[0031] As a result, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected three-dimensional model in which a region of the road surface covered by the obstacle is corrected to a region not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to conduct an investigation regarding the width of the road (the position of the edge of the road) at the time of repair work start and the position of planar elements including lane marking lines and other demarcation lines based on the captured images taken from above.

[0032] In the method for creating a three-dimensional model according to the present invention, the first feature point and the second feature point are each assigned three-dimensional coordinates. In the three-dimensional model creation system according to the present invention, the three-dimensional model creation means creates a three-dimensional model in which at least a part of the region of the road surface covered by the obstacle is replaced with a region not covered by the obstacle, based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, three-dimensional coordinates of first feature points included in at least two first captured images, and three-dimensional coordinates of second feature points included in at least two second captured images.

[0033] As a result, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected three-dimensional model in which a region of the road surface covered by the obstacle is corrected to a region not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to conduct an investigation regarding the width of the road (the position of the edge of the road) at the time of repair work start and the position of planar elements including lane marking lines and other demarcation lines based on the captured images taken from above.

[0034] The method for creating a three-dimensional model according to the present invention creates a three-dimensional model in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle, based on a plurality of first captured images taken so that a part of the road surface is a region covered by the obstacle for a road with an obstacle above the road surface, a plurality of second captured images in which the road surface in the region covered by the obstacle is captured, first feature points included in at least two first captured images, and second feature points included in at least two second captured images. The 3D model creation system according to the present invention includes: a first captured image storage means for storing a plurality of first captured images taken so that a part of the road surface is an area covered by the obstacle for a road having an obstacle above the road surface; a second captured image storage means for storing a plurality of second captured images in which the road surface in the area covered by the obstacle is captured; a plurality of first captured images stored in the first captured image storage means; a plurality of second captured images stored in the second captured image storage means; first feature points included in at least two first captured images; and second feature points included in at least two second captured images, and 3D model creation means for creating a 3D model in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle.

[0035] Thereby, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected 3D model in which the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to conduct an investigation on the width of the road (the position of the edge of the road) at the time of repair start and the position of planar elements including lane marking lines and other demarcation lines based on the captured images taken from above.

[0036] In the method for creating a 3D model according to the present invention, 3D coordinates are respectively assigned to the first feature points and the second feature points. In the 3D model creation system according to the present invention, the 3D model creation means creates a 3D model in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, 3D coordinates of first feature points included in at least two first captured images, and 3D coordinates of second feature points included in at least two second captured images.

[0037] This makes it possible to create a corrected three-dimensional model in which, even when part of the road surface is covered by an obstacle when viewed from above, the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering part of the road surface, it is possible to conduct an investigation regarding the width of the road (the position of the edge of the road) at the time of repair start and the position of planar elements including lane marking lines and other demarcation lines based on the captured image taken from above.

Effect of the Invention

[0038] As described above, according to the present invention, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road state at the time of repair work start based on the captured images taken from above.

Brief Description of the Drawings

[0039]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Mode for Carrying Out the Invention

[0040] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, there are street trees that become obstacles around the road, and a case where a part of the road surface is covered by the street trees when photographing the periphery of the road from above will be described.

[0041] An orthoimage creation system 1 according to an embodiment of the present invention includes a total station 2 installed at a known point (for example, a reference point), a UAV 3 (Unmanned Aerial Vehicle) which is an imaging device, and an orthoimage creation device 10 to which the total station 2 and the UAV 3 are wirelessly connected.

[0042] The total station 2 emits ranging light toward each point on the road surface, receives the reflected light reflected at each point, and based on the number of oscillations of the light wave from emission to reception, acquires the three-dimensional coordinates of each point relative to a known point, and supplies the three-dimensional coordinates to the ortho-image creation system 10. In the present embodiment, the total station 2 is used to acquire the three-dimensional coordinates of a plurality of air markers 6.

[0043] The UAV 3 has a photographing device, photographs the road surface from above, acquires photographing data, and supplies the photographing data to the ortho-image creation device 10. The UAV 3 can photograph while flying at an altitude higher than an obstacle and can also photograph while flying at an altitude lower than the obstacle.

[0044] The ortho-image creation device 10 is composed of, for example, a microcomputer, and includes a CPU, a ROM storing a program for controlling the operation of the ortho-image creation device 10, and a RAM for temporarily storing data and the like used when executing the above program.

[0045] As shown in FIG. 1, the ortho-image creation device 10 has a coordinate storage unit 11, a photographed image storage unit 12, an ortho-image creation unit 13, a shape detection unit 14, and a display control unit 15. The coordinate storage unit 11 has a first coordinate storage unit 11a and a second coordinate storage unit 11b. The photographed image storage unit 12 has a first photographed image storage unit 12a and a second photographed image storage unit 12b. The ortho-image creation unit 13 has a first ortho-image creation unit 13a and a second ortho-image creation unit 13b. Further, the ortho-image creation device 10 has a display unit 5 such as a display screen.

[0046] The coordinate storage unit 11 stores the three-dimensional coordinates of feature points such as a plurality of air targets 6 obtained separately by the total station 2. When the UAV 3 flies at an altitude higher than an obstacle covering a part of the road surface, the first coordinate storage unit 11a stores the three-dimensional coordinates of the air target 6 (first feature point) installed outside the area covered by the obstacle. When the UAV 3 flies at an altitude lower than an obstacle covering a part of the road surface, the second coordinate storage unit 11b stores the three-dimensional coordinates of the air target 6 (second feature point) installed in the area covered by the obstacle.

[0047] The photographed image storage unit 12 stores a plurality of images obtained by photographing the road from above by the UAV 3 flying at a substantially constant altitude above the road. The first photographed image storage unit 12a stores a plurality of first photographed images obtained by photographing a road with a part of the road surface covered by an obstacle when viewed from above at an altitude higher than the obstacle. The second photographed image storage unit 12b stores a plurality of second photographed images obtained by photographing the area covered by the obstacle at an altitude lower than the obstacle.

[0048] In this embodiment, the UAV 3 flies at an altitude of 20 meters or less above the ground during shooting, for example, at an altitude of 3 to 20 meters, preferably at an altitude of 3 to 15 meters.

[0049] Obstacles around the road that cover a part of the road surface when viewed from above are, for example, about 3 to 10 meters high. Therefore, when the UAV 3 shoots at an altitude higher than the obstacle, it flies at an altitude of, for example, 10 to 20 meters above the ground, and when shooting at an altitude lower than the obstacle, it flies at an altitude of, for example, 3 to 10 meters above the ground.

[0050] When photographing a road with a UAV3 flying at an altitude higher than an obstacle, as shown in FIG. 2, for example, a plurality of air-to-air markers 6 are installed as a plurality of feature points near both ends of the road. The plurality of air-to-air markers 6 are installed in an area not covered by an obstacle and are installed, for example, at intervals of 5 to 15 meters along the end of the road (in the longitudinal direction of the road). The plurality of air-to-air markers 6 are installed in consideration of creating an orthoimage by connecting a plurality of photographed images taken from above. The air-to-air marker 6 is a feature point supplied with three-dimensional coordinates and is used as an evaluation point. In addition, when creating an orthoimage by connecting a plurality of photographed images, in addition to the air-to-air marker 6, feature points included in the plurality of photographed images and not supplied with three-dimensional coordinates may be used.

[0051] The air-to-air marker 6 is a square plate-like member as shown in FIG. 3. The air-to-air marker 6 is provided with a pattern that makes its center position clear. The air-to-air marker 6 has an adhesive layer formed on its back surface and is in the form of a sticker with a backing paper attached so as to cover the adhesive layer. By removing the backing paper and attaching it to the road, it can be easily fixed to the installation location. Therefore, when using the air-to-air marker 6, the backing paper covering the adhesive layer is removed, and the back surface of the air-to-air marker 6 is attached to the road surface for use. The air-to-air marker 6 of the present embodiment is, for example, a square shape of 9 cm × 9 cm, but the type, shape, size, pattern, etc. of the air-to-air marker 6 are not limited thereto.

[0052] A plurality of images photographed by the UAV3 flying at an altitude higher than an obstacle are photographed such that each air-to-air marker 6 is included in at least two photographed images, as shown in FIG. 4. Therefore, at least one common air-to-air marker 6 is photographed in two adjacent photographed images. In addition, in FIG. 4, the case where all the photographed images include the air-to-air marker 6 is illustrated, but the plurality of images photographed by the UAV3 may be photographed such that either the air-to-air marker 6 or feature points other than the air-to-air marker 6 are included in at least two photographed images.

[0053] When the aerial marker 6 is photographed by the UAV 3 flying at an altitude higher than the obstacle covering a part of the road surface, as shown in FIG. 2, it is installed outside the area covered by the obstacle. On the other hand, when the UAV 3 flying at an altitude lower than the obstacle covering a part of the road surface is photographed, as shown in FIG. 8, it is installed on a part of the road surface or the obstacle in the area covered by the obstacle.

[0054] The ortho-image creation unit 13 creates an ortho-image based on the three-dimensional coordinates of the aerial marker 6 stored in the coordinate storage unit 11 and a plurality of photographed images stored in the photographed image storage unit 12. Specifically, the ortho-image creation unit 13 performs SfM (Structure from Motion) analysis or the like on the data of a plurality of photographed images, and connects two adjacent photographed images based on the common aerial marker 6 photographed thereon to create a three-dimensional model based on three-dimensional data (point cloud data), and creates an ortho-image based on the three-dimensional model.

[0055] Specifically, the first ortho-image creation unit 13a creates a temporary ortho-image based on a plurality of first photographed images stored in the first image storage unit 12a and the three-dimensional coordinates of the aerial marker 6 installed outside the area covered by the obstacle stored in the first coordinate storage unit 11a. In the temporary ortho-image, a part of the road surface is covered by the obstacle.

[0056] The second ortho-image creation unit 13b creates a corrected ortho-image based on the temporary ortho-image created by the first ortho-image creation unit 13a and the shape (3D shape) of the road surface and the part other than the road surface (including obstacles) in the area covered by the obstacles detected by the shape detection unit 14. Specifically, the second ortho-image creation unit 13b designates a predetermined range including the area covered by the obstacles in the temporary ortho-image, and replaces the predetermined range with the shape (mesh data) detected by the shape detection unit 14, thereby creating a corrected ortho-image in which the area covered by the obstacles is corrected to an area not covered by the obstacles. In the method of creating the corrected ortho-image of the present embodiment, with the temporary ortho-image as the background, a correction ortho-image (mesh data → ortho-image) is created on the background, and the correction ortho-image is integrated with the temporary ortho-image to create a corrected ortho-image (finally, it is the ortho-images that are integrated). Therefore, the integration method for creating the corrected ortho-image of the present embodiment corresponds to the integration method 3 (described later) for creating the corrected ortho-image.

[0057] The shape detection unit 14 creates mesh data corresponding to the shape (3D shape) of the road surface and the part other than the road surface (including obstacles) in the area covered by the obstacles based on the plurality of second captured images stored in the second capture storage unit 12b and the 3D coordinates of the air target sign 6 installed in the area covered by the obstacles and stored in the second coordinate storage unit 11b.

[0058] The display control unit 15 displays the ortho-image created by the ortho-image creation unit 13 on the display unit 5.

[0059] (Creation of Ortho-Image) A method for creating an ortho-image in the ortho-image creation device 10 will be described with reference to FIG. 5. In the present embodiment, when there are obstacles around the road and a part of the road surface is covered by the obstacles when viewed from above, a method for creating an ortho-image of the road where the road surface is not covered by the obstacles will be described.

[0060] In step S1 (first coordinate acquisition step), for a plurality of predetermined positions, that is, predetermined positions where a plurality of air-facing markers 6 are installed, around a repair location where road repair is to be carried out, a total station 2 acquires three-dimensional coordinates, that is, a planar position (latitude, longitude) and an elevation (height).

[0061] In step S2 (first photographing step), as shown in FIGS. 6(a) and 6(b), a road is photographed from above by a UAV 3 flying at an altitude higher than an obstacle T around the road. FIG. 6(b) shows an example of an orbit along which the UAV 3 flies. When the photographing is carried out, a plurality of air-facing markers 6 are previously installed at a plurality of predetermined positions surveyed in step S1. Therefore, a plurality of photographed images are taken such that each air-facing marker 6 is included in at least two photographed images.

[0062] In step S3 (first orthoimage creation step), a provisional orthoimage is created based on the three-dimensional coordinates acquired in step S1 and the plurality of photographed images taken in step S2. As shown in FIG. 7, a part of the road surface in the provisional orthoimage is covered by the obstacle T.

[0063] In step S4, in a region where a part of the road surface is covered by the obstacle T, as shown in FIG. 8, a plurality of air-facing markers 6 are installed on the road surface and on the side surface of the obstacle T, which is a part other than the road surface. The region covered by the obstacle T is a region hidden by the obstacle T when viewed from above. In FIG. 8, the air-facing markers 6 are installed on the side surface of the obstacle T on the road side.

[0064] In step S5 (second coordinate acquisition step), for a plurality of predetermined positions where a plurality of air-facing markers 6 are installed in a region covered by the obstacle T, a total station 2 acquires three-dimensional coordinates, that is, a planar position (latitude, longitude) and an elevation (height).

[0065] In step S6 (second photographing step), as shown in FIGS. 9(a) and 9(b), the road surface and the obstacle T are photographed by the UAV3 flying at an altitude lower than that of the obstacle T around the road. FIG. 9(b) shows an example of the trajectory along which the UAV3 flies. Therefore, a plurality of photographed images are taken such that each of the plurality of air-to-air markers 6 is included in at least two photographed images.

[0066] In step S7 (shape detection step), based on the plurality of second photographed images stored in the second photographing storage unit 12b and the three-dimensional coordinates of the air-to-air marker 6 installed in the area covered by the obstacle T stored in the coordinate storage unit 11b, mesh data corresponding to the shape of the road surface and the portion other than the road surface (including the obstacle T) in the area covered by the obstacle T is created.

[0067] In step S8 (second ortho-image creation step), based on the temporary ortho-image created in step S3 and the mesh data corresponding to the shape of the road surface and the portion other than the road surface (including the obstacle T) in the area covered by the obstacle T created in step S7, as shown in FIG. 10, a corrected ortho-image in which the road surface is not covered by the obstacle T is created. In the temporary ortho-image of FIG. 7, a part of the road surface is covered by the obstacle T and the road surface in that area is hidden, whereas in the corrected ortho-image of FIG. 10, the obstacle T around the road is cut off near its lower end, and the road surface that was hidden in FIG. 7 is visible.

[0068] In step S9 (display step), the corrected ortho-image in which the road surface is not covered by the obstacle T is displayed on the display unit 5. In the present embodiment, the ground pixel size of the ortho-image is 5 millimeters or less.

[0069] The difference in the three-dimensional areas of the temporary ortho-image (FIG. 7) and the corrected ortho-image (FIG. 10) will be described with reference to FIG. 11.

[0070] For example, in a provisional ortho-image created based on a captured image taken by a UAV 3 flying at an altitude higher than an obstacle T around a road, since there is no captured image of the area covered by the obstacle T, as shown in Fig. 11(a), the area below the obstacle T does not take a proper shape. Even if the obstacle T is cut out near its lower end in Fig. 11(a), as shown in Fig. 11(b), it is impossible to supplement the information of the end portion of the road surface (the end portion on the side close to the obstacle T) because such information is lacking.

[0071] On the other hand, when using the shape (mesh data) of the road surface and the portions other than the road surface (including the obstacle T) in the area covered by the obstacle T, as shown in Fig. 11(c), the area below the obstacle T is also three-dimensionalized. Although not all of the area below the obstacle T is three-dimensionalized, at least the portion near the ground is three-dimensionalized. Therefore, when the obstacle T is cut out near its lower end in Fig. 11(c), as shown in Fig. 11(d), it is possible to correct the area not covered by the obstacle T by supplementing the end portion of the road surface (the end portion on the side close to the obstacle T).

[0072] (Road survey method using ortho-image) The ortho-image created by the ortho-image creation device 10 as described above is used for various surveys conducted when road repairs are carried out.

[0073] For example, using the ortho-image created by the ortho-image creation device 10, surveys are conducted on (1) the crack state of the road surface, (2) the positions of planar elements around the road including the locations where repairs are to be made, (3) surveys for repairing the area around manholes, (4) surveys on the distance between two specified points on the road surface, (5) surveys on the area of a specified range on the road surface, etc.

[0074] FIG. 12 and FIG. 13 are enlarged views of a road surface with cracks formed thereon. As described above, in the ortho-image created by the ortho-image creation device 10 of the present embodiment, for example, cracks formed on the road surface can be clearly discriminated. Therefore, based on the ortho-image displayed on the display unit 5, as an investigation of the crack state of the road surface, the location where cracks are formed on the road surface including the repair location where road repair is to be performed, and the crack rate and patching rate of that part are investigated.

[0075] As described above, when a part of the road surface is covered by an obstacle when viewed from above, the obstacle is removed, and an ortho-image of a road whose road surface is not covered by the obstacle is created. In the process until the ortho-image is created, by performing SfM (Structure from Motion) analysis or the like on the data of a plurality of captured images, two adjacent captured images are connected based on the common air target marker 6 captured therein, and a three-dimensional model (a three-dimensional model based on three-dimensional data (point cloud data)) is created.

[0076] (Creation of three-dimensional model) A three-dimensional model creation system and a three-dimensional model creation method for creating a three-dimensional model of the periphery of a road will be described with reference to FIGS. 14 and 15. In the present embodiment, when there are obstacles around the road and a part of the road surface is covered by the obstacles when viewed from above, a method for creating a three-dimensional model of a road whose road surface is not covered by the obstacles will be described.

[0077] As shown in FIG. 14, the three-dimensional model creation system 101 according to the embodiment of the present invention includes a total station 2 installed at a known point (for example, a reference point), a UAV 3 (Unmanned Aerial Vehicle) as a photographing device, and an ortho-image creation device 110 to which the total station 2 and the UAV 3 are wirelessly connected.

[0078] That is, in the ortho-image creation system shown in FIG. 1, when the following contents are changed, as shown in FIG. 14, a 3D model creation system that creates a 3D model by the 3D model creation method of the present embodiment is shown. · Change the ortho-image creation unit 13 to a 3D model creation unit 113 that creates a 3D model. · Change the first ortho-image creation unit 13a to a first creation unit 113a that creates a temporary 3D model. · Change the second ortho-image creation unit 13b to a second creation unit 113b that creates a corrected 3D model. Also, in the ortho-image creation method shown in FIG. 5, when the following contents are changed, as shown in FIG. 15, the 3D model creation method of the present embodiment is shown. · Step S3: Change the creation of the temporary ortho-image to the creation of a temporary 3D model. · Step S8: Change the creation of the corrected ortho-image to the creation of a corrected 3D model. · Step S9: Change the display of the corrected ortho-image to the display of the corrected 3D model.

[0079] As described above, the corrected ortho-image in which the road surface is not covered by the obstacle T is created based on the temporary ortho-image in which a part of the road surface is covered by the obstacle T and the mesh data corresponding to the shape of the road surface and the part other than the road surface (including the obstacle T) in the area covered by the obstacle T. Similarly, the corrected 3D model in which the road surface is not covered by the obstacle T is created based on the temporary 3D model in which a part of the road surface is covered by the obstacle T and the point cloud data corresponding to the shape of the road surface and the part other than the road surface (including the obstacle T) in the area covered by the obstacle T. Therefore, the detailed description is omitted. Therefore, in the 3D model creation system 101 shown in FIG. 14, the shape detection unit 14 creates point cloud data corresponding to the shape (3D shape) of the road surface and the part other than the road surface (including the obstacle) in the area covered by the obstacle based on the plurality of second captured images stored in the second captured image storage unit 12b and the 3D coordinates of the air target marker 6 installed in the area covered by the obstacle stored in the second coordinate storage unit 11b.

[0080] In the method of creating the corrected 3D model of the present embodiment, with the provisional 3D model as the background, a 3D model (mesh data) for correction is created on top of that background, and the 3D model for correction is integrated into the provisional 3D model to create the corrected 3D model. Therefore, the integration method for creating the corrected 3D model of the present embodiment corresponds to (Integration Method 1 for creating the corrected 3D model), which will be described later.

[0081] Note that, for example, by rotating the 3D model of the road surface on the display screen, it is possible to change the 3D model of the road surface seen from various directions. In the following description, the case where the 3D model of the road surface is seen from various directions will be described.

[0082] Figures 16 and 17 are 3D models showing a state where a part of the road surface is covered by obstacles such as street trees, traffic lights, and road signs when photographing the road surface from above. Therefore, in Figures 16 and 17, a part of the road surface is in a state of being hidden by obstacles, and that part of the road surface is not visible.

[0083] In contrast, Figures 18 and 19 are 3D models showing a state where obstacles such as street trees, traffic lights, and road signs covering a part of the road surface are removed when photographing the road surface from above. Therefore, in Figures 18 and 19, a part of the road surface that was in a state of being hidden by obstacles and not visible in Figures 16 and 17 is now visible.

[0084] Figure 20 is a 3D model created when photographing the road surface from a shooting direction different from Figures 16 and 17. In Figure 20, most of the road surface is in a state of being hidden by street trees. In contrast, Figure 21 is a 3D model showing a state where the street trees are removed from Figure 20. In Figure 21, most of the road surface that was in a state of being hidden by street trees and not visible in Figure 20 is now visible.

[0085] Therefore, in FIGS. 18, 19, and 21, since the entire road surface is visible, it is possible to conduct an investigation on the width of the road (the position of the road edge) and the position of planar elements including lane marking lines and other demarcation lines.

[0086] The ortho-image creation method of the present embodiment includes: a first imaging step of imaging a road with a part of the road surface covered by an obstacle from a height higher than the obstacle by a first imaging device (UAV3) from above to obtain a plurality of first imaging images; a second imaging step of imaging the area covered by the obstacle from a height lower than the obstacle by the second imaging device (UAV3) to obtain a plurality of second imaging images; a first coordinate acquisition step of acquiring three-dimensional coordinates of first feature points that are outside the area covered by the obstacle and included in at least two of the plurality of first imaging images; a second coordinate acquisition step of acquiring three-dimensional coordinates of second feature points that are in the area covered by the obstacle and included in at least two of the plurality of second imaging images; and an ortho-image creation step of creating a corrected ortho-image in which at least a part of the area of the road surface covered by the obstacle is corrected to the area not covered by the obstacle based on the plurality of first imaging images taken in the first imaging step, the plurality of second imaging images taken in the second imaging step, the three-dimensional coordinates of the first feature points obtained in the first coordinate acquisition step, and the three-dimensional coordinates of the second feature points obtained in the second coordinate acquisition step.

[0087] The ortho-image creation system 1 of the present embodiment includes a first captured image storage unit 12a that stores a plurality of first captured images captured by a first imaging device (UAV3) from an altitude higher than that of an obstacle for a road whose part of the road surface is covered by an obstacle when viewed from above, a second captured image storage unit 12b that stores a plurality of second captured images captured by a second imaging device (UAV3) from an altitude lower than that of the obstacle for an area covered by the obstacle, a first coordinate storage unit 11a that stores three-dimensional coordinates of first feature points that are outside the area covered by the obstacle and are included in at least two of the plurality of first captured images, a second coordinate storage unit 11b that stores three-dimensional coordinates of second feature points that are in the area covered by the obstacle and are included in at least two of the plurality of second captured images, a plurality of first captured images stored in the first captured image storage unit 12a, a plurality of second captured images stored in the second captured image storage unit 12b, three-dimensional coordinates of the first feature points stored in the first coordinate storage unit 11a, and three-dimensional coordinates of the second feature points stored in the second coordinate storage unit 11b, and an ortho-image creation unit 13 that creates a corrected ortho-image in which at least a part of the area covered by the obstacle on the road surface is corrected to an area not covered by the obstacle.

[0088] Accordingly, in the ortho-image creation method and the ortho-image creation system 1 of the present embodiment, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected ortho-image in which a part of the road surface in the area covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road condition at the time of repair start based on the captured images taken from above.

[0089] In the ortho-image creation method of the present embodiment, the ortho-image creation step includes a first ortho-image creation step of creating a temporary ortho-image in which a part of the road surface is covered by an obstacle based on a plurality of first captured images captured in the first capturing step and the three-dimensional coordinates of the first feature points acquired in the first coordinate acquisition step; a shape detection step of detecting the shape of at least a part of the road surface and a part other than the road surface in the area covered by the obstacle based on a plurality of second captured images captured in the second capturing step and the three-dimensional coordinates of the second feature points acquired in the second coordinate acquisition step; and a second ortho-image creation step of creating a corrected ortho-image in which at least a part of the area covered by the obstacle in the temporary ortho-image created in the first ortho-image creation step is corrected to an area not covered by the obstacle.

[0090] In the ortho-image creation system 1 of the present embodiment, the ortho-image creation unit 13 includes a first ortho-image creation unit 13a that creates a temporary ortho-image in which a part of the road surface is covered by an obstacle based on a plurality of first captured images stored in the first image storage unit 12a and the three-dimensional coordinates of the first feature points stored in the first coordinate storage unit 11a; a shape detection unit 14 that detects the shape of at least a part of the road surface and a part other than the road surface in the area covered by the obstacle based on a plurality of second captured images stored in the second captured image storage 12b and the three-dimensional coordinates of the second feature points stored in the second coordinate storage unit 11b; and a second ortho-image creation unit 13b that creates a corrected ortho-image in which at least a part of the area covered by the obstacle in the temporary ortho-image created by the first ortho-image creation unit 13a is corrected to an area not covered by the obstacle.

[0091] Thereby, in the ortho-image creation method and the ortho-image creation system 1 according to the present invention, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected ortho-image in which a part of the road surface in the temporary ortho-image is corrected from an area covered by the obstacle to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to easily investigate the road condition at the time of repair work start based on the captured images taken from above.

[0092] In the ortho-image creation method of this embodiment, the first imaging device is the UAV3 flying at an altitude higher than the obstacle, and the second imaging device is the UAV3 flying at an altitude lower than the obstacle.

[0093] In the ortho-image creation system 1 of this embodiment, the first imaging device is the UAV3 flying at an altitude higher than the obstacle, and the second imaging device is the UAV3 flying at an altitude lower than the obstacle is.

[0094] Thereby, in the ortho-image creation method and the ortho-image creation system 1 of this embodiment, by the UAV3, a plurality of first imaging images of the road surface not covered by the obstacle taken from an altitude higher than the obstacle, and a plurality of second imaging images of the road surface covered by the obstacle taken from an altitude lower than the obstacle can be easily obtained.

[0095] In the ortho-image creation method of this embodiment, in the second imaging step, a plurality of air-to-air markers 6 are installed on the road surface and the obstacle in the area covered by the obstacle, and the plurality of air-to-air markers 6 are imaged so as to be included in at least two second imaging images as the second feature points.

[0096] In the ortho-image creation system 1 of this embodiment, the plurality of second imaging images stored in the second imaging image storage unit 12b are imaged such that a plurality of air-to-air markers 6 installed on the road surface and the obstacle in the area covered by the obstacle are included in at least two second imaging images as the second feature points.

[0097] Thereby, in the ortho-image creation method and the ortho-image creation system 1 of this embodiment, it is possible to accurately grasp the shape of the road surface and the obstacle in the area covered by the obstacle from the plurality of second imaging images taken from an altitude lower than the obstacle.

[0098] The air-to-air marker 6 of the present embodiment is a marker used in the ortho-image creation method of the present embodiment, and is in the form of a sticker with an adhesive layer formed on its back surface. Thereby, with the air-to-air marker 6 of the present embodiment, it is possible to easily fix the marker to the installation location.

[0099] The air-to-air marker 6 of the present embodiment is a marker used in the ortho-image creation system 1 of the present embodiment, and is in the form of a sticker with an adhesive layer formed on its back surface. Thereby, with the air-to-air marker 6 of the present embodiment, it is possible to easily fix the marker to the installation location.

[0100] The three-dimensional model creation method of the present embodiment includes a first photographing step of photographing a road with a part of the road surface covered by an obstacle from above with a first photographing device (UAV3) at an altitude higher than the obstacle to obtain a plurality of first photographed images; a second photographing step of photographing the area covered by the obstacle from a lower altitude than the obstacle with a second photographing device (UAV3) to obtain a plurality of second photographed images; a first coordinate acquisition step of acquiring three-dimensional coordinates of first feature points that are outside the area covered by the obstacle and are included in at least two of the plurality of first photographed images; a second coordinate acquisition step of acquiring three-dimensional coordinates of second feature points that are in the area covered by the obstacle and are included in at least two of the plurality of second photographed images; a three-dimensional model creation step of creating a corrected three-dimensional model in which at least a part of the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle based on the plurality of first photographed images photographed in the first photographing step, the plurality of second photographed images photographed in the second photographing step, the three-dimensional coordinates of the first feature points acquired in the first coordinate acquisition step, and the three-dimensional coordinates of the second feature points acquired in the second coordinate acquisition step.

[0101] The 3D model creation system 101 of this embodiment includes a first captured image storage unit 12a that stores a plurality of first captured images captured by a first imaging device (UAV3) from an altitude higher than an obstacle of a road where a part of the road surface is covered by the obstacle when viewed from above, a second captured image storage unit 12b that stores a plurality of second captured images captured by a second imaging device (UAV3) from an altitude lower than the obstacle of the area covered by the obstacle, a first coordinate storage unit 11a that stores 3D coordinates of first feature points that are outside the area covered by the obstacle and are included in at least two of the plurality of first captured images, a second coordinate storage unit 11b that stores 3D coordinates of second feature points that are in the area covered by the obstacle and are included in at least two of the plurality of second captured images, a plurality of first captured images stored in the first captured image storage unit 12a, a plurality of second captured images stored in the second captured image storage unit 12b, 3D coordinates of the first feature points stored in the first coordinate storage unit 11a, and 3D coordinates of the second feature points stored in the second coordinate storage unit 11b, and a 3D model creation unit 113 that creates a corrected 3D model in which at least a part of the area of the road surface covered by the obstacle is corrected to an area not covered by the obstacle.

[0102] Accordingly, in the 3D model creation method and the 3D model creation system 101 of this embodiment, even when a part of the road surface is covered by an obstacle when viewed from above, it is possible to create a corrected 3D model in which a part of the road surface in the area covered by the obstacle is corrected to an area not covered by the obstacle. Therefore, even when there is an obstacle covering a part of the road surface, it is possible to conduct an investigation regarding the width of the road (the position of the edge of the road) at the time of repair start work and the position of planar elements including lane marking lines and other demarcation lines based on the captured images taken from above.

[0103] The 3D model creation method of this embodiment includes: a 3D model creation step including a first creation step of creating a temporary 3D model in which a part of the road surface is covered by an obstacle based on a plurality of first captured images captured in the first capturing step and the 3D coordinates of the first feature points obtained in the first coordinate acquisition step; a shape detection step of detecting at least a part of the shape of the road surface and a part other than the road surface in the area covered by the obstacle based on a plurality of second captured images captured in the second capturing step and the 3D coordinates of the second feature points obtained in the second coordinate acquisition step; and a second 3D model creation step of creating a corrected 3D model by correcting at least a part of the area covered by the obstacle in the temporary 3D model created in the first creation step to an area not covered by the obstacle.

[0104] The 3D model creation system 101 of this embodiment includes: a 3D model creation unit 113 including a first creation unit 113a that creates a temporary 3D model in which a part of the road surface is covered by an obstacle based on a plurality of first captured images stored in the first captured image storage unit 12a and the 3D coordinates of the first feature points stored in the first coordinate storage unit 11a; a shape detection unit 14 that detects at least a part of the shape of the road surface and a part other than the road surface in the area covered by the obstacle based on a plurality of second captured images stored in the second captured image storage unit 12b and the 3D coordinates of the second feature points stored in the second coordinate storage unit 11b; and a second creation unit 113b that creates a corrected 3D model by correcting at least a part of the area covered by the obstacle in the temporary 3D model created by the first creation unit 113a to an area not covered by the obstacle.

[0105] As a result, in the three-dimensional model creation method and the three-dimensional model creation system 101 according to the present invention, even when a part of the road surface is covered by an obstacle when viewed from above, in the temporary three-dimensional model, a corrected three-dimensional model in which a part of the road surface in the area covered by the obstacle is corrected to an area not covered by the obstacle can be created. Therefore, even when there is an obstacle covering a part of the road surface, based on the captured image taken from above, it is possible to investigate the width of the road (the position of the edge of the road) at the time of repair start work and the position of planar elements including lane marking lines and other demarcation lines.

[0106] In the three-dimensional model creation method of the present embodiment, the first imaging device is a UAV3 flying at an altitude higher than that of the obstacle, and the second imaging device is a UAV3 flying at an altitude lower than that of the obstacle.

[0107] In the three-dimensional model creation system 101 of the present embodiment, the first imaging device is a UAV3 flying at an altitude higher than that of the obstacle, and the second imaging device is a UAV3 flying at an altitude lower than that of the obstacle.

[0108] As a result, in the three-dimensional model creation method and the three-dimensional model creation system 101 of the present embodiment, by the UAV3, it is possible to easily obtain a plurality of first captured images of the road surface not covered by the obstacle taken from an altitude higher than that of the obstacle and a plurality of second captured images of the road surface covered by the obstacle taken from an altitude lower than that of the obstacle.

[0109] In the three-dimensional model creation method of the present embodiment, in the second imaging step, a plurality of air-to-air markers 6 are installed on the road surface and the obstacle in the area covered by the obstacle, and the plurality of air-to-air markers 6 are imaged so as to be included in at least two second captured images as the second feature points.

[0110] In the 3D model creation system 101 of the present embodiment, the plurality of second captured images stored in the second captured image storage unit 12b are captured such that the road surface and the plurality of air-facing signs 6 installed on the obstacles are included in at least two second captured images as second feature points in the area covered by the obstacles.

[0111] Accordingly, in the 3D model creation method and the 3D model creation system 101 of the present embodiment, it is possible to accurately grasp the shape of the road surface and the obstacles in the area covered by the obstacles from the plurality of second captured images captured from a height lower than that of the obstacles.

[0112] The air-facing sign 6 of the present embodiment is a sign used in the 3D model creation method of the present embodiment, and is in the form of a sticker with an adhesive layer formed on its back surface. Accordingly, with the air-facing sign 6 of the present embodiment, it is possible to easily fix the sign to the installation location.

[0113] The air-facing sign 6 of the present embodiment is a sign used in the 3D model creation system 101 of the present embodiment, and is in the form of a sticker with an adhesive layer formed on its back surface. Accordingly, with the air-facing sign 6 of the present embodiment, it is possible to easily fix the sign to the installation location.

[0114] As described above, the embodiments of the present invention have been described. However, the specific configuration of each part is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention.

[0115] In the above embodiment, a 3D model and an orthoimage are created based on the captured images taken from above the road by the UAV 3 flying at a substantially constant altitude at an altitude of 20 meters or less above the ground. However, the present invention includes those that create a 3D model and an orthoimage based on the captured images taken from above the road by the UAV 3 flying at an altitude exceeding 20 meters above the ground.

[0116] In the above embodiment, the three-dimensional coordinates of the air target sign 6 are obtained by the total station 2. However, the three-dimensional coordinates of the air target sign 6 may be obtained by a GNSS (Global Navigation Satellite System), which is a positioning system using satellites such as GPS. The three-dimensional coordinates of the air target sign 6 may be obtained by scanning with the 3D scanner 4. The three-dimensional coordinates of the air target sign 6 installed around the road may be obtained by scanning with the 3D scanner 4. Also, for a predetermined position where a plurality of air target signs 6 are installed, the three-dimensional coordinates are obtained by the total station 2. However, if the three-dimensional coordinates of the predetermined position have already been obtained, those three-dimensional coordinates may be obtained.

[0117] In the above embodiment, the air target sign 6 is square and has a pattern that makes the center position used as the evaluation point clear. However, the shape of the air target sign 6 and the center position of the air target sign 6 are not limited to the case where they are feature points. The air target sign 6 has a pattern in which a position other than its center position is specified, and a position other than its center position may be used as an evaluation point.

[0118] Also, in the above embodiment, the plate-shaped air target sign 6 is installed on the road surface or the like. However, instead of using the plate-shaped air target sign 6, a pattern similar to the air target sign 6 may be formed on the road surface or the like with an arbitrary material such as paint. For example, on the asphalt surface of the road, a pattern having the same shape as the white portion in the air target sign 6 of FIG. 3 may be sprayed with paint of a color different from the asphalt surface to form a pattern similar to the air target sign 6. When the air target sign is formed on the road surface with an arbitrary material such as paint, the type, shape, size, pattern, etc. of the air target sign are arbitrary. The above content is the same when, for example, fixing an air target sign to an obstacle such as a street tree.

[0119] Also, in the above-described embodiment, the case where the air-facing sign 6 formed in a seal shape with the backing paper attached so as to cover the adhesive layer on the back surface is used, and the air-facing sign 6 is fixed to the installation location by removing the backing paper and attaching it to the installation location has been described. However, the method of fixing the air-facing sign 6 to the installation location is arbitrary. For example, an air-facing sign with holes formed may be fixed to the installation location by a fixing tool (e.g., an anchor pin, etc.). In that case, an attachment hole may be formed in the installation location such as the road surface, and the air-facing sign may be fixed to the attachment hole by a fixing tool. Further, for example, an air-facing sign may be fixed to the installation location by embedding a rod-shaped sign post having the air-facing sign formed at the tip thereof. In that case, by embedding the sign post in the ground surface or the like, the air-facing sign formed at the tip of the sign post is disposed on the ground surface. The above content is the same, for example, when fixing an air-facing sign to an obstacle such as a street tree.

[0120] In the above-described embodiment, the method of creating the ortho-image has been described. However, after acquiring the three-dimensional coordinates of the air-facing sign 6, it is possible to take a photographed image, and after taking the photographed image, it is possible to acquire the three-dimensional coordinates of the air-facing sign 6.

[0121] In the above-described embodiment, when a part of the road surface is covered by an obstacle when viewed from above, a corrected three-dimensional model and a corrected ortho-image in which all of the area covered by the obstacle are corrected to the area not covered by the obstacle are created. However, a corrected three-dimensional model and a corrected ortho-image in which a part of the area covered by the obstacle (e.g., at least a part of the road surface and a part other than the road surface) are corrected to the area not covered by the obstacle may be created. Therefore, a corrected three-dimensional model and a corrected ortho-image may be created in which the road surface of the area covered by the obstacle is corrected to the area not covered by the obstacle, and the part other than the road surface of the area covered by the obstacle is not corrected to the area not covered by the obstacle.

[0122] In the above-described embodiment, as the first imaging device and the second imaging device, an unmanned aircraft (including an imaging device) flying at an altitude higher than an obstacle and an unmanned aircraft (including an imaging device) flying at an altitude lower than an obstacle are used, but the present invention is not limited thereto. In the present invention, the types of the first imaging device and the second imaging device are arbitrary. For example, as the first imaging device and the second imaging device, at least one of the cases of imaging a road from an altitude higher than an obstacle and imaging a road from an altitude lower than an obstacle may be performed by a model aircraft (including an imaging device) flying at an altitude higher than an obstacle or a model aircraft (including an imaging device) flying at an altitude lower than an obstacle. In the present invention, an unmanned aircraft is an aircraft, a rotary-wing aircraft, a dirigible, etc. that cannot be occupied by a person and can fly by remote control or automatic control, and is, for example, a drone (multicopter), a radio-controlled aircraft, etc. Further, a model aircraft is, for example, a multicopter, a radio-controlled aircraft, etc., and has a total weight of less than 200 grams, which is the sum of the weight of the airframe main body and the weight of the battery. In the above-described embodiment, when a road is imaged by an unmanned aircraft flying in the sky, obstacles covering a part of the road surface are not limited to street trees around the road. The present invention is applicable, for example, when obstacles such as a pedestrian bridge or a traffic signal arranged above the road surface cover a part of the road surface.

[0123] In the above-described embodiment, as the first imaging device and the second imaging device, an unmanned aircraft (including an imaging device) flying at an altitude higher than an obstacle and an unmanned aircraft (including an imaging device) flying at an altitude lower than an obstacle are used, but the present invention is not limited thereto. For example, as the first imaging device and the second imaging device, at least one of the cases of imaging a road from an altitude higher than an obstacle and imaging a road from an altitude lower than an obstacle may be performed by a camera arranged at an altitude higher than an obstacle or a camera arranged at an altitude lower than an obstacle.

[0124] For example, as shown in FIG. 22, the orthoimage creation system according to a modification of the present invention may include a total station 2 installed at a known point (for example, a reference point), a UAV 3 (Unmanned Aerial Vehicle) as a photographing device, a camera 103 as a photographing device, and an orthoimage creation device 10 to which the total station 2, the UAV 3, and the camera 103 are wirelessly connected.

[0125] In this modification, when photographing a road where a part of the road surface is covered by an obstacle from a height higher than the obstacle when viewed from above, it is photographed by the UAV 3, and when photographing the area covered by the obstacle from a height lower than the obstacle, it is photographed by the camera 103. That is, an inspector in the area where a part of the road surface is covered by the obstacle T operates the camera 103 to photograph the area (the road surface and the obstacle T) including a plurality of air-facing markers 6 installed on the side surface of the obstacle T, which is a part other than the road surface, as shown in FIG. 23. In that case, while moving, the inspector takes a plurality of photographed images such that each air-facing marker 6 is included in at least two photographed images.

[0126] In the above-described embodiment, when creating an ortho-image based on the three-dimensional coordinates of the air target sign 6 and a plurality of captured images, SfM (Structure from Motion) analysis software is used. The process by SfM analysis software includes: (1) a point cloud generation process that constitutes a collection of a plurality of point cloud data corresponding to a plurality of positions; (2) a TIN process (a meshing process that meshes the collection of point cloud data formed in the point cloud generation process) that converts the point cloud data into a three-dimensional TIN model (irregular triangular network), which is a collection of triangular planes connected by vertices; and (3) an ortho-image creation process that creates an ortho-image based on the three-dimensional TIN model converted in the TIN process. The ortho-images (provisional ortho-image and corrected ortho-image) of the present invention are obtained by performing the process up to the ortho-image creation process with SfM analysis software, and the three-dimensional models (provisional three-dimensional model and corrected three-dimensional model) of the present invention are obtained by performing the process up to the point cloud generation process with SfM analysis software, or by performing the process up to the TIN process after the point cloud generation process with SfM analysis software. That is, a three-dimensional model (point cloud data) is created by performing the process up to the point cloud generation process with SfM analysis software, and a three-dimensional model (TIN-converted point cloud data) is created by performing the process up to the TIN process after the point cloud generation process with SfM analysis software.

[0127] (Method for creating corrected ortho-image) In the present invention, SfM analysis is performed on a plurality of captured images taken from an altitude higher than that of the obstacle and the three-dimensional coordinates of the air target sign 6 included therein, and SfM analysis is performed on a plurality of captured images taken from an altitude lower than that of the obstacle and the three-dimensional coordinates of the air target sign 6 included therein, and a corrected ortho-image is created by integrating these data.

[0128] There is the following method for creating a corrected ortho-image by integrating the results of SfM analysis performed on a plurality of captured images taken from an altitude higher than an obstacle and the three-dimensional coordinates of the air target marker 6 included therein, and the results of SfM analysis performed on a plurality of captured images taken from an altitude lower than the obstacle and the three-dimensional coordinates of the air target marker 6 included therein. In the above embodiment, the case of using the following (Integration Method 3 for creating a corrected ortho-image) was described, but the integration method for creating a corrected ortho-image is arbitrary.

[0129] (Integration Method 1 for creating a corrected ortho-image) Integrate the results of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude lower than the obstacle and the three-dimensional coordinates of the air target marker 6 included therein with the results of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude higher than the obstacle and the three-dimensional coordinates of the air target marker 6 included therein. Then, perform a TIN step and an ortho-image creation step on the integrated data to create a corrected ortho-image.

[0130] (Integration Method 2 for creating a corrected ortho-image) Integrate the results of performing SfM analysis up to the TIN step after the point cloud generation step on a plurality of captured images taken from an altitude lower than the obstacle and the three-dimensional coordinates of the air target marker 6 included therein with the results of performing SfM analysis up to the TIN step after the point cloud generation step on a plurality of captured images taken from an altitude higher than the obstacle and the three-dimensional coordinates of the air target marker 6 included therein. Then, perform an ortho-image creation step on the integrated data to create a corrected ortho-image.

[0131] (Integration Method 3 for creating a corrected ortho-image) After performing SfM analysis on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target 6 included therein through a point cloud generation process and a TIN process, and then performing up to an ortho-image creation process, the result is integrated with the one obtained by performing SfM analysis on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target 6 included therein through a point cloud generation process and a TIN process, and then performing up to an ortho-image creation process, to create a corrected ortho-image.

[0132] (Integration method 4 for creating a corrected ortho-image) For the combined data of a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target 6 included therein, and a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target 6 included therein, a corrected ortho-image is created by performing an SfM analysis, a point cloud generation process, a TIN process, and an ortho-image creation process.

[0133] (Method for creating a corrected 3D model) In the present invention, an SfM analysis is performed on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target 6 included therein, and an SfM analysis is also performed on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target 6 included therein, and a corrected 3D model is created by integrating those data.

[0134] As a method for creating a corrected 3D model by integrating the result of performing SfM analysis on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target 6 included therein, and the result of performing SfM analysis on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target 6 included therein, there is the following method. In the above embodiment, the case of using the following (Integration method 1 for creating a corrected 3D model) was described, but the integration method for creating a corrected 3D model is arbitrary.

[0135] (Integration method 1 for creating a corrected 3D model) Integrate the result of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target marker 6 included therein with the result of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target marker 6 included therein, and create a corrected 3D model.

[0136] (Integration method 2 for creating a corrected 3D model) Integrate the result of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target marker 6 included therein with the result of performing SfM analysis up to the point cloud generation step on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target marker 6 included therein. Then, perform a TIN process on the integrated data to create a corrected 3D model.

[0137] (Integration method 3 for creating a corrected 3D model) Integrate the result of performing SfM analysis up to the TIN step after the point cloud generation step on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target marker 6 included therein with the result of performing SfM analysis up to the TIN step after the point cloud generation step on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target marker 6 included therein to create a corrected 3D model.

[0138] (Integration method 4 for creating a corrected 3D model) Create a corrected 3D model by performing SfM analysis up to the point cloud generation step on the combined data of a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target marker 6 included therein and a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target marker 6 included therein.

[0139] (Integration method 5 for creating a corrected 3D model) Perform SfM analysis on the integrated data of a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target signs 6 included therein, and the integrated data of a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target signs 6 included therein, from the point cloud generation step to the TIN step to create a corrected 3D model.

[0140] In the above integration method for creating the corrected ortho-image or the integration method for creating the corrected 3D model, the data obtained by performing SfM analysis on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target signs 6 included therein is integrated with the data obtained by performing SfM analysis on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target signs 6 included therein. However, the data obtained by performing SfM analysis on a plurality of captured images taken from an altitude higher than the obstacle and the 3D coordinates of the air target signs 6 included therein may be integrated with the data obtained by performing SfM analysis on a plurality of captured images taken from an altitude lower than the obstacle and the 3D coordinates of the air target signs 6 included therein.

Explanation of Signs

[0141] 1 Ortho-image creation system 2 Total station 3 UAV (Unmanned Aerial Vehicle) 4 3D Scanner (3D Scanning Device) 5 Display unit 6 Air target sign 10 Ortho-image creation device 11 Coordinate storage unit 11a First coordinate storage unit (first coordinate storage means) 11b Second coordinate storage unit (second coordinate storage means) 12 Captured image storage unit 12a First captured image storage unit (first captured image storage means) 12b Second captured image storage unit (second captured image storage means) 13 Ortho-image creation section 13a First ortho-image creation section (first ortho-image creation means) 13b Second ortho-image creation section (second ortho-image creation means) 14 Shape detection unit (shape detection means) 15 Display control unit 101 3D model creation system 103 Camera 113 3D model creation unit 113a First 3D model creation unit (first 3D model means) 113b Second 3D model creation unit (second 3D model means)

Claims

1. For a road including an area where a part of the road surface is covered by an obstacle when viewed from above, a plurality of first captured images captured so that a part of the road surface becomes an area covered by the obstacle, a plurality of second captured images of the road surface in the area covered by the obstacle, at least two first feature points included in at least two first captured images, and at least two second feature points included in at least two second captured images, based on these, a method for creating an ortho-image, characterized in that at least a part of the area of the road surface covered by the obstacle is replaced with an area not covered by the obstacle.

2. The method for creating an ortho-image according to Claim 1, characterized in that three-dimensional coordinates are respectively assigned to the first feature point and the second feature point.

3. For a road with an obstacle above the road surface, a plurality of first captured images captured so that a part of the road surface becomes an area covered by the obstacle, a plurality of second captured images of the road surface in the area covered by the obstacle, at least two first feature points included in at least two first captured images, and at least two second feature points included in at least two second captured images, based on these, a method for creating an ortho-image, characterized in that at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle.

4. The method for creating an ortho-image according to Claim 3, characterized in that three-dimensional coordinates are respectively assigned to the first feature point and the second feature point.

5. For a road including an area where a part of the road surface is covered by an obstacle when viewed from above, a first captured image storage means for storing a plurality of first captured images captured so that a part of the road surface becomes an area covered by the obstacle, a second captured image storage means for storing a plurality of second captured images of the road surface in the area covered by the obstacle. An ortho-image creation system, comprising: ortho-image creation means for creating an ortho-image in which at least a part of a region covered by the obstacle on the road surface is replaced with a region not covered by the obstacle, based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, first feature points included in at least two first captured images, and second feature points included in at least two second captured images. **Claim 6** The ortho-image creation means creates an ortho-image in which at least a part of a region covered by the obstacle on the road surface is replaced with a region not covered by the obstacle, based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, three-dimensional coordinates of first feature points included in at least two first captured images, and three-dimensional coordinates of second feature points included in at least two second captured images. The ortho-image creation system according to claim 5, characterized in that. **Claim 7** A first captured image storage means for storing a plurality of first captured images taken such that a part of the road surface where there is an obstacle above the road surface becomes a region covered by the obstacle, a second captured image storage means for storing a plurality of second captured images of the road surface in the region covered by the obstacle, an ortho-image creation system, comprising: ortho-image creation means for creating an ortho-image in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle, based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, first feature points included in at least two first captured images, and second feature points included in at least two second captured images. **Claim 8** The ortho-image creation means Based on the plurality of first captured images stored in the first captured image storage means, the plurality of second captured images stored in the second captured image storage means, the three-dimensional coordinates of the first feature points included in at least two first captured images, and the three-dimensional coordinates of the second feature points included in at least two second captured images, creating an ortho-image in which at least a part of the obstacle above the road surface is replaced by the road surface below the obstacle. The ortho-image creation system according to claim 7, characterized in that.

9. For a road including a region where a part of the road surface is covered by an obstacle when viewed from above, a plurality of first captured images taken so that the region where a part of the road surface is covered by the obstacle becomes the region, and a plurality of second captured images of the road surface in the region covered by the obstacle, and based on the first feature points included in at least two first captured images and the second feature points included in at least two second captured images, creating a three-dimensional model in which at least a part of the region of the road surface covered by the obstacle is replaced by a region not covered by the obstacle. A method for creating a three-dimensional model, characterized in that.

10. The method for creating a three-dimensional model according to claim 9, characterized in that three-dimensional coordinates are respectively assigned to the first feature point and the second feature point.

11. For a road with an obstacle above the road surface, a plurality of first captured images taken so that a part of the road surface becomes a region covered by the obstacle, a plurality of second captured images of the road surface in the region covered by the obstacle, and based on the first feature points included in at least two first captured images and the second feature points included in at least two second captured images, creating a three-dimensional model in which at least a part of the obstacle above the road surface is replaced by the road surface below the obstacle. A method for creating a three-dimensional model, characterized in that.

12. The method for creating a three-dimensional model according to claim 11, characterized in that three-dimensional coordinates are respectively assigned to the first feature point and the second feature point.

13. For a road including an area where a part of the road surface is covered by an obstacle when viewed from above, a first captured image storage means for storing a plurality of first captured images captured such that a part of the road surface becomes an area covered by the obstacle; a second captured image storage means for storing a plurality of second captured images in which the road surface in the area covered by the obstacle is captured; Based on the plurality of first captured images stored in the first captured image storage means, the plurality of second captured images stored in the second captured image storage means, the first feature points included in at least two first captured images, and the second feature points included in at least two second captured images, a three-dimensional model creation means for creating a three-dimensional model in which at least a part of the area of the road surface covered by the obstacle is replaced with an area not covered by the obstacle. A three-dimensional model creation system characterized by comprising.

14. The three-dimensional model creation means Based on the plurality of first captured images stored in the first captured image storage means, the plurality of second captured images stored in the second captured image storage means, the three-dimensional coordinates of the first feature points included in at least two first captured images, and the three-dimensional coordinates of the second feature points included in at least two second captured images, the three-dimensional model creation system according to claim 13, characterized in that a three-dimensional model in which at least a part of the area of the road surface covered by the obstacle is replaced with an area not covered by the obstacle is created.

15. A first captured image storage means for storing a plurality of first captured images captured such that a part of the road surface becomes an area covered by an obstacle for a road having an obstacle above the road surface; a second captured image storage means for storing a plurality of second captured images in which the road surface in the area covered by the obstacle is captured; Based on the plurality of first captured images stored in the first captured image storage means, the plurality of second captured images stored in the second captured image storage means, the first feature points included in at least two first captured images, and the second feature points included in at least two second captured images, a three-dimensional model creation means for creating a three-dimensional model in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle. A three-dimensional model creation system characterized by comprising.

16. The three-dimensional model creation means Based on a plurality of first captured images stored in the first captured image storage means, a plurality of second captured images stored in the second captured image storage means, three-dimensional coordinates of first feature points included in at least two first captured images, and three-dimensional coordinates of second feature points included in at least two second captured images, creating a three-dimensional model in which at least a part of the obstacle above the road surface is replaced with the road surface below the obstacle, the three-dimensional model creation system according to claim 15.

Images