3D map data creation program

Abstract

To create three-dimensional map data for an unmanned aircraft flight.SOLUTION: A three-dimensional map data creation program for an unmanned aircraft flight causes a computer to perform steps of: acquiring a surrounding image photographed by a movable body mounting camera, a photographing direction of the camera, positional coordinate information of the movable body when photographing, and information on distance to an object reflected in the surrounding image of the movable body (S201); determining a type of the object in the surrounding image and a position in the image of the object using a learned mathematical model (S202); calculating a positional coordinate of the object whose type is determined based on the positional coordinate information of the movable body and the photographing direction and the distance information of the camera, and calculating a size of the object based on a size in the surrounding image of the object whose type is determined and the distance information (S203); and generating three-dimensional map data in which the type of the object, the positional coordinate of the object, and the size of the object are combined for the object whose type is determined (S204).SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a 3D map data generating program. [Background technology]

[0002] Patent Document 1 discloses three-dimensional map data that expresses a flight space in which an aircraft flies using a plurality of three-dimensional spaces. This three-dimensional map data is said to include position data that specifies the position that each three-dimensional space occupies in the flight space, and property data that is assigned to each three-dimensional space and indicates the property of the three-dimensional space for the aircraft. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-180359 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to the Ministry of Economy, Trade and Industry's "Roadmap for the Air Industrial Revolution 2019," flights of unmanned aerial vehicles (UAVs) beyond visual line of sight (Level 4) in populated areas are scheduled to take place from fiscal year 2022. There is a demand for three-dimensional maps to support the flights of such unmanned aerial vehicles.

[0005] The present invention aims to create a relief map database for unmanned aerial vehicle flight. [Means for solving the problem]

[0006] In order to achieve the above object, a three-dimensional map data creation program for creating three-dimensional map data for flying an unmanned aerial vehicle is provided. This program includes an acquisition step of acquiring an image of the surroundings of a moving body captured by a camera mounted on the moving body, the imaging direction of the camera, position coordinate information of the moving body at the time of the image capture, and distance information between the moving body and an object captured in the image of the surroundings of the moving body; And Using a mathematical model that has been trained as training data, the type of object appearing in the surrounding image is determined. Similar the computer is caused to execute a determination step of determining a position of the object whose type has been determined in the determination step based on position coordinate information of the moving body, the imaging direction of the camera, and the distance information, a calculation step of calculating position coordinates of the object whose type has been determined in the determination step based on the size of the object in the surrounding image and the distance information, a creation step of creating three-dimensional map data for each object whose type has been determined in the determination step, combining the type of the object, the position coordinates of the object, and the size of the object, and a step of storing the created group of three-dimensional map data as a three-dimensional map database for reuse. Effect of the Invention

[0007] According to the present invention, a relief map database for flying an unmanned aerial vehicle can be created. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing a system for creating a 3D map database. [Diagram 2] FIG. 2 is an explanatory diagram showing the configuration of a moving body and a center. [Diagram 3] FIG. 2 is an explanatory diagram showing an example of a still image. [Figure 4] FIG. 2 is an explanatory diagram illustrating an example of a functional configuration of an edge computer. [Diagram 5]13 is a flowchart showing a startup process performed by an edge computer. [Figure 6] 13 is a flowchart showing a 3D map database creation process performed by an edge computer. [Figure 7] FIG. 13 is an explanatory diagram showing an example of a viewer display based on a 3D map database; [Figure 8] FIG. 2 is an explanatory diagram illustrating an example of a functional configuration of a server. [Figure 9] 13 is a flowchart showing a 3D map database creation process performed by the server. [Figure 10] 13 is a flowchart showing a learning process performed by the server. [Figure 11] 13 is a flowchart showing another learning process performed by the server. [Figure 12] FIG. 13 is an explanatory diagram showing an example of a viewer display of a horizontal cross section based on a 3D map database; [Figure 13] FIG. 2 is an explanatory diagram illustrating an example of a hardware configuration of a server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present invention will be described based on the illustrated embodiment, however, the present invention is not limited to the embodiment described below.

[0010] 1 and 2 show a system S for creating a 3D map database used for flying an unmanned aerial vehicle. This system S includes a moving body 1 and a center (or a ground station) 2. In the system S, the moving body 1 for creating the 3D map database is not limited to an unmanned aerial vehicle 1a, but may be any movable body such as a vehicle 1b.

[0011] The moving object 1 is equipped with a camera 11, a GNSS receiver 12, an inertial measurement unit (IMU) 13, a LiDAR device 14, and an edge computer 15. The edge computer 15 is connected to a recording medium 16 such as an SD card.

[0012] The camera 11 captures video of the surroundings of the mobile object 1 while the mobile object 1 is moving or stopped. Fig. 3 shows an example of a still image IM1 obtained from the video of the camera 11. The GNSS receiver 12 receives electromagnetic waves emitted by GNSS satellites such as GPS satellites. These electromagnetic waves include data for calculating the position coordinates of the mobile object 1.

[0013] The inertial measurement unit 13 measures the angular velocity of the moving body 1 about three mutually orthogonal axes and the acceleration of the moving body 1. The LiDAR device 14 uses a technology called LiDAR (Light Detection and Ranging, or Laser Imaging Detection and Ranging) to obtain an image capturing the surroundings of the moving body 1 and measures the distance (depth) between the moving body 1 and an object shown in the image.

[0014] The video captured by the camera 11, the electromagnetic wave data received by the GNSS receiver 12, the measurement data of the inertial measurement unit 13, and the measurement data of the LiDAR device 14 are sent to the edge computer 15.

[0015] The center 2 includes a server 21 connected to the edge computer 15 in the mobile object 1 via a communication network, and an updated map database 22 connected to the server. The updated map database 22 stores the latest map data at that time.

[0016] As shown in FIG. 4, the edge computer 15 includes a data acquisition unit 151, an object determination unit 152, a position and size calculation unit 153, and a 3D modeler (a modeler within the edge computer 15) 154.

[0017] The data acquisition unit 151 acquires data transmitted from the camera 11, the GNSS receiver 12, the inertial measurement unit 13, and the LiDAR device 14. In addition, the data acquisition unit 151 is configured to be able to acquire various types of data from the server 21.

[0018] The object determination unit 152 determines an object appearing in a still image obtained from a moving image captured by the camera 11, such as the still image IM1, acquired by the data acquisition unit 151, and the position of the object within the still image. To perform this determination, the object determination unit 152 uses a mathematical model such as a CNN (convolutional neural network). This mathematical model is also referred to as a first mathematical model.

[0019] The input of this mathematical model is a still image, such as still image IM1, obtained from a video captured by camera 11. The output of this mathematical model is the type of object appearing in the input still image and the position of the object within the still image.

[0020] This mathematical model is trained in advance using the input learning image, the target object shown in the learning image, and the label (type of object) and position in the image of the object as training data. In other words, the object indicated in the learning image, the label of the object, and the position in the image are annotations for the learning image. Examples of objects that can be annotated include buildings, pedestrian bridges, utility poles, pillars, roadside trees, electric wires, manholes, and mailboxes. Objects that must be avoided in order for the drone to fly safely can be the subject of annotation. The learning flow will be described later.

[0021] Using the learned mathematical model, the object determination unit 152 determines the type of object appearing in a still image obtained from a video captured by the camera 11 and the position of the object in the still image. If the learning image has eight objects as annotations, namely, a building, a pedestrian bridge, a utility pole, a pillar, a roadside tree, an electric wire, a manhole, and a mailbox, the object determination unit 152 determines the type of the object appearing in the still image as one of these eight objects.

[0022] The object determination unit 152 can specify the shape of each object whose type has been determined. For each object type, the corresponding shape is defined as a table. This table is shown as a geometric shape model table 163 in the recording medium 16 in FIG. 4. The size of an object obtained by depth determination can be determined by applying a numerical value based on the shape derived from this table. In this table, the object type and the corresponding shape are defined, for example, as follows: Building: A combination of 1 or more cubes Utility pole: Cylinder Support: Combination of cylinder and thin rectangular prism (plate) Street trees: Cone

[0023] The position and size calculation unit 153 calculates the position coordinates (latitude, longitude) of the moving body 1 based on the data of the GNSS electromagnetic wave acquired by the data acquisition unit 151. The position and size calculation unit 153 also calculates the imaging direction of the camera 11 based on the measurement data of the inertial measurement unit 13 acquired by the data acquisition unit 151. Furthermore, the position and size calculation unit 153 uses data from the LiDAR device 14 to calculate the distance between each object determined by the object determination unit 152 and the moving body 1. The position and size calculation unit 153 calculates the position coordinates (latitude, longitude) of each object using the position coordinates (latitude, longitude) of the moving body 1 obtained in this manner, the imaging direction of the camera 11, and the distance between each object determined by the object determination unit 152 and the moving body 1.

[0024] The position and size calculation unit 153 also calculates the actual size of each object determined by the object determination unit 152. This calculation is performed based on the depth data acquired by the data acquisition unit 151 from the LiDAR device 14, the size of the object in the still image, and the shape derived from the geometric model table 163.

[0025] In this manner, position and calculation unit 153 calculates the position coordinates (latitude, longitude) and actual size of each object determined by object determination unit 152.

[0026] The 3D modeler 154 creates a relief map database (also called a simplified relief map database) using the determination result of the object determination unit 152 and the calculation result of the position and size calculation unit 153. This relief map database has a relief map data record created for each object determined by the object determination unit 152. Each relief map data record has an identification number of the corresponding object, the type of the object, the position coordinates (latitude, longitude) of the object, and the actual size of the object. Furthermore, the 3D modeler 154 records the created simplified relief map data record in the recording medium 16 as a simplified relief map database 161.

[0027] 5 shows the flow of the startup process performed by the edge computer 15 in the moving object 1. In step S101, the server 21 receives a designation of the target area of ​​the moving object 1 (the area in which the moving object 1 is about to fly or drive) from the user of the system S. An example of the target area data is "a 3 km square area around Nihonbashi 3-chome, Chuo-ku, Tokyo." Furthermore, in the same step, the data acquisition unit 151 acquires the target area data of the moving object 1 from the server 21.

[0028] In step S102, the data acquisition unit 151 acquires the latest map data in the latest map database 22 through the server 21. The initial latest map data contains only topographical information and highly accurately surveyed coordinate control point information obtained from outside the system S, such as a map company, or only topographical information, building information, and coordinate information obtained from outside. Topographical information is information about topography, such as hills and valleys, obtained from a digital elevation model or the like. Coordinate control point information is information about the latitude and longitude of utility poles, manholes, etc., with guaranteed accuracy. Building information is information about the latitude and longitude of the building and the size of the building.

[0029] In the automatic flight of the unmanned aerial vehicle 1a, the route is usually set at the center 2 (ground station) before flight. The software of this ground station (or the software of the control system) usually has a two-dimensional map, and the flight route is defined on the map. The unmanned aerial vehicle 1a flies according to this route while obtaining its own position coordinates using GNSS. Alternatively, the flight route may be defined based on the latest map data read in step S102. The latest map data acquired in step S102 can be used when the mobile object 1 moves.

[0030] Through processing described later, detailed 3D map data is created from the simplified 3D map data obtained by the mobile object 1 in the system S. This detailed 3D map data is then added to the latest map database 22.

[0031] In step S103, the data acquisition unit 151 acquires parameters of a mathematical model used by the object determination unit 152 when determining an object from the server 21. Note that, regarding step S102 and step S103, one of the steps may be performed before the other step, or both steps may be performed simultaneously.

[0032] In step S104, the data acquisition unit 151 writes into the recording medium 16 the latest map data acquired in step S102 and the parameters acquired in step S103.

[0033] Fig. 6 shows a flow of a process for creating a simplified 3D map database performed by the edge computer 15 in the mobile object 1. This creation process is performed following the startup process shown in Fig. 5. In step S201, the data acquisition unit 151 acquires data. Specifically, the data acquisition unit 151 acquires still images from the video captured by the camera 11, and acquires data of received electromagnetic waves from the GNSS receiver 12. The data acquisition unit 151 further acquires measurement data from the inertial measurement unit 13, and acquires measurement data from the LiDAR device 14 using LiDAR technology.

[0034] In step S202, the object determination unit 152 uses the above-mentioned mathematical model to determine the type of object appearing in the still image acquired in step S201 and the position of the object in the image. The details of this determination are as described above.

[0035] In step S203, position and size calculation section 153 calculates the position coordinates (latitude, longitude) and actual size of each object determined by object determination section 152. Details of this calculation are as described above.

[0036] In step S204, the 3D modeler 154 creates simplified 3D map data.

[0037] In addition, the server 21 generates a distance estimation model (distance estimation AI) that estimates the distance between the object and the moving body 1 based on the type of the object and the size of the object in the still image. This distance estimation model is also a mathematical model such as a convolutional neural network. The input of the distance estimation model is the surrounding image of the moving body, and the output is the type of object shown in the surrounding image, the position of the object in the image, and the distance to the object. In a moving body that does not have a LiDAR device 14 mounted thereon, the distance is estimated using this distance estimation model. Details will be described later. In addition, the distance estimation model is also called a second mathematical model.

[0038] In step S205, the 3D modeler 154 records the simplified 3D map data as a simplified 3D map database 161 in the recording medium 16. This post-arrangement data is used for collision avoidance of the drones as well as for parameter calculation of a distance estimation model (distance estimation AI) in the server 21. In the same step, the data acquisition unit 151 further records the video data captured by the camera 11 as a video database 162 in the recording medium 16.

[0039] In step S206, the 3D modeler 154 sends the simplified 3D map database to the server 21, and the data acquisition unit 151 sends the video database to the server 21. Note that step S205 is not essential. For example, if the capacity of the communication network between the mobile object 1 and the center 2 is limited, step S206 may not be performed.

[0040] Regarding steps S205 and S206, one of the steps may be performed before the other step, or both steps may be performed simultaneously.

[0041] Image IM2 is shown in Figure 7. For ease of understanding, this image shows the map after objects approximated by cubes, cones, etc. (i.e., simple models) have been placed on it. This image IM2 can be rendered on a display device by the viewer through normal perspective transformation according to the evaluator's position, line of sight, and projection table distance.

[0042] The area surrounded by dashed lines in image IM2 indicates the space between 5 meters and 150 meters above ground, which is the main flight space of the unmanned aerial vehicle.

[0043] Fig. 8 shows an example of the functional configuration of the server 21. The server 21 includes a data acquisition unit 211, a first correction processing unit 212, a second correction processing unit 213, and a parameter calculation unit 214. As shown in Fig. 1, the latest map database 22 is connected to the server 21, and in addition, a main structure database 23, a simplified 3D map database 161, and a video database 162 are also connected to the server 21. The simplified 3D map database 161 and the video database 162 are either transmitted from the moving object 1 to the server 21 in step S206 in Fig. 6, or acquired from the recording medium 16 after the recording is completed in step S205 in the same figure.

[0044] The position coordinates and sizes of major structures whose latitude and longitude have already been measured and determined as coordinates are stored in the major structure database 23. Examples of major structures include existing buildings, triangulation points, manholes, and utility poles.

[0045] 9 shows a flow of a process for creating a detailed 3D map database performed by the server 21. In step S301, the data acquisition unit 211 reads out records in the main structure database 23, records in the simplified 3D map database 161, and video data in the video database 162.

[0046] In step S302, the first correction processing unit 212 judges whether or not the simplified 3D map database 161 read in step S301 contains a record relating to the same structure (as a reference point) as the structure stored in the main structure database 23. If the result of this judgment is positive, step S311 is carried out next, and if not, step S321 is carried out.

[0047] In step S311, the first correction processing unit 212 rewrites the position coordinate information (latitude, longitude) of the record in the simplified 3D map database 161 found in step S302 to the corresponding position coordinate information (latitude, longitude) in the main structure database 23. This rewriting process is the first correction process. In the same step, the first correction processing unit 212 further writes the record after the first correction process in the latest map database 22.

[0048] In step S312, the first correction processing unit 212 writes a history that the correction processing was performed in step S311 into a predetermined log file. This log file can be stored in, for example, a storage device in the server 21. After step S312, the process ends.

[0049] Usually, the position coordinate information in the main structure database 23 is based on a reference point measured and specified by the Ministry of Land, Infrastructure, Transport and Tourism or a highly accurate coordinate actually measured from that reference point; these highly accurate coordinates are referred to as confirmed coordinates here. In contrast, the position coordinate information in the simplified 3D map database 161 is based on electromagnetic wave data emitted by GNSS satellites, and therefore the accuracy is not guaranteed. Therefore, in step S311, a correction process is performed to replace the position coordinate information in the simplified 3D map database 161 with the coordinates in the main structure database 23.

[0050] However, there may be cases where the position coordinate information in the main structure database 23 has a larger error than the position coordinate information in the simplified 3D map database 161. For example, this may occur when data in the main structure database 23 is acquired before a disaster occurs and records in the simplified 3D map database are created after the disaster occurs. In preparation for such cases, the log file recorded in step S312 of the coordinate corrections can be used to retrospectively verify whether the correction process in step S311 was appropriate.

[0051] In step S321, the second correction processing unit 213 judges whether or not triangulation is possible for objects that are not main structures among the objects in the simplified 3D map database 161. If at least two main structures necessary for triangulation of an object that is not a main structure are shown in the image in which the object appears, the second correction processing unit 213 judges that the object is surveyable, otherwise it judges that the object is not surveyable. If it is judged that the object is surveyable, step S322 is performed next, and if it is judged that the object is not surveyable, step S331 is performed.

[0052] In step S322, the second correction processing unit 213 performs triangulation on an object that is not a main structure from an image in which the object and at least two of the main structures are captured, and obtains position coordinates for the object. The position coordinates obtained by triangulation calculation from the two confirmed coordinates are recognized as confirmed coordinates.

[0053] In step S323, the second correction processing unit 213 rewrites the coordinate information of the object that is not a main structure in the simplified 3D map database 161 to the coordinate acquired in step S322, and simultaneously stores the target object as a main structure in the main structure database together with the new confirmed coordinates. This operation gradually increases the number of main structures, and the accuracy of the 3D map improves as the main structure database increases. This rewriting process is also called the second correction process. Then, the process ends.

[0054] In step S331, the second correction processing unit 213 writes the uncorrected coordinate information for an object that is not a main structure as calculated coordinates, not confirmed coordinates, in the latest map database 22. Then, the process ends.

[0055] Steps S311 and S312 can be performed for each record in the simplified 3D map database 161 that is determined to be a main structure in step S302. Also, step S321 and the subsequent steps can be performed for each record in the simplified 3D map database 161 that is determined not to be a main structure in step S302.

[0056] In this way, with regard to location coordinate data, in addition to location coordinate information from GNSS satellites or highly accurate location calculations using RTK (Real Time Kinematics), the location accuracy is adjusted using buildings, utility poles, manholes, and other ground reference points with fixed coordinates, improving the accuracy of the map data on the server. Buildings, utility poles, manholes, etc. that serve as ground reference points can be surveyed in advance and a set of their shape and confirmed coordinates can be stored as known data in a reference point data table (major structure database), or a system (beacon) can be created that transmits information about the reference point (the identifier and confirmed coordinates of the reference point) from the reference point via radio waves, etc. When creating a map or when a mobile object moves, the shape of the reference point is recognized from the camera image and matched with the reference point data table, or an identifier is acquired from the received radio wave and matched with the reference point data, or if the reference point transmits an identifier and location information, it can be used as the location data as it is. The ground reference point group is designed to cover the area efficiently (for example, a mesh structure with one point every 1 km is created), and by measuring it separately, it can be used as a useful base information for unmanned flight and unmanned driving (making it possible to fly in places where GNSS radio waves cannot reach). The location data of the reference point is treated as a confirmed coordinate. The confirmed coordinates are multiplied by step S323. When a new ground control point is installed on a map and more accurate position information is obtained compared to GNSS data, the server corrects the map data up to the surrounding control points. The position data obtained using data obtained from the ground control point is stored on the map on the server as confirmed coordinates, and in principle is not corrected thereafter.

[0057] To explain again, a confirmed coordinate refers to position information measured by ground surveying, or a coordinate calculated from two or more confirmed coordinates. The calculated coordinates are coordinates calculated using one confirmed coordinate and depth-attached point cloud data or GNSS, or coordinates calculated using GNSS and depth-attached point cloud data. The calculated coordinates have lower accuracy than the confirmed coordinates. The calculated coordinates may be promoted to confirmed coordinates due to an increase in the confirmed coordinate points (step S323). The accuracy of the final data may be impaired due to ground movements, disasters, etc., so by providing an update mode (step S311), the data can be reset. The resetting can be performed for each region.

[0058] 10 shows a process performed by the server 21 to generate parameters (weighting coefficients) of a mathematical model used by the object determination unit 152 for making a determination. In step S401, the data acquisition unit 211 acquires training data. As described above, this training data includes a training image, an object appearing in the training image, and the position of the object in the training image.

[0059] In step S402, the parameter calculation unit 214 calculates parameters of the above mathematical model based on the teacher data, that is, a learned first mathematical model is constructed.

[0060] 11 shows a flow of parameter generation processing for a distance estimation model (second mathematical model) performed by server 21. In step S501, data acquisition unit 211 acquires a learning image, an object appearing in the learning image, and the position of the object in the learning image, similar to step S401.

[0061] In step S502, the data acquisition unit 211 acquires data (LiDAR data or depth data) relating to the distance between an object captured in the learning image in step S501 and the image capture point. Regarding steps S501 and S502, one of the steps may be performed before the other step, or both steps may be performed simultaneously.

[0062] In step S503, the parameter calculation unit 214 calculates parameters of a distance estimation model (second mathematical model) using the data acquired in step S501 and the distance data acquired in step S502 as teacher data. That is, a learned second mathematical model is constructed.

[0063] Unlike the moving body 1, a moving body not equipped with the LiDAR device 14 can obtain the type of object captured in the input image, the position of the object in the image, and the distance to the object by using this second mathematical model. That is, even a moving body not equipped with the LiDAR device 14 can perform the process of creating the simplified 3D map database shown in FIG. 6.

[0064] In addition, a horizontal cross-sectional image at a specified altitude can be created from the latest map database 22. As shown in Fig. 12, a horizontal cross-sectional image IM11 at a first altitude X and a horizontal cross-sectional image IM12 at a second altitude Y can be created from the latest map database 22 by a viewer installed in the server 21. The server 21 can create these cross-sectional images by receiving an input of the specified altitude from the user.

[0065] 13 shows an example of a hardware configuration of the server 21. The server 21 includes a processor 21a, an interface device 21b, a display device 21c, an input device 21d, a drive device 21e, an auxiliary storage device 21f, and a memory device 21g, which are interconnected by a bus 21h.

[0066] The processor 21a includes a CPU and a GPU. The GPU is capable of performing highly parallel processing.

[0067] A program for realizing the functions of the server 21 is provided by a recording medium 21i such as a CD-ROM. When the recording medium 21i on which the program is recorded is set in the drive device 21e, the program is installed in the auxiliary storage device 21f from the recording medium 21i via the drive device 21e. Alternatively, the program does not necessarily have to be installed by the recording medium 21i, but can also be installed via a network. The auxiliary storage device 21f stores the installed program as well as necessary files, data, and the like.

[0068] The memory device 21g reads out and stores the program from the auxiliary storage device 21f when an instruction to start the program is received. The processor 21a realizes the functions of the server 21 according to the program stored in the memory device 21g. The interface device 21b is used as an interface for connecting to other computers through a network. The display device 21c displays a GUI (Graphical User Interface) or the like according to the program. The input device 21d is a keyboard, a mouse, or the like.

[0069] The edge computer 15 also has a hardware configuration similar to that of the server 21.

[0070] The embodiments described above have aspects as an apparatus, a method, and a computer program.

[0071] According to the embodiment described with reference to Figs. 1 to 7 and 10, a 3D map database for flying an unmanned aerial vehicle can be efficiently created using a mathematical model based on machine learning. The main space in which an unmanned aerial vehicle flies is a space between 5 meters above the ground and 150 meters above the ground. A 3D map database with a certain degree of accuracy can be created to avoid collisions with objects such as buildings that exist in this space and fly safely.

[0072] When flying an unmanned aerial vehicle, it is necessary to recognize objects such as buildings in the airspace. However, it is not necessary to recognize the patterns of the objects or the windows and other features. In addition, except for the takeoff and landing points, it is not necessary to recognize the detailed unevenness of the objects in the airspace. In light of this situation, it is possible to increase the judgment speed by cutting out unnecessary parts, and perform object recognition that focuses on improving real-time performance.

[0073] According to the processes shown in Figures 8 and 9, the accuracy of the 3D map database created by the process shown in Figure 6 can be improved. This contributes to safe flight of the unmanned aerial vehicle. Depending on the performance of the edge computer 15, the process of Figure 9 may be performed by the edge computer 15 instead of the server 21.

[0074] According to the mathematical model created in the process shown in FIG. 11, a 3D map database can be efficiently created even for a moving body that is not equipped with a relatively expensive LiDAR device.

[0075] In addition, manholes, which serve as ground control points, serve as nodes between map information on underground structures and aboveground maps, making it possible to integrate underground maps and aboveground maps.

[0076] Regarding the embodiments described above, the following supplementary notes are disclosed. [Appendix 1] A 3D map data creation program for creating 3D map data for an unmanned aerial vehicle flight, comprising: an acquisition step of acquiring an image of the surroundings of the moving body captured by a camera mounted on the moving body, an imaging direction of the camera, position coordinate information of the moving body at the time of the imaging, and distance information between the moving body and an object captured in the image of the surroundings of the moving body; a determination step of determining the type of object shown in the surrounding image and the in-image position of the object shown in the surrounding image by using a mathematical model that has been trained using an input image, the type of object shown in the input image, and the in-image position of the object shown in the input image as training data; a calculation step of calculating position coordinates of the object, the type of which has been determined in the determination step, based on position coordinate information of the moving body, the imaging direction of the camera, and the distance information, and calculating a size of the object, the type of which has been determined in the determination step, based on a size of the object in the surrounding image and the distance information; a creating step of creating three-dimensional map data in which the type of each object determined in the determining step is combined with the type of the object, the position coordinates of the object, and the size of the object; storing the created relief map data group as a relief map database for reuse; A program for creating 3D map data that causes a computer to execute the above. [Appendix 2] In the calculation step, a size of the object whose type has been determined in the determination step is calculated based on a shape of the object, The shape of the object is derived from a geometric shape model table in which a relationship between the type of the object and the shape of the object is defined. 3. A 3D map data creation program as set forth in appendix 1. [Appendix 3] a first determination step of determining whether or not to rewrite coordinate information of an object in the 3D map data to the determined coordinates of a structure to be compared with an object in the 3D map data based on the 3D map data generated by the 3D map data generation program according to appendix 1 or 2 and known determined coordinates stored in a coordinate reference point database of the structure to be compared with the object in the 3D map database; a first correction step of rewriting coordinate information of the object in the 3D map data to the confirmed coordinates of the structure when the determination result of the first determination step is affirmative; a history creating step of creating a history of the execution of the correction step; a second judgment step of judging whether or not a survey of position coordinates of an object in the 3D map data can be performed based on the determined coordinates of the structure when the judgment result of the first judgment step is negative; a surveying step of performing the survey when the judgment result of the second judgment step is positive; a second correction step of rewriting coordinate information of an object in the 3D map data to coordinate information obtained in the surveying step; a step of adding a combination of the object and the coordinate information obtained by the surveying step as a new confirmed coordinate to a coordinate reference point database; The 3D map data updating method according to claim 1 or 2, further comprising causing a computer to execute the steps of: [Appendix 4] A first acquisition step of acquiring an image, a type of an object appearing in the image, and a position of the object appearing in the image; A second acquisition step of acquiring a distance from a shooting point of the image to an object shown in the image; a generation step of generating parameters of a mathematical model that outputs, when an input image is input, the type of object shown in the input image, the position of the object in the input image, and the distance to the object, using the image, the type of object, the position of the object in the image, and the distance as training data; A mathematical model learning program that runs on a computer.

[0077] Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and various modifications and changes can be made based on the technical concept of the present invention. [Explanation of symbols]

[0078] S System 1. Mobile 1a unmanned aerial vehicle 1b Vehicle 2. Center 11 Camera 12 GNSS receiver 13 Inertial Measurement Unit (IMU) 14 LiDAR device 15 Edge Computers 16 Recording media 2. Center 21 Server 22 Database 151 Data Acquisition Section 152 Object determination section 153 Position and size calculation unit 154 3D Modeler 211 Data Acquisition Department 212 First correction processing unit 213 Second correction processing unit 214 Parameter Calculation Unit

Claims

1. A 3D map data creation program for creating 3D map data for an unmanned aerial vehicle flight, comprising: an acquisition step of acquiring an image of the surroundings of the moving body captured by a camera mounted on the moving body, an imaging direction of the camera, position coordinate information of the moving body at the time of the imaging, and distance information between the moving body and an object captured in the image of the surroundings of the moving body; a determination step of determining a type of object in the surrounding image by using a mathematical model that has been trained using an input image and a type of object in the input image as training data; a calculation step of calculating position coordinates of the object, the type of which has been determined in the determination step, based on position coordinate information of the moving body, the imaging direction of the camera, and the distance information, and calculating a size of the object, the type of which has been determined in the determination step, based on a size of the object in the surrounding image and the distance information; a creating step of creating three-dimensional map data in which the type of each object determined in the determining step is combined with the type of the object, the position coordinates of the object, and the size of the object; storing the created relief map data group as a relief map database for reuse; A program for creating 3D map data that causes a computer to execute the above.

2. In the calculation step, a size of the object whose type has been determined in the determination step is calculated based on a shape of the object, The shape of the object is derived from a geometric shape model table in which a relationship between the type of the object and the shape of the object is defined.

2. The 3D map data generating program according to claim 1.

3. a first determination step of determining whether or not to rewrite coordinate information of an object in the relief map data to the determined coordinates of a structure to be compared with an object in the relief map database, based on the relief map data generated by the relief map data generation program according to claim 1 or 2 and known determined coordinates stored in a coordinate reference point database of the structure to be compared with the object in the relief map database; a first correction step of rewriting coordinate information of the object in the 3D map data to the confirmed coordinates of the structure when a result of the determination in the first determination step is affirmative; a history creating step of creating a history of the execution of the correction step; a second judgment step of judging whether or not a survey of position coordinates of an object in the 3D map data can be performed based on the determined coordinates of the structure when the judgment result of the first judgment step is negative; a surveying step of performing the survey when the judgment result of the second judgment step is positive; a second correction step of rewriting coordinate information of an object in the three-dimensional map data to coordinate information obtained in the surveying step; a step of adding a combination of the object and the coordinate information obtained by the surveying step as a new confirmed coordinate to a coordinate reference point database; 3. The 3D map data updating method according to claim 1, further comprising the steps of:

4. A first acquisition step of acquiring an image, a type of an object appearing in the image, and a position of the object appearing in the image; A second acquisition step of acquiring a distance from a shooting point of the image to an object shown in the image; a generation step of generating parameters of a mathematical model that outputs, when an input image is input, the type of object shown in the input image, the position of the object in the input image, and the distance to the object, using the image, the type of object, the position of the object in the image, and the distance as training data; A mathematical model learning program that runs on a computer.

Images