3D map creation device, method, and program

Abstract

To provide a map creation system capable of effectively create a map.SOLUTION: In a three-dimensional map creation system that acquires video taken by a video acquisition device which images a field while moving within the field, obtains a group of three-dimensional points by performing feature matching on images constituting the video, and thus creates a three-dimensional map of the field, a condition EX1 in which a wheel of a wheeled moving body gets on a step h or a condition EX2 in which the wheeled moving body gets off from the step is extracted from the group of three-dimensional points, and thus information on the step is linked to the three-dimensional map.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a 3D map creation device, method, and program for creating a 3D map that can be used by a mobile robot or the like. [Background technology]

[0002] Technologies that reduce the effort required to create maps that can be used by mobile robot systems, which recognize their own location using maps and measurement information of the surrounding environment and move autonomously, are being investigated.

[0003] Patent Document 1 discloses a 3D map generation system that "includes a robot and a remote control device, and while moving the robot using the remote control device, scans the surroundings with a 3D laser distance measuring system to obtain scan data (data set), generates a 2D grid map using the multiple data sets, aligns the 2D grid map with a predetermined 3D coordinate system, corrects the positional deviation of all points in the data set in the 3D coordinate system, and then plots each point on a voxel-format 3D map to generate a 3D map."

[0004] Patent document 2 discloses a mobile robot system comprising: "a mobile robot that is equipped with a transmitting means for adding an ID to measurement information that identifies the operating location and / or operator of the mobile robot, and transmitting the measurement information with the ID added to a map generation device; the map generation device that is equipped with a receiving means for receiving the measurement information with the ID added, a means for identifying the operating location and / or operator of the mobile robot that obtained the measurement information from the measurement information with the ID added, a processing means for creating an updated map from pre-stored map information and the measurement information, a display means for displaying the created updated map, and further a means for accepting input of a correction command for the update map, the processing means correcting the update map in accordance with the correction command."

[0005] Non-Patent Document 1 discloses an autonomous driving map creation system for robots in which a map creator carries a backpack on his / her back and moves around to create a map for the robot. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-198517 A [Patent Document 2] JP 2013-211025 A [Non-patent literature]

[0007] [Non-Patent Document 1] ZMP, "RoboMap (registered trademark, RoboMap) - a map creation system for autonomous driving", Internet<URL:https: / / www.zmp.co.jp / ad-tech / robomap> Summary of the Invention [Problem to be solved by the invention]

[0008] However, prior art methods leave room for improvement in making maps more efficient.

[0009] That is, the conventional technologies have limited effect in reducing the effort required for map creation. The methods of Patent Document 1 and Patent Document 2 require the effort of running a special mobile robot, which is different from a general-purpose user terminal device, in the location where a 3D map is to be created. The method of Non-Patent Document 1 also requires the effort of a person wearing a special backpack as a measuring device and walking in the location where a 3D map is to be created. Furthermore, even a small step can be an obstacle to a mobile robot's movement, but such information was not included in the 3D map created by the conventional technologies.

[0010] In view of the above problems of the prior art, the first object of the present invention is to provide a map creation device and a program capable of creating a map effectively. In view of the fact that there is generally a huge amount of video to be used as an analysis target when adding additional information to a map, the second object of the present invention is to provide a method that can contribute to creating a map effectively by narrowing down the video in advance. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, the present invention has a first feature in that an image is acquired by an image acquisition device that captures images while moving within a field, and a 3D point cloud is obtained by feature matching between images that constitute the image, and in a 3D map creation device that creates a 3D map of the field, a state in which a wheel of a wheeled mobile body is climbing up or descending a step is extracted from the 3D point cloud, and information about the step is linked to the 3D map.

[0012] The present invention is characterized in that it is a method for narrowing down images to be analyzed in advance when analyzing images captured by an image capture device that captures images while moving within a field to obtain additional information for a map of the field, and the method narrows down images to be analyzed in advance by using information that affects the possibility of the presence, capturing capability, or the possibility or ease of movement of a moving object moving within the field.The present invention is also characterized in that it is a program that causes a computer to function as the 3D map creation device. Effect of the Invention

[0013] According to the first feature, a 3D map can be created by linking information on steps from images captured by an image capture device as a general-purpose device having an image capture function, so that map creation can be performed effectively. According to the second feature, images are narrowed down in advance using information that affects the possibility of the existence, photographing ability, or the possibility or ease of movement of a moving object, so that map creation can be performed effectively. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing a configuration of a 3D map creation system according to each embodiment. [Diagram 2] FIG. 1 is a functional block diagram of a 3D map creation system according to each embodiment. [Diagram 3] 4 is a flowchart of the operation of the 3D map creation system according to the first and second embodiments. [Figure 4] 11A and 11B are diagrams illustrating a schematic example of calculating a step height from the inclination of a wheel of a wheeled mobile body; [Diagram 5] FIG. 2 is a diagram illustrating a configuration of a distributed 3D map. [Figure 6] FIG. 1 is a diagram illustrating an example of a hardware configuration of a typical computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] 1 is a diagram showing the configuration of a 3D map creation system according to each embodiment (first and second embodiments). In the first embodiment, the 3D map creation system 10 includes a plurality of (n units) image collection devices 1-1, 1-2, ..., 1-n, such as automobiles equipped with drive recorders and bicycles equipped with cycling recorders, that can move within a field F, such as on the street and / or indoors, a 3D map creation device 2 that creates a 3D map, and a plurality of (m units) mobile robots 3-1, 3-2, ..., 3-m that move autonomously within the field F.

[0016] In the second embodiment, the 3D map creation system 10 includes, in addition to the configuration of the first embodiment, a plurality of (j units) mobile terminals 4-1, 4-2, ..., 4-j that are carried by users and move within a field F. Note that the values ​​of n, m, and j can be any values ​​according to the configuration of the 3D map creation system 10.

[0017] 2 is a functional block diagram of the 3D map creation system 10 according to each embodiment, in which any one of the multiple (n units) image collection devices 1-1, 1-2, ..., 1-n shown in FIG. 1 is designated as the image collection device 1, any one of the multiple (m units) mobile robots 3-1, 3-2, ..., 3-m is designated as the mobile robot 3, any one of the multiple (j units) mobile terminals 4-1, 4-2, ..., 4-j is designated as the mobile terminal 4, and the functional block configurations of these image collection devices 1, mobile robots 3, mobile terminals 4, and 3D map creation device 2 are shown. (That is, in FIG. 2, the functional block configuration common to each of the image collection devices 1-1, 1-2, ..., 1-n is designated as the image collection device 1, the functional block configuration common to each of the mobile robots 3-1, 3-2, ..., 3-m is designated as the mobile robot 3, and the functional block configuration common to each of the mobile terminals 4-1, 4-2, ..., 4-j is designated as the mobile terminal 4.)

[0018] 2, configurations not enclosed in parentheses are configurations commonly used in both the first and second embodiments, and configurations enclosed in parentheses (configurations with reference numerals 217-223 in the 3D map creation device 2 and the mobile terminal 4) are configurations additionally used in the second embodiment. With this additional configuration, the second embodiment can perform additional processing in addition to the processing in the first embodiment.

[0019] Each of the video collection devices 1 may be a device mounted on a moving object such as a vehicle such as an automobile or a mobile robot 3, or may be a device (e.g., a surveillance camera or a live camera) installed in a city or on a roadside. In the following description, a video collection device 1 mounted on a vehicle is taken as an example. The vehicle may be, for example, an automobile, a bicycle, a wheelchair, or a stroller. The video collection device 1 may be, for example, a drive recorder mounted on a vehicle. The 3D map creation system 10 according to each embodiment includes a plurality of video collection devices 1. The plurality of video collection devices 1 are mounted on separate vehicles. Each of the video collection devices 1 can communicate with the 3D map creation device 2 via a communication network NW such as the Internet.

[0020] In the first and second embodiments, the video collection device 1 includes an imaging unit 101, a position measurement unit 102, an acceleration measurement unit 103, a video file storage unit 104, a meta information storage unit 105, a meta information transmission unit 106, a corresponding file search unit 107, and a video file transmission unit 108.

[0021] The 3D map creation device 2 is, for example, a server device existing on a cloud. In the first embodiment, the 3D map creation device 2 includes a meta information storage unit 201, a 2D map storage unit 202, a robot permitted road and driving location storage unit 203, an image acquisition unit 204, a feature point extraction unit 205, a feature point correspondence processing unit 206, a 3D map generation unit 207, a 3D map storage unit 208, a wheeled moving object extraction unit 209, a slope extraction unit 210, a step height calculation unit 211, a 3D map step reflection unit 212, an in-map moving object exclusion unit 213, a map distribution unit 214, a fee calculation unit 215, and an important road presentation unit 216.

[0022] In the second embodiment, the 3D map creation device 2 further includes, in addition to the configuration of the first embodiment, an attribute information storage unit 217, a position information storage unit 218, a low-speed wheeled moving object extraction unit 219, a low-speed moving object passing tendency extraction unit 220, a weather information storage unit 221, a clear image extraction unit 222, and a lifting motion detection unit 223.

[0023] The mobile robot 3 is an autonomously movable robot, and includes a map storage unit 301 in the first and second embodiments.

[0024] The mobile terminal 4 according to the second embodiment is, for example, a smartphone, which is carried by the user while moving around, and also when using a bicycle, a stroller, a wheelchair, a car, etc. The mobile terminal 4 includes an attribute information storage unit 417 and a position information acquisition unit 418.

[0025] 3 is a flowchart of the operation of the 3D map creation system 10 according to the first and second embodiments. In step S1, in the first and second embodiments, each image collection device 1 acquires images by linking them with meta information and transmits them to the 3D map creation device 2. In the second embodiment, in step S1, each mobile terminal 4 further acquires location information history and the like and transmits them to the 3D map creation device 2.

[0026] In step S2, in the first and second embodiments, the 3D map creation device 2 analyzes each image constituting the collected video transmitted from each video collection device 1 in step S1 to create a 3D map and find steps in the 3D map. In the second embodiment, in step S2, the 3D map creation device 2 can perform additional processing when creating a map and finding steps by additionally using information acquired from each mobile terminal 4 in step S1.

[0027] In step S3, in the first and second embodiments, the mobile robot 3 moves autonomously within the field F using the three-dimensional map of the field F that includes the step information created in step S2.

[0028] The following describes in detail each of the functional units of the 3D map creation system 10 in Fig. 2 that realizes each of the processes in steps S1 to S3 in Fig. 3 in the first embodiment. (Note that, as described above, the second embodiment involves additional processes using additional configurations in addition to the processes in the first embodiment. The second embodiment will be described later after the first embodiment has been described.)

[0029] In step S1, each unit of the video collection device 1 and some of the functional units 201-204 of the 3D map creation device 2 perform the following processing, thereby acquiring video associated with meta information and transmitting it to the 3D map creation device 2. At this time, some of the functional units 201-204 of the 3D map creation device 2 perform processing to instruct the video collection device 1 to transmit only video within the range in which the mobile robot 3 moves (i.e., video within the range required for the 3D map created by the 3D map creation device 2) from among all the videos acquired by the video collection device 1.

[0030] The imaging unit 101 is realized by a camera as hardware, and captures the outside of the vehicle (own vehicle) on which the video collection device 2 is mounted. The imaging direction of the imaging unit 101 is set so as to capture the outside of the vehicle (own vehicle) on which the video collection device 1 is mounted. The imaging direction is the traveling direction of the own vehicle, the opposite direction to the traveling direction of the own vehicle, the side of the own vehicle, etc. For example, the imaging unit 101 may capture the traveling direction of the own vehicle. Also, the imaging unit 101 may be capable of capturing images in a plurality of imaging directions at the same time. For example, the imaging unit 101 may capture the traveling direction of the own vehicle and the opposite direction to the traveling direction at the same time. The imaging unit 101 temporarily holds the captured imaging data in an imaging buffer, and converts the imaging data held in the imaging buffer into a file in a format such as MPEG-4 at a certain period or when a certain storage capacity is reached. A camera that realizes the imaging unit 101 as hardware is provided in, for example, a drive recorder mounted on an automobile, and the imaging data is imaging data capturing an image of the surrounding situation, such as the traveling direction of the automobile.

[0031] The position measurement unit 102 measures the position (imaging position) at the time of imaging by the imaging unit 101. For example, if the drive recorder includes a GPS (Global Positioning System) receiver, the position measurement unit 102 acquires the position measured by the GPS (outdoor GPS) as hardware as the imaging position. The position measurement unit 102 may also acquire the imaging position by any existing method such as the outdoor GPS and / or the indoor GPS.

[0032] The acceleration measuring unit 103 can be realized by using an acceleration sensor as hardware, and measures the axle acceleration when capturing an image by the imaging unit 101. The acceleration measuring unit 103 acquires the acceleration (vertical vibration) generated in the axle in order to grasp steps such as unevenness on the road surface.

[0033] The video file storage unit 104 stores the video files captured and filed by the imaging unit 101. The video file storage unit 104 stores identification information (video file ID) of the video file in association with the video file.

[0034] The meta information storage unit 105 stores meta information corresponding to a video file captured by the imaging unit 101 and filed in association with the video file ID of the video file. The meta information corresponding to a video file is information including at least the imaging time of the video file and the imaging position of the measurement result of the position measurement unit 102. The imaging time of the video file may be the imaging start time when imaging of the video of the video file started, or may be the imaging time period from the imaging start time when imaging of the video of the video file started to the imaging end time when imaging ended. The meta information storage unit 105 may also include the axle acceleration which is the measurement result of the acceleration measurement unit 103.

[0035] In the above manner, the video file storage unit 104 and the meta information storage unit 105 store the video file MV k= {F(t)|t=1,2,...,N(k)} (k is the ID of the video file, F(t) is the frame image at time t, and N(k) is the video file MV k (length) and the video file MV k Meta information MT k ={(t,Pos(t),Acc(t))|t=1,2,...,N(k)} (Pos(t) is position information at time t, and Acc(t) is acceleration information at time t) is stored. (Note that, although time t has been described as an integer number representing the frame F(t), time t may be configured with information on the corresponding real time.)

[0036] The units 101-105, the details of which have been described above, play a role in acquiring and storing images together with meta information in the image collection device 1. The units 106-108 and 201-204, the details of which will be described below, play a role in using meta information as a search key to acquire only images that are necessary for creating a 3D map in the 3D map creation device 2 from all images stored in the image collection device 1, and transmitting them from the image collection device 1 to the 3D map creation device 2. This allows only the necessary images to be transmitted over the network NW, thereby making it possible to save on communication traffic.

[0037] The meta information transmitting unit 106 transmits meta information with the video collection device ID, which is obtained by adding video collection device identification information (video collection device ID) for identifying its own video collection device 1, to the meta information storage unit 201 of the 3D map creation device 2. The timing for transmitting the meta information with the video collection device ID to the 3D map creation device 2 may be at a fixed cycle or at the timing of connection to the communication network NW.

[0038] The meta information transmission unit 106 may transmit information using any communication technology. For example, it may transmit meta information with a video collection device ID via LTE (Long Term Evolution) communication at a fixed interval, or may transmit multiple pieces of meta information with a video collection device ID all at once when connected to a wireless communication network such as a wireless LAN (Local Area Network).

[0039] The corresponding file search unit 107 searches the meta information storage unit 105 for meta information corresponding to the image capture time and image capture position indicated in the video file request received from the 3D map creation device 2. The corresponding file search unit 107 notifies the video file transmission unit 108 of the video file ID associated with the meta information resulting from the search.

[0040] The video file transmission unit 108 acquires the video file associated with the video file ID notified by the corresponding file search unit 107 from the video file storage unit 104 , and transmits the acquired video file to the 3D map creation device 2 .

[0041] The meta-information storage unit 201 stores the meta-information received from each video collection device 1 in association with the video collection device ID of the video collection device 1.

[0042] The two-dimensional map information storage unit 202 stores nationwide road map data as data relating to field F. The two-dimensional map information storage unit 202 may further store indoor two-dimensional map data relating to indoor facilities, store interiors, and the like as data relating to field F.

[0043] The robot-permitted road and driving location storage unit 203 stores location information of roads on which the mobile robot 3 can travel, linked to nationwide road map data stored in the two-dimensional map information storage unit 202. The robot-permitted road and driving location storage unit 203 stores a geographic identifier that uniquely identifies a road on which the mobile robot 3 can travel. Furthermore, the robot-permitted road and driving location storage unit 203 stores, as the range of position information, the lanes (driving locations) on the target road on which the mobile robot 3 is permitted to travel.

[0044] The above 2D map data stored in the 2D map information storage unit 202 and data related to the robot permitted roads and areas on the 2D map where the robot can move, etc., stored in the travel location storage unit 203 are prepared in advance. The format of these data may be any existing format of 2D map data.

[0045] Based on the image capture time and image collection device ID associated with the image capture location for which a 3D map is to be created, which are stored in the meta information storage unit 201, the image acquisition unit 204 requests (image file request) a video file corresponding to the image capture time and image capture location from the image collection device 1 of that image collection device ID. Here, the image capture location for which a 3D map is to be created is either a robot-permitted road or a location range stored in the driving location storage unit 203. In response to the video file request, the image acquisition unit 204 receives the video file corresponding to the image capture time and image capture location from the image collection device 1 of that image collection device ID.

[0046] In addition, the information on the imaging location for which a 3D map is to be created, which serves as a search key for the video acquisition unit 204 to request a video file, can be information manually specified by the user, such as a robot-permitted road or any (all or part) of the location range stored in the driving location storage unit 203.

[0047] The above has described the details of each unit for implementing step S1. Next, in step S2, the remaining functional units 205-216 (other than the functional units 201-204 responsible for step S1) of the 3D map creation device 2 perform the following processing, thereby creating a 3D map and providing step information on the 3D map.

[0048] The feature point extraction unit 205 extracts images (a plurality of frame images F(t) in which a common spatial range is captured at a common time t with various camera arrangements) from the video files acquired by the video acquisition unit 204, and extracts a sufficient number of feature points from the images, in order to synthesize video files in which the same position (common spatial range) is captured at a common time from various cameras at various angles, and to create a 3D map for each time and spatial range. (It should be noted that the frame images (t) capturing a common spatial range may be determined by dividing the spatial range for which the map is to be created into predetermined partition ranges in advance, and determining whether the position information Pos(t) in the meta information belongs to the same partition range, or the like.) The feature point extraction unit 205 extracts points or small region images that should be feature points from the images. For example, an algorithm such as SIFT (Scale Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF) may be used to extract feature points.

[0049] The feature point extraction unit 205 may apply object recognition to the image and exclude areas of moving objects such as people (excluding objects extracted by the wheeled moving object extraction unit 209 described later) that are assumed to be unnecessary for creating a 3D map from areas from which feature points should be extracted.

[0050] The feature point correspondence processing unit 206 targets multiple video files (for example, M frame images F(t1), F(t2), ..., F(tM), where each frame image may belong to a different video file) captured at the same location based on the imaging position information stored in the meta information storage unit 201, and matches areas that are commonly observed in different images based on the feature points extracted by the feature point extraction unit 205.

[0051] The 3D map generating unit 207 generates a 3D map based on the 3D positions of each feature point associated by the feature point correspondence processing unit 206. For example, the 3D map generating unit 207 calculates the 3D coordinates (x, y, z) of feature points from pairs of feature points (feature points whose positions are given by 2D pixel positions (u, v) on the 2D frame image) associated by the feature point correspondence processing unit 206 using a 5-point method, 8-point method, or the like, and converts the 3D coordinates into 3D coordinates in a map coordinate system to generate a 3D map.

[0052] In addition, each of the above units 205 to 207 may use any existing method for generating a 3D map as a 3D point group by obtaining 3D coordinates from the correspondence of feature points on the 2D image.

[0053] The 3D map storage unit 208 stores the 3D map generated by the 3D map generation unit 207 .

[0054] Here, since it is expected that mobile robots 3 are often smaller than (passenger) automobiles, have small wheel diameters, and have poor ability to overcome steps (instead, they are flexible enough to be used for transporting goods, including indoors, where travel is considered difficult for automobiles, etc.), it is desirable to reflect the step height on a 3D map so that the mobile robot 3 that has delivered the 3D map can move around while avoiding steps that are difficult to overcome (or steps that require special movements other than normal movement movements, such as transformation movements to overcome the steps).

[0055] In this embodiment, as an example, the step height is estimated by calculating the inclination of the wheels of a moving object (such as a bicycle, a wheelchair, or a stroller) whose wheels are reflected in the three-dimensional map.

[0056] The wheeled object extraction unit 209 extracts wheeled objects reflected in the 3D map stored in the 3D map storage unit 207 within the location range of the lane information in which the mobile robot 3 is permitted to travel, which is stored in the robot-permitted road and travel location storage unit 203. The wheeled object extraction unit 209 extracts wheeled objects such as bicycles, wheelchairs, strollers, and mobile robots. The wheeled object extraction unit 209 uses, for example, deep learning to learn from various wheeled objects as teacher data, and detects wheeled objects on the 3D map.

[0057] As described above, the three-dimensional map is obtained as a point cloud for each time and space range. Therefore, even if the three-dimensional space range R is the same, it is possible to obtain three-dimensional maps for various time ranges (for example, ranges {t|t1≦t≦t2}, {t|t2≦t≦t3}, etc.) in a differentiated manner. When the wheel moving object extraction unit 209 attempts to extract from the three-dimensional map obtained in this way (as a snapshot image (animation of point cloud) for each time range and space range), the wheel moving object may be detected in the point cloud PG(R,t1≦t≦t2) in a certain time range {t|t1≦t≦t2} for the same three-dimensional space range R, but not detected in the point cloud PG(R,t11≦t≦t12) in another time range {t|t11≦t≦t12}.

[0058] In addition, when the wheel moving object extraction unit 209 detects a wheel moving object using deep learning, in addition to the three-dimensional position (x, y, z) of each point group constituting the point group PG(R, t) for region R at each time t, information on the features when the three-dimensional position (x, y, z) is extracted from a two-dimensional image (original frame image F(t)) and information on neighboring pixels of the corresponding pixel position (u, v) in the two-dimensional image may also be used as input information to the deep learning network of the deep learning.

[0059] The inclination extraction unit 210 acquires ground inclination information from the 3D map information stored in the 3D map information storage unit 208. The inclination extraction unit 210 may refer to data such as an inclination map published by the Geospatial Information Authority of Japan, Ministry of Land, Infrastructure, Transport and Tourism, for the inclination information. The inclination extraction unit 210 may calculate the inclination from, for example, the change value of the axle acceleration acquired by the acceleration measurement unit 103 and the moving speed of the moving object.

[0060] The step height calculation unit 211 calculates the step height within the location range of the lane information in which the mobile robot 3 is permitted to travel, which is stored in the robot-permitted road and travel location storage unit 203. The step height calculation unit 211 calculates the step height from, for example, the inclination of the wheels of the wheeled mobile object extracted by the wheeled mobile object extraction unit 209 and the ground inclination extracted by the inclination extraction unit 210.

[0061] Here, a schematic example of calculating the step height from the inclination of the wheels of the wheeled vehicle will be described with reference to FIG. 4. In calculating the step height h of the step 5, the moment when the wheeled vehicle climbs onto the step as shown in Example EX1, or the moment when it descends from the step as shown in Example EX2, is extracted. Then, the step height h is calculated by utilizing the angle at the moment when one of the front wheel 61 and the rear wheel 62 of the wheeled vehicle contacts the ground and the other contacts the step. For example, the step height h is calculated as "h=c*sinA" using trigonometric ratios. Here, c is the length between the front and rear wheels of the wheeled vehicle. The step height is calculated from the ground surface representing the ground inclination (angle θ) extracted by the inclination extraction unit 210, as shown in the example of FIG. 4.

[0062] In addition, the front wheel 61 and the rear wheel 62 are identified as a point group in the point group PG(R,t) obtained as a so-called three-dimensional animation for each time t=1, 2, ... of the three-dimensional region R as described above, so that the line segment L(P61,P62,t) connecting the three-dimensional position P61 of the center of the front wheel 61 and the three-dimensional position P62 of the center of the rear wheel 62 can be obtained at each time t, and the angle α(t) that this line segment L(P61,P62,t) makes with the ground GR can also be obtained at each time t. In addition, this line segment L(P61,P62,t) is also shown as the line segment L in the examples EX1 and EX2 of FIG. 4. When the front wheel 61 and the rear wheel 62 are present on the left and right of the wheel moving body, the line segment L may be determined using either the left or right. Alternatively, when the front wheels 61 and rear wheels 62 are detected on the point cloud for both the left and right sides, as shown as example EX3 in Figure 4, a line segment L can be obtained for the left and right sides, and a planar area PL (such as a rectangle corresponding to the bottom surface of the wheeled moving body) formed by the left and right line segments L can be obtained, and the angle that this planar area PL makes with the ground can be taken as angle α(t).

[0063] When the vehicle is not standing on a step, this angle α(t) roughly matches the known inclination angle θ of the ground surface GR, and is therefore "α(t) ≒ θ" (for example, the difference can be determined to be within a threshold value TH, |α(t) - θ| ≦ TH, and thus the angle is determined to roughly match). However, when the vehicle stands on a step h, or conversely, when the vehicle descends from a step h, it is expected that the value of α(t) will temporarily increase or decrease from the inclination angle θ, thereby exhibiting an extreme value. Therefore, from the behavior of the graph (t, α(t)) of the angle α(t) as a function of time t, when the angle is determined to have temporarily increased or decreased from a constant value θ and exhibited an extreme value, the amount of fluctuation in the extreme value from a constant value θ can be obtained as angle A.

[0064] In FIG. 4, a schematic example of a graph (t, α(t)) of angle α(t) is shown as graph G. At time t1, the maximum value θ+A is reached, so it can be estimated as corresponding to the climbing moment in example EX1, and at time t2, the minimum value θ-A is reached, so it can be estimated as corresponding to the descending moment in example EX2. (The period from a little after time t1 to a little before time t2 corresponds to a state in which both the front wheel 61 and the rear wheel 62 ride on the step h and the inclination angle θ is maintained.) Note that since the graph of angle α(t) is expected to be affected by noise in the actual measured values, it is possible to use a graph that has been subjected to curve fitting or the like or one that has had noise removed, and to determine the extreme value θ±A.

[0065] As another embodiment, the step height calculation unit 211 may calculate the step height from axle acceleration received from a wheeled mobile object (such as a bicycle, wheelchair, stroller, or mobile robot) that has traveled in a lane in which the mobile robot 3 is permitted to travel. The step height calculation unit 211 may derive an estimation formula for calculating the step height by regression analysis from actual measurement data of the change in axle acceleration when the wheeled mobile object climbs up or descends a step, the travel speed of the wheeled mobile object, and the step height, and calculate the step height using this estimation formula.

[0066] In addition, the step height calculation unit 211 may calculate the step height by performing correction using a combination of an embodiment in which the step height is calculated from the inclination of the wheels of the wheeled moving body extracted from the three-dimensional map and an embodiment in which the step height is calculated from the axle acceleration.

[0067] In addition, the step height calculation unit 211 may use as training data data where the step height is clear due to regulations such as the protrusion height of blocks for guiding visually impaired persons, or where the step height can be clearly determined on a 3D map, and use deep learning or the like to learn from this data, thereby improving the accuracy of step height calculation based on the wheel inclination and axle acceleration of a wheeled vehicle.

[0068] The 3D map step reflecting unit 212 reflects the step height calculated by the step height calculating unit 211 in the 3D map stored in the 3D map storage unit 208, and stores the 3D map reflecting the step height in the 3D map storage unit 208. The 3D map step reflecting unit 212, for example, virtually superimposes a step object on the 3D map. Here, if all steps are reflected on the 3D map, there is a concern that the amount of data will become enormous. Therefore, a step height threshold value may be determined in advance, such as a step height that can be overcome by all mobile robots 3 expected to be handled, and only steps exceeding the threshold value may be reflected. In addition, since it is expected that different mobile robots 3 will be handled at different locations, the step height threshold value may be changed for each location.

[0069] In the example of Figure 4, the step h is shown only as a two-dimensional cross section, but assuming that the wheeled vehicle has four wheels, the front wheels 61 and the rear wheels 62 have left and right sides and therefore a left and right width W, so the value of the left and right width W of the wheels can be used as the width of the step h.

[0070] The in-map moving object exclusion unit 213 creates a 3D map from the 3D map stored in the 3D map storage unit 208, excluding moving objects (wheeled moving objects, people, etc.), and stores the created 3D map in the 3D map storage unit 208. The in-map moving object exclusion unit 213 may, for example, generate two 3D maps based on images captured at different times of the day at the same location, and exclude objects that appear only in one of the 3D maps as moving objects. Therefore, the 3D map stored in the 3D map storage unit 208 includes both a 3D map in which moving objects are reflected for calculating step heights, and a 3D map in which moving objects are excluded for delivery to the mobile robot 3.

[0071] The map distribution unit 214 distributes the 3D map stored in the 3D map storage unit 208 to each mobile robot 3. Here, the 3D map distributed by the map distribution unit 214 is preferably a map in which step heights are reflected by the processing of the 3D map step reflection unit 212 and in which moving objects are excluded from within the map by the processing of the in-map moving object exclusion unit 213. Note that the 3D map distributed by the map distribution unit 214 to the mobile robot 3 is preferably a 3D map limited to places where the mobile robot 3 is expected to travel.

[0072] Fig. 5 is a diagram showing a schematic configuration of a 3D map to be delivered. The 3D map includes information on a solid object OB1 that appears in a sensing range SR where LiDAR, which is used for self-position estimation when the mobile robot 3 performs local autonomous movement, and image capture, etc., is used in the form of a 3D point cloud, etc., and includes information on the ground GR, including information on the inclination angle θ as described in Fig. 4, as information on a 2D plane stored in the 2D map information storage unit 202 and the robot-permitted road, traveling location storage unit 203 (information on the ground GR as a 2D plane arranged in a 3D space including information on the inclination angle θ and altitude).

[0073] The three-dimensional map information has information on a three-dimensional object OB1 within a sensing range SR (e.g., a range within a specified altitude range from the ground) for the mobile robot 3 to estimate its own position in the form of a point cloud, etc., and information on a step h can be stored in the form of a virtual object OB2, etc., as accompanying information to the information on the ground GR.

[0074] Here, information on the three-dimensional object OB1 in the sensing range SR can be obtained in the form of a point cloud or the like by converting from two-dimensional feature point matching to a three-dimensional position through processing by the respective units 205, 206, and 207. On the other hand, information on the step h that can be expressed in the form of a virtual object OB2 or the like can also be obtained in principle by converting from two-dimensional feature point matching to a three-dimensional position through processing by the respective units 205, 206, and 207. However, since it is expected that the step h is often outside the sensing range SR, it is not necessary as three-dimensional map information for self-location estimation purposes, and in this embodiment, it can be defined as an object to be avoided on the ground surface GR, as accompanying information for the ground surface GB.

[0075] Furthermore, since the step h is disposed or formed on the ground GR and is considered to have few patterns or other features, it is assumed that even if the processing of the respective units 205, 206, and 207 is applied, the matching accuracy is low and the shape of the step h cannot be extracted with high accuracy as a point cloud. In contrast, in this embodiment, a moving object having wheels reflected in a 3D map, which is usually an object to be excluded when generating 3D map information, as shown in Fig. 4, is actively utilized, and information on the step h can be obtained from an analysis of the inclination of the wheels.

[0076] The compensation calculation unit 215 calculates compensation for collecting video files from the video collection device 1. By providing a monetary or other benefit for collecting video files, it is expected that users who use automobiles equipped with drive recorders or bicycles equipped with cycling recorders will be motivated to actively collect video. The compensation can be set in the form of points that can be used by users in electronic commerce or the like (which may be a value according to the importance level presented by the next important road presentation unit 216).

[0077] The important road presentation unit 216 presents important roads that require video collection in particular. The important road presentation unit 216 treats, as important roads, roads on which the mobile robot 3 can travel, roads for which a 3D map has not yet been created, roads for which the accuracy of the 3D map is poor, and roads for which the 3D map needs to be updated due to road changes, for example. The closer the date on which the mobile robot 3 is scheduled to travel, the higher the importance of the road may be. The higher the likelihood that the road will be used as a travel road for the mobile robot 3, for example, when a delivery service using the mobile robot 3 is provided, the higher the importance of the road may be. Note that a user may set a rule base in advance regarding the importance of roads with certain attributes, and the importance may be calculated automatically from this rule base.

[0078] The important road presentation unit 216 can present the important road and its fee to the user who uses the video collection device 1, for example, by displaying it on a Web application or distributing it to a car navigation system. It is also desirable for the important road presentation unit 216 to increase the fee calculated by the fee calculation unit 215 depending on whether the road whose video has been collected by the video collection device 1 is an important road or not and on the level of its importance.

[0079] The above has described the details of each part when the 3D map creation device 2 creates a 3D map in step S2 and distributes it to the mobile robot 3. Finally, in step S3, the mobile robot 3 that has received the 3D map stores the map distributed by the map distribution unit 214 in the map storage unit 301 and uses it for making decisions regarding the autonomous driving of the mobile robot 3.

[0080] Although the individual functional block configuration for realizing autonomous driving is omitted in FIG. 2, as described in FIG. 5, the mobile robot 3 can move from the current location to the destination by repeating local movement while detecting an object OB1 within the sensing range SR of the mobile robot 3 and comparing it with 3D map information to estimate its own position. Any existing method can be used for this self-position estimation. In addition, in the route set when the mobile robot 3 moves from the current location to the destination, the step h on the ground GR can be avoided by not including it on the route, or a special operation control can be performed to overcome the step h. Any existing method can also be used for the route search from the current location to the destination, and the mobile robot 3 can realize various applications such as transporting goods and monitoring the field F.

[0081] As described above, in the first embodiment, by combining a mechanism for collecting and synthesizing a large number of drive recorder images with a mechanism for calculating step heights from images that utilizes wheeled moving objects, which would normally get in the way of a map, as markers, which is effective precisely because there are a large number of drive recorder images, it is possible to create a 3D map that also reflects step heights.

[0082] Next, the second embodiment will be described, focusing on the differences from the first embodiment. In the second embodiment, in addition to the same processing as in the first embodiment, a process of selecting (narrowing down in advance) appropriate drive recorder footage is performed in order to reduce the confirmation of footage that does not contribute to step height extraction. Although it is considered possible to automatically detect the situation as shown in FIG. 4, since there may be a huge amount of original footage, it is also considered that, at least in part, a user may visually check the original footage to select footage that is considered to be valid until automatic detection can be performed with high accuracy. In the second embodiment, it is possible to reduce the effort of such selection.

[0083] In steps S1 and S2 of the second embodiment, in addition to the same processes as those in the first embodiment, the following processes are performed by the mobile terminal 4 and the additional units 217 to 223 in the 3D map creation device 2.

[0084] First, in step S1 for collecting information in advance, additional processing is performed by the following units 217, 417, 218, and 418.

[0085] The attribute information storage unit 217 receives and stores information that associates the mobile terminal ID with the attribute information of the subscribing user, which is stored in the attribute information storage unit 417 of each of the multiple mobile terminals 4, from the attribute information storage unit 417. The attribute information is, for example, gender, generation, family structure, and age of children.

[0086] The location information storage unit 218 receives and stores information associating the mobile terminal ID of each of the multiple mobile terminals 4, which is acquired by the location information acquisition unit 418 of each of the multiple mobile terminals 4, with the location history of the mobile terminal 4. The location history information, which is information indicating the location history of the mobile terminal, is a combination of the acquisition time and the location information. Here, the location history information is processed using the attribute information stored in the attribute information storage unit 217 so that individuals cannot be identified.

[0087] In addition, the position information acquisition unit 418 of each mobile terminal 4 can acquire the position history of the mobile terminal 4 (position information Pos(t) at each time t=1, 2, ...) by an existing method such as GPS, similar to that described for the position measurement unit 102 and the accelerometer unit 103 of the video acquisition device 1. (Video file MV k The meta information MT k may be acquired as location history information by the location information acquisition unit 418.)

[0088] Next, in step S2 of the second embodiment, the 3D map creation device 2 performs additional processing of the following units 219 to 223, thereby narrowing down in advance images that are highly likely to correspond to the moment of climbing over or descending a step h as described in Fig. 4. (Note that this narrowing down processing is not required for creating the 3D map itself, and the additional processing of the following units can be applied to narrow down images used when obtaining information on the step h to be linked to the 3D map.)

[0089] The low-speed wheel moving object extraction unit 219 extracts a low-speed wheel moving object whose step height is easily detected from the image. Examples of low-speed wheel moving objects are baby strollers and wheelchairs. The low-speed wheel moving object extraction unit 219 estimates the movement of the low-speed wheel moving object from the position history information stored in the position information storage unit 218. The low-speed wheel moving object extraction unit 219 stores information on the standard movement speed (information on the movement distance at the position acquisition time T seconds and the position acquisition time T+1 seconds) of each low-speed wheel moving object such as a baby stroller or a wheelchair (assuming that a user carrying the mobile terminal 4 is moving together with the low-speed wheel moving object) in advance for judgment, and estimates whether or not the movement is made using a low-speed wheel moving object based on the degree of approximation to the standard movement speed. In addition, the accuracy of estimating whether or not the movement is made using a low-speed wheel moving object may be improved by using attribute information such as generation, family structure, and age of children.

[0090] The low-speed moving object passing tendency extraction unit 220 extracts the days of the week and time periods when there are many low-speed moving objects passing for each road. The low-speed moving object passing tendency extraction unit 220 counts the movements of low-speed wheel moving objects estimated by the low-speed moving object passing tendency extraction unit 220 for each road, each day of the week, and each time period, and extracts the days of the week and time periods when many low-speed wheel moving objects pass for each road. The low-speed moving object passing tendency extraction unit 220 adds information on the days of the week and time periods when many low-speed wheel moving objects pass for each road to the meta information storage unit 201, thereby efficiently extracting video with a high probability of capturing low-speed wheel moving objects.

[0091] The weather and sunshine information storage unit 221 stores the history of weather and sunshine information for each region. The weather and sunshine information storage unit 221 can obtain and store weather information such as rain, snow, fog, etc., and sunshine information indicating sunshine hours, by accessing a server or the like that provides the past history of weather information, etc.

[0092] The clear image extraction unit 222 narrows down the use of only clear drive recorder images using weather information and sunshine information stored in the weather and sunshine information storage unit 221. The clear image extraction unit 222 narrows down the date and time with similar conditions using the weather information and sunshine information conditions when the moment when the wheeled moving object runs up or descends from a step can be clearly extracted from the drive recorder images. The clear image extraction unit 222 adds information on the date and time that results in clear drive recorder images to the meta information storage unit 201, thereby efficiently extracting images in which the low-speed wheeled moving object is clearly shown.

[0093] The lifting motion detection unit 223 extracts the motion when the handlebars of a bicycle or a stroller are lifted. When the step height calculation unit 211 extracts the moment when the wheeled moving body runs up a step, the lifting motion detection unit 223 captures the motion when the handlebars of a bicycle or a stroller are lifted, thereby efficiently extracting the drive recorder video of the moment when the step is run up.

[0094] The motion analysis from video by the lifting motion detection unit 223 can use any existing method such as deep learning to extract a person and their joints from the video, detect the person's movement information (joint movement information), and identify the type of movement (for example, whether the person is walking, running, lifting, etc.).

[0095] In the second embodiment, it is possible to effectively narrow down in advance the images that can be used to detect the step h by additional processing by all or some of the above-mentioned units 219 to 223. Note that the low-speed moving object passing tendency extraction unit 219 and the road day-of-week / time-of-day tendency extraction unit 220 narrow down the images from information that affects the possibility of the presence of a moving object, the clear image extraction unit 222 narrows down the images from information that affects the imaging capability of the moving object (imaging capability using a camera or the like), and the lifting motion detection unit 223 narrows down the images from information that affects the possibility and ease of movement of the moving object.

[0096] Various supplementary examples, alternative examples, modifications, etc. will be described below.

[0097] (1) As an application example of the embodiment of the present invention, it is possible to perform work, etc., using a mobile robot 3 in a certain environment (field F) such as a work site by remote control, more smoothly by taking into consideration steps h that may be an obstacle to the movement of the mobile robot 3. This makes it unnecessary for a worker to actually go to the work site, etc., and reduces carbon dioxide emissions by saving energy resources required for human movement, making it possible to contribute to Goal 13 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "take urgent action to combat climate change and its impacts."

[0098] (2) The second embodiment has been described with respect to a specific application of narrowing down images corresponding to the riding state of the wheels of a wheeled mobile body when determining the step h to be linked to the 3D map in the first embodiment. The narrowing down of images by the processing of all or part of the additional configuration (functional units 219 to 223 and the mobile terminal 4) in the second embodiment compared to the first embodiment is not limited to this specific application but can also be applied to generalized applications. Not limited to the step h, the second embodiment can also be applied to adding information that affects the possibility or ease of movement of the mobile robot 3, etc., as additional information to a 3D map created by the 3D map creation device 2 (or an existing map including a 2D map, not limited to a 3D map). When associating information that influences the possibility or ease of movement to a map, that is, instead of analyzing all of a huge amount of video as original data for adding some additional information (for example, the information on the step h in the first embodiment is additional information, but is not limited to this) to a 3D map (or an existing map including a 2D map, not limited to a 3D map) created by the 3D map creation device 2, it is effective to analyze only the video selected by filtering and automatically selecting the video in advance, in order to reduce time and effort, and for this purpose, the second embodiment can be applied in a generalized form (using the above-mentioned additional configuration). As specific examples (specific examples of additional information obtained by analyzing a filtered video), the following 2A to 2C can be mentioned in addition to the step h in the first embodiment, but are not limited to these.

[0099] (2A) Time periods during which there are many people and wheeled vehicles (wheelchairs, strollers) passing by on sidewalks where the mobile robot 3 is also permitted to pass can be extracted, and it can be confirmed using a dashcam whether the mobile robot 3 is likely to be able to pass safely during those times.

[0100] (2B) On narrow roads, it is possible to extract time periods when there are a lot of pedestrians and use a dashcam to check whether people are able to walk safely during those times.

[0101] (2C) In roads where dog droppings are a problem, it is possible to extract times when dogs are likely to be walked and check the situation using a dashcam.

[0102] For the above 2A, 2B, and 2C, information other than that extracted in the additional configuration may be used for filtering. For example, by applying object detection to the video stored in the video file storage unit 104 in the meta information storage unit 105 (applying object detection in the video acquisition device 1), the number N(t) of people, wheeled moving objects, and dogs detected in the frame image F(t) at each time t (multiple frame images F(t) capturing a common spatial range) can be defined as indicating the degree of congestion (as an example of information that affects the possibility or ease of movement of a moving object such as the mobile robot 3) as additional meta information. By using the number of detections N(t) reflecting the degree of congestion as meta information, it is possible to filter only the video corresponding to the time range and spatial range in which the number of detections N(t) is determined to be large. For example, information regarding the possibility and ease of movement of the mobile robot 3 (information on steps h, etc.) can be linked to a map using the filtered video, but the filtering application is not limited to this.

[0103] (3) Fig. 6 is a diagram showing an example of the hardware configuration of a general computer device 70. Each of the image acquisition device 1, the 3D map creation device 2, the mobile robot 3, and the mobile terminal 4 constituting the 3D map creation system 10 can be realized as one or more computer devices 70 having such a configuration. When a partial configuration of the 3D map creation system 10 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network NW. The computer device 70 includes a CPU (Central Processing Unit) 71 for executing predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor for executing some or all of the instructions executed by the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device providing a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 for receiving user inputs via a mouse, a keyboard, a touch panel, or the like, a camera 81 and a sensor 82 used for the self-position estimation function of the mobile robot 3, an actuator 83 such as a motor and a moving mechanism 84 such as wheels (including a speed change mechanism such as an axle and gears) driven by the actuator 83 for providing an autonomous moving function of the mobile robot 3, and a bus BS for transmitting and receiving data between them. The camera 81 and the sensor 82 also correspond to the imaging unit 101, the position measurement unit 102, and the accelerometer unit 103 in the case of the image acquisition device 1 realized as hardware.

[0104] Each functional unit of the 3D map creation system 10 can be realized by a CPU 71 and / or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of arithmetic device (processor). Here, when display-related processing is performed, a display 76 configured as a monitor such as a liquid crystal display or a lighting device such as an LED also operates in conjunction with the above, and when communication-related processing related to data transmission and reception is performed, a communication interface 75 also operates in conjunction with the above. The communication functions of each transmission unit or distribution unit 106, 108, 214, etc. can be realized by the communication interface 75. [Explanation of symbols]

[0105] 10...3D map creation system, 1...image acquisition device, 2...3D map creation device, 3...mobile robot, 4...mobile terminal 101: imaging unit, 102: position measurement unit, 103: accelerometer unit, 104: video file storage unit, 105: meta information storage unit, 106: meta information transmission unit, 107: corresponding file search unit, 108: video file transmission unit 201...Meta information storage unit, 202...2D map storage unit, 203...Robot permitted road, driving location storage unit, 204...Image acquisition unit, 205...Feature point extraction unit, 206...Feature point correspondence processing unit, 207...3D map generation unit, 208...3D map storage unit, 209...Wheel moving object extraction unit, 210...Inclination extraction unit, 211...Step calculation unit, 212...3D map step reflection unit, 213...In-map moving object exclusion unit, 214...Map distribution unit, 215...Compensation calculation unit, 216...Important road presentation unit, 217...Attribute information storage unit, 218...Location history storage unit, 219...Low-speed moving object passing tendency extraction unit, 220...Road day of the week, time period tendency extraction unit, 221...Weather, sunshine information storage unit, 222...Clear image extraction unit, 223...Lifting motion detection unit 301: map storage unit, 417: attribute information storage unit, 418: location information acquisition unit

Claims

1. A three-dimensional map creation device that creates a three-dimensional map of a field by acquiring an image captured by an image capture device that captures images while moving within the field and obtaining a three-dimensional point cloud by feature matching between images that constitute the image, comprising: A 3D map creation device characterized by extracting from the 3D point cloud a state in which the wheels of a wheeled vehicle are climbing up or descending a step, and linking information about the step to the 3D map.

2. 2. The three-dimensional map generating device according to claim 1, wherein the three-dimensional map is generated by excluding the wheeled vehicle.

3. The 3D map creation device according to claim 1, characterized in that the wheels are detected using deep learning.

4. The 3D map creation device according to claim 1, characterized in that the wheels are detected as front wheels and rear wheels, and a state in which it is determined that the time change in the angle (α(t)) that the line segment connecting the front and rear wheels makes with respect to the ground has shown an extreme value is extracted as a state in which the wheels of the wheeled vehicle are climbing up or descending a step.

5. The three-dimensional map creation device according to claim 4, characterized in that information on the slope of the ground within the field is obtained as given information, and then the angle (α(t)) that a straight line connecting the front wheels and the rear wheels makes with the ground is obtained.

6. The 3D map creation device according to claim 4, characterized in that the wheels are detected as front and rear wheels, and the height of the step relative to the ground is obtained when it is determined that the time change in angle (α(t)) that the line segment connecting the front and rear wheels makes with the ground has reached an extreme value.

7. The 3D map creation device according to claim 1, characterized in that the 3D map is created for use by a mobile robot, using images of the range within which the mobile robot can move, and relates to the range within which the mobile robot can move.

8. The 3D map creation device according to claim 1, characterized in that only steps higher than a threshold value are extracted as steps that are difficult for a mobile robot to overcome or that require special operations for a mobile robot to overcome and are linked to the 3D map.

9. In order to narrow down in advance images corresponding to a three-dimensional point cloud to be referenced in order to extract a state in which a wheel of the wheeled vehicle is climbing up a step or descending from a step, Only images corresponding to a time and space range in which a user moves at a slow speed in the movement history of the mobile terminal carried by the user, Based on the result of aggregating the movement history for each predetermined spatial range such as a road, only images corresponding to days and / or times when it is determined that there is a lot of movement determined to be slow movement are displayed. Only images corresponding to weather and / or sunshine conditions that are judged to provide clear images by referring to the weather and / or sunshine history, or The three-dimensional map creation device according to claim 1, characterized in that by applying motion analysis to the video, only video in which it is determined that a user moving together with the wheeled vehicle is performing a lifting motion is referenced.

10. The three-dimensional map creation device according to claim 1, characterized in that the user is notified of compensation information according to a level of importance that is preset for each road in the field, as information to motivate the user using the image acquisition device to move around and acquire images.

11. A three-dimensional map creation method for creating a three-dimensional map of a field by acquiring an image captured by an image capture device that captures images while moving within the field and obtaining a three-dimensional point cloud by feature matching between images that constitute the image, comprising: A 3D map creation method characterized by extracting from the 3D point cloud a state in which the wheels of a wheeled vehicle are climbing up or descending a step, and linking information about the step to the 3D map.

12. 11. A program for causing a computer to function as the three-dimensional map creation device according to claim 1.

Images