3D map creation device and 3D map creation method

Abstract

To solve the problem that it is necessary to eliminate a three-dimensional point group unnecessary for a map, even outside an angle of view of a camera.SOLUTION: A three-dimensional map creation device 1 includes: a distance sensor 101 for acquiring a three-dimensional point group; a camera 102 for acquiring an image; an unnecessary object detection part 103 for detecting an unnecessary object unnecessary for a three-dimensional map, and outputting an unnecessary object detection result 303; a self-position estimation part 104 for estimating a self-position; an unnecessary object region management map creation part 105 for creating an unnecessary object region management map indicating a region where the unnecessary object is likely to exist; an elimination part 106 for eliminating a point group from the obtained three-dimensional point group 110 of the distance sensor 101 using the unnecessary object detection result 303 and the unnecessary object region management map; and a three-dimensional map creation part 107 for creating the three-dimensional map using the point group which has not been eliminated by the elimination part 106, and thereby a three-dimensional map can be created while eliminating the point group unnecessary for the three-dimensional map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a device and method for removing unnecessary objects that are not required for a three-dimensional map in a three-dimensional map creation device and a three-dimensional map creation method. [Background technology]

[0002] As seen in Non-Patent Document 1, there is a view that high-definition maps are not necessary for autonomous driving. However, autonomous vehicles generally use sensors such as onboard cameras and LiDAR to recognize their surroundings, distinguish between and analyze other vehicles, pedestrians, roadside trees, lanes, etc., and then make decisions on how to control the vehicle. In this case, by integrating high-definition maps that include precise road information and location information for traffic lights, roadside trees, etc. with information detected by sensors, it becomes possible to more accurately determine the vehicle's position. This is also effective when sensor performance is reduced, such as during bad weather.

[0003] From this perspective, as shown in Patent Document 1, an invention has been proposed that estimates the surrounding environment with higher accuracy from observation data obtained using sensing equipment and creates an occupancy grid map that describes the occupancy probability.

[0004] However, since there is no way to determine whether vehicles, people, etc. are unwanted objects, there is a risk that parked vehicles, etc. will be judged as stationary obstacles and registered on the map.

[0005] Furthermore, an invention has been proposed that uses continuously acquired camera images to find moving objects that may become obstacles (Patent Document 2).

[0006] However, methods that use images can only find moving objects within the images. Since the distance sensors used in typical 3D map creation devices measure a wider area than the camera's angle of view to create the 3D map, Patent Document 1 cannot remove unwanted objects outside the angle of view of the image, resulting in unwanted objects remaining in the 3D map. Furthermore, it cannot remove objects that are not moving within a series of images, such as temporarily stopped parked vehicles. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Patent Publication No. 2017-166966 [Patent Document 2] Patent Publication No. 2000-123183 [Non-patent literature]

[0008] [Non-Patent Document 1] https: / / jidounten-lab.com / v_3dmap-autonomous-need3D Maps: "Need vs Don't" - The ruthless battle in the autonomous driving industry: Are dynamic maps necessary? Summary of the Invention [Problem to be solved by the invention]

[0009] One possible method is to use sensors installed in automobiles or mobile vehicles to acquire 3D point clouds of surrounding objects over time and create a 3D map. In order to make the 3D point cloud usable as a map, it is necessary to include only point clouds of primarily fixed objects in the 3D map. When acquiring sensor data, the 3D map also includes point clouds of unwanted objects, so the point clouds must be manually removed from the map. For example, in a road environment, unwanted objects include parked vehicles and people, and in an indoor environment such as a factory, unwanted objects include temporarily placed luggage.

[0010] In view of the above, an object of the present invention is to automate the process of removing unwanted objects outside the camera's angle of view from a three-dimensional map. [Means for solving the problem]

[0011] The present invention provides a three-dimensional map creation device including a distance sensor installed on a mobile object that acquires a three-dimensional point cloud representing the surrounding environment of the mobile object, a camera that acquires an image representing the surrounding environment within a field of view, and a calculation means that creates a three-dimensional map based on the three-dimensional point cloud acquired from the distance sensor and the image acquired from the camera, wherein the calculation means includes an unwanted object detection unit that detects unwanted objects not required for the three-dimensional map based on the image and outputs the detected unwanted objects as unwanted object detection results, a self-location estimation unit that estimates the position of the calculation means, an unwanted object area management map creation unit that creates an unwanted object area management map representing unwanted object areas where the unwanted objects may exist based on the three-dimensional point cloud and the unwanted object detection results, a removal unit that removes specific points from the three-dimensional point cloud using the unwanted object area management map, and a three-dimensional map creation unit that creates a three-dimensional map using the points not removed by the removal unit, thereby enabling unwanted objects not required for the three-dimensional map to be removed even if they are outside the field of view of the camera.

[0012] A "three-dimensional point cloud" refers to a three-dimensional collection of objects generated as points, created based on the distance between the object and the surrounding environment measured by a distance sensor.

[0013] The "camera" includes a general camera used to create three-dimensional maps.

[0014] "Within the camera's angle of view" refers to a specific area within the range, expressed in terms of angle, that is actually captured when the surrounding environment is photographed with a camera.

[0015] A "distance sensor" is a sensor that measures the distance to the surrounding environment.

[0016] "Unnecessary objects not required for the three-dimensional map" are, for example, people, cars, bicycles, and the like in the surrounding environment.

[0017] The "self-position estimation unit" estimates the self-position of the calculation means on a three-dimensional map.

[0018] The "area where the unwanted matter may exist" is a specific area excluding areas where the unwanted matter may not exist.

[0019] The "unwanted object area management map" is a map for managing areas showing point clouds that are determined to be unwanted objects.

[0020] The "specific point cloud" is at least a point cloud corresponding to an unwanted object area. For example, a point cloud managed as a road surface area may also be removed.

[0021] The present invention automates the removal of unwanted objects from 3D maps, which has previously been done manually. It can also remove unwanted objects measured by distance sensors that measure a wider range than cameras.

[0022] A specific aspect of the method for creating an unwanted object area management map is that the unwanted object area management map divides the space for creating the map into grids, and the unwanted object detection unit manages the grids in the space where unwanted objects are determined to exist as the unwanted object area.

[0023] A specific aspect that takes into consideration how to deal with objects of different sizes, such as vehicles and people, is characterized in that the unwanted object area management map creation unit changes the number of grids managed as the unwanted object area depending on the type of unwanted object detected by the unwanted object detection unit.

[0024] A specific aspect of the removal unit that uses unnecessary object areas is that the removal unit removes a group of points that exist on a grid that is managed as the unnecessary object area in the unnecessary object area management map.

[0025] A specific aspect of the removal unit that sets the range of heights to be removed is characterized in that the removal unit sets a height threshold for the height of the point cloud to be managed in the grid managed as the unwanted object area, and removes the point cloud that falls within the range indicated by the height threshold.

[0026] A specific aspect of the road surface confirmed area is that the three-dimensional map creation device further detects the road surface based on the image or the three-dimensional point cloud, and the unnecessary object area management map creation unit manages the grid in which the area detected as the road surface exists as the road surface confirmed area.

[0027] As a specific aspect of using self-location information, the unnecessary object area management map creation is further characterized in that the space that the three-dimensional map creation device has passed through is determined based on the time-series self-location estimated by the self-location estimation unit, and the grid in which the passed space exists is managed as a road surface determined area.

[0028] As a specific aspect of the road surface determined area, the road surface is detected by the unnecessary object area management map creation unit determining an area to be judged as a road surface from the three-dimensional point cloud acquired by the distance sensor.

[0029] A feature of the removal unit that uses unnecessary object areas is that the removal unit removes point clouds that exist on a grid managed as road surface determined areas in the unnecessary object area management map and are at a specified height.

[0030] In a specific aspect of the unnecessary object detection unit, the unnecessary object detection unit further detects unnecessary objects from the three-dimensional point cloud and outputs the unnecessary object detection result.

[0031] A specific aspect of the detection unit that utilizes the feature that a camera has higher detection performance than a distance sensor is that the unwanted object detection unit detects unwanted objects from the image within the field of view of the camera, and detects unwanted objects from the three-dimensional point cloud outside the field of view of the camera.

[0032] A specific aspect of changing the grid size depending on the type of detected object is characterized in that the unwanted object area management map creation unit changes the size of the grid to be managed as the unwanted object area depending on the type of unwanted object detected by the unwanted object detection unit.

[0033] A specific aspect that can be obtained by having a display unit is that the three-dimensional map creation device further comprises a road map and a display unit, and in the created unwanted object area management map, the positions indicated by the grids that are managed as unwanted object areas are displayed on the road map.

[0034] A specific aspect of having an unmeasured area is that the display unit displays on the road map an area that is not determined to be either an unwanted object area or a determined road surface area, and for which a point cloud could not be obtained by the distance sensor, as an unmeasured area.

[0035] As a display mode that utilizes the feature that areas far from roads are unnecessary for autonomous driving, the display unit is characterized in that it displays on the road map the unmeasured areas or the unnecessary object areas that are at a distance shorter than a threshold value from the road on the road map.

[0036] In a specific aspect in which unwanted object areas or unmeasured areas are not displayed when they are not densely packed, the display unit displays on the road map a grid around the unwanted object area or unmeasured area in which there are more unwanted object areas or unmeasured areas than a threshold value.

[0037] The three-dimensional map creation method of the present invention comprises an unwanted object detection step of detecting unwanted objects from an image acquired from a camera and creating an unwanted object detection result; an unwanted object management map creation step of creating an unwanted object management map showing a grid indicating unwanted object areas; an unwanted object point cloud removal step of using the unwanted object management map to remove a point cloud indicating unwanted objects from a three-dimensional point cloud acquired from a distance sensor; and a three-dimensional map creation step of creating a three-dimensional map using the point cloud from which the unwanted objects have been removed. [Brief explanation of the drawings]

[0038] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration according to a first embodiment of the present invention. [Figure 2] FIG. 1 is an explanatory diagram showing an example of a configuration according to a first embodiment of the present invention. [Figure 3] 1 is a block diagram showing an example of a hardware configuration according to a first embodiment of the present invention. [Figure 4] 1 is a flowchart of a first embodiment of the present invention. [Figure 5] FIG. 10 is an explanatory diagram showing an example of an acquired image and a projected point cloud. [Figure 6] FIG. 10 is an explanatory diagram showing an example of an unwanted object detection result. [Figure 7] FIG. 10 is an explanatory diagram showing an example of the unwanted object management map. [Figure 8] FIG. 10 is an explanatory diagram showing an example of the unwanted object management map. [Figure 9] FIG. 10 is a block diagram showing an example of a configuration according to a second embodiment of the present invention. [Figure 10] FIG. 10 is an explanatory diagram for explaining a display image displayed on a display unit according to the second embodiment of the present invention. [Figure 11] FIG. 10 is an explanatory diagram for explaining a display image displayed on a display unit according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0039] A 3D map creation device 1 and a 3D map creation method according to a first embodiment of the present invention will be described below with reference to FIGS.

[0040] As shown in Figure 2, the three-dimensional map creation device 1 is installed on a moving body 100 such as a person, an automobile, or a mobile robot, and is used. It includes a distance sensor 101 that acquires a three-dimensional point cloud 110, a camera 102 that acquires an image 120 shown in Figure 5(A), an unwanted object detection unit 103 that detects unwanted object 404 that is not needed in the three-dimensional map, a self-position estimation unit 104 that estimates the self-position P (see Figures 7 and 8) of the device 1, an unwanted object area management map creation unit 105 that creates an unwanted object area management map 401 that indicates an area where unwanted object 404 (see Figure 7) may exist, a removal unit 106 that removes point clouds 405, 505, 515 (see Figures 7 and 8) that indicate unwanted object 404 from the three-dimensional point cloud 110 acquired by the distance sensor 101 using the unwanted object area management map 401, and a three-dimensional map creation unit 107 that creates a three-dimensional map using the point clouds that were not removed by the removal unit 106.

[0041] The distance sensor 101 measures the distance to objects around the 3D map creation device 1, thereby acquiring a 3D point cloud 110 that indicates the shapes of objects in the surrounding environment, and outputs the information to the unnecessary object area management map creation unit 105 and the removal unit 106. The distance sensor 101 is, for example, a laser distance measuring device. Examples of the distance sensor 101 include various specific examples such as a TOF (Time of Flight) method, such as LiDAR (Light Detection and Ranging), which determines the distance to the object from the time it takes for laser light to travel back and forth.

[0042] The camera 102 acquires an image 120 (see FIG. 5 ) and outputs the image 120 to the unwanted object detection unit 103. The camera 102 and the distance sensor 101 are installed on the mobile object 100 so that the acquisition range of the 3D point cloud 110 and the angle of view 403 of the camera 102 at least partially overlap, so that the 3D point cloud 110 acquired by the distance sensor 101 can be projected onto the image 120. For example, the distance sensor 101 acquires the 3D point cloud 110 of the entire periphery of the 3D map creation device 1, and the camera 102 is installed facing the direction in which the mobile object 100 and the 3D map creation device 1 are moving. The 3D point cloud 110 shown in FIG. 2 is not a point cloud acquired by the distance sensor 101 at a certain time, but a point cloud that is a completed map made up of point clouds acquired at all times. In the following explanation, the 3D point cloud 110 in this embodiment refers to a point cloud at a certain time, and is a 3D point cloud corresponding to the limited environment shown in the lower right of FIG. 2, and will be explained in that sense.

[0043] The unwanted object detection unit 103 detects unwanted objects 404 from the image 120 using a technique such as a neural network.

[0044] The self-position estimation unit 104 estimates the self-position P. For example, a GPS or a measuring device using radio waves (not shown) can be used. Alternatively, the self-position P can be obtained by acquiring sensor data in time series from the distance sensor 101 or the camera 102 and comparing the time series data.

[0045] The unnecessary object area management map creating unit 105 obtains a projected point cloud 130 by projecting the three-dimensional point cloud 110 onto the image 120 acquired by the camera 102, and creates unnecessary object area management maps 401 and 501.

[0046] An example of an arithmetic circuit 200 that controls the creation of a three-dimensional map and the like will now be described with reference to Fig. 3. This arithmetic circuit 200 has a CPU 201, RAM 202, ROM 203, counter 204, timer 205, and input / output interface 209 interconnected via a bus 210. The input / output interface 209 is connected to the distance sensor 101, camera 102, input unit 211, and the like. The CPU 201 performs initial settings or specific calculations upon receiving input information.

[0047] The CPU 201 generates calculation information to be output to each unit and executes 3D map creation by outputting the calculation information under program control. The RAM 202 temporarily reads and writes data for 3D map creation and the like. The ROM 203 stores programs for 3D map creation and the like in read-only mode. The CPU 201 executes calculations related to 3D map creation using data from the distance sensor 101, camera 102, etc. Instead of program control, this can also be implemented by hardware control such as LSI logic.

[0048] The counter 204 functions as a count value for creating a three-dimensional map, and after power is turned on, the initial value of the count value is set to "0" and the counter 204 performs count calculations by referring to various input information.

[0049] The timer 205 performs time calculations related to the creation of three-dimensional maps.

[0050] The input unit 211 may include a mouse, a keyboard, etc. The input unit 211 may be omitted.

[0051] Next, a processing procedure for creating a three-dimensional map by the three-dimensional map creation device 1 for removing three-dimensional point clouds on unnecessary objects and creating a three-dimensional map will be described with reference to the flowchart of FIG.

[0052] When the process starts, information is input from the distance sensor 101, the camera 102, etc. (step 601).

[0053] When the process of step 601 is completed, the unwanted object detection unit 103 detects unwanted objects 404 (step 602).

[0054] In the junk detection step 602 , junk 404 is detected based on the image 120 .

[0055] In this embodiment, the unwanted object detection unit 103 detects unwanted objects 404 from the image 120 because the camera 102 has higher detection performance than the distance sensor 101. However, for locations that are always in a blind spot (outside the angle of view 403) from the camera 102, which is fixed forward in the direction of travel, the unwanted objects 404 can also be detected from the three-dimensional point cloud 110 acquired by the distance sensor 101.

[0056] The unwanted object detection unit 103 detects unwanted objects 404 captured in the input image 120 and outputs an unwanted object detection result 303 indicating an area surrounded by a rectangle as shown in FIG. 6 to the unwanted object area management map creation unit 105. For example, a neural network can be used to detect the unwanted objects 404. The unwanted objects 404 can be detected by having the neural network learn about objects that are the target of unwanted objects 404. The unwanted object detection result 303 indicates the unwanted object area on the image 120 obtained by inputting the image 120. In this example, a car 301 and a person 302 are detected as unwanted objects 404 that are not needed on the three-dimensional map. The unwanted object detection unit 103 outputs not only the position on the image 120 but also the type of the detected unwanted object 404 as the unwanted object detection result 303 to the unwanted object area management map creation unit 105.

[0057] The unnecessary object detection unit 103 also trains the neural network to learn road surface areas, and outputs the road surface areas on the image 120 to the unnecessary object area management map creation unit 105 .

[0058] When the process of step 602 is completed, the process proceeds to step 603, where the unnecessary object area management map creating unit 105 creates the unnecessary object area management map 401 (step 603).

[0059] As shown in Figure 7, the unnecessary object area management map 401 divides the space in which the three-dimensional map is to be created into grids, and manages the grids 406 in the space in which the unnecessary object detection unit 103 determines that unnecessary object 404 exists as unnecessary object areas U.

[0060] 7, the unnecessary object area management map creation unit 105 divides the space within which the three-dimensional map is to be created into a grid, and creates an unnecessary object area management map 401 that manages whether or not unnecessary object 404 may exist in the grid-like space at a certain time t. Fig. 8 shows an example of an unnecessary object area management map 501 managed by the three-dimensional map creation device 1 at a certain time t+1, which is after the time t.

[0061] 5(A) and 6, a car 301 and a person 302 are captured as unwanted objects 404 within the angle of view 403 of the camera 102. As shown in Fig. 7, the unwanted object area management map creation unit 105 first creates a projected point cloud 130 from the three-dimensional point cloud 110 and extracts a three-dimensional point cloud 405 included in the unwanted object 404. Next, the position of a grid 406 in which the three-dimensional point cloud 405 exists is calculated from the self-position P estimated by the self-position estimation unit 104 and the position of the three-dimensional point cloud 405. The unwanted object area management map creation unit 105 manages the grid at the calculated position as the grid 406 in which the unwanted object 404 may exist.

[0062] The unwanted object area management map creation unit 105 can eliminate point clouds obtained from the same unwanted object 404 that are farther away from the center of the point cloud, for example, by a threshold value or more, based on the representative value of the point cloud 405, and can also exclude them from being considered as grids in which the unwanted object 404 may exist. The representative value can be selected by selecting the median, average, or shortest distance to each point cloud 405. Furthermore, if the type of unwanted object 404 is obtained from the unwanted object detection unit 103, the representative value can be selected according to the type of unwanted object 404. For example, if the unwanted object 404 is detected as a person 302, the representative value can be set to a short value, and if the unwanted object 404 is detected as a car 301, the representative value can be set to a long value. For example, for the car at the right end of Figures 5(A) and (B), the background point cloud included in the rectangular area is located farther away than the car 301. Therefore, a representative value, such as the average distance of the point cloud included in this rectangle, is calculated, and points farther away from the representative value by a threshold value or more than the representative value are not removed. Grids containing point clouds that have not been removed are not managed as unwanted object areas.

[0063] The unwanted object area management map creation unit 105 can also manage grids that are closer than the management threshold to the point cloud 405 managed as unwanted object 404 as grids where unwanted object 404 may exist. If the type of unwanted object 404 is obtained from the unwanted object detection unit 103, the management threshold can also be set according to the type. For example, if it is detected as a person 302, the management threshold is set short, and if it is detected as a car 301, the management threshold is set long. Although the point cloud behind the car 301 is not obtained in the projection point cloud 120, the type "car" is known, so grids in a range of sizes likely to be for a car are managed as unwanted object areas, but in the case of a person, the management threshold is set short.

[0064] The unwanted object area management map creation unit 105 can also detect individual unwanted objects 404 from the three-dimensional point cloud 110 measured by the distance sensor 101, and extract the point cloud of the unwanted objects 404. However, for point clouds within a range (within the angle of view 403) that can be projected onto the image 120 that can be acquired by the camera 102, the unwanted object detection result 303 from the unwanted object detection unit 103 takes priority.

[0065] The unwanted object area management map creation unit 105 can also create multiple unwanted object area management maps 401, 501 with different grid sizes depending on the type of detected unwanted object 404. For example, if people 302 and automobiles 301 are targeted as unwanted objects 404, it is possible to have an unwanted object area management map for people and an unwanted object management map for automobiles, with the grid size of the unwanted object area management map for people being smaller than the grid size for automobiles.

[0066] The unwanted object area management map creation unit 105 can change the number and / or size of the grids managed as unwanted object areas depending on the type of unwanted object 404 detected by the unwanted object detection unit 103. This is to take into consideration how to handle objects of different sizes, such as automobiles 301 and people 302.

[0067] When the unnecessary object area management map creation unit 105 obtains a road surface area (see Figure 5(A)) from the unnecessary object detection unit 103, it manages the grid in which the point cloud 409 (see Figure 5(B)) projected onto the road surface area exists as a road surface determined area R.

[0068] The unnecessary object area management map creation unit 105 can extract points from the three-dimensional point cloud 110 whose height is lower than a set threshold as a road surface point cloud, and can also manage the grid in which the road surface point cloud 409 (see Figure 5 (B)) is located as a road surface determined area R.

[0069] In addition, the unnecessary object area management map creation unit 105 can manage the grid within the range of the size of the three-dimensional map creation device 1 or the mobile body 100 on which the three-dimensional map creation device 1 is mounted as the road surface determination area R.

[0070] The unnecessary object area management map creation unit 105 determines the space that the three-dimensional map creation device 1 has passed through based on the self-position P at time series t and t+1 estimated by the self-position estimation unit 104, and can manage the grid in which the passed space exists as a road surface determination area R.

[0071] The three-dimensional map creation device 1 detects the road surface as described above, and the unnecessary object area management map creation unit 105 manages, as a road surface determined area R, the grid in which the area detected as the road surface exists.

[0072] When the unwanted object area management map creation unit 105 calculates a grid that will become a road surface confirmed area R using the currently acquired point cloud, if the grid has previously been managed as a grid where unwanted object 404 may exist, it updates the grid to be managed as a road surface confirmed area R.

[0073] When the process of step 603 is completed, the process proceeds to step 604, where the removal unit 106 performs a process of removing the point cloud 405 of the unnecessary object 404 (step 604).

[0074] In step 604, using the unwanted object area management maps 401, 501 and the unwanted object detection results 303, the point cloud 405 existing on the grid managed as the unwanted object existence area U and the road surface confirmation area R is removed from the acquired three-dimensional point cloud 110 based on predetermined criteria.

[0075] FIG. 8 shows an example of a junk object area management map 501 and a point cloud created at time t+1, the time after FIG. 7 . A three-dimensional point cloud 505 shows a point cloud newly extracted as junk object 404 at time t+1, when the junk object management map 501 in FIG. 8 was created. Since the junk object management map 501 at time t+1 basically inherits the grid information managed in the junk object management map 401 at time t, it can be seen that grids located in positions managed as junk object area U and road surface area R in 401 are still managed as junk object area U and road surface area R. Furthermore, because the information of the newly extracted point cloud can be used, it can also be seen that the number of grids managed as junk object area U is increasing. A three-dimensional point cloud 515 shows a point cloud that will be removed at time t+1, when the junk object management map in FIG. 8 was created, because the grids are located on grid 506 managed as junk object area U at time t or grid 508 managed as determined road surface area R. By using the unwanted object management map 501, it is possible to remove point clouds 515 of unwanted objects 404 outside the camera's angle of view 403. One advantage is that unwanted objects 404 can be removed without needing to predict the movement of the unwanted objects by managing them by location rather than by object.

[0076] A three-dimensional map creation unit 107 creates a three-dimensional map by overlapping the point clouds that were not removed by the removal unit 106 in time series.

[0077] The removal unit 106 removes the three-dimensional point clouds that exist on the grids 408, 508 managed as the road surface determined area R in the unnecessary object area management maps 401, 501 and are at a specified height.

[0078] The removal unit 106 also removes point clouds extracted as unwanted objects 404 by the unwanted object detection unit 103 from the three-dimensional point cloud 110 acquired by the distance sensor 101. Furthermore, among point clouds on grids 406 managed as unwanted object areas U on the unwanted object area management maps 401, 501 or grids managed as road surface determined areas R, point clouds located at positions higher than the position of point clouds determined to be road surfaces are removed. A height threshold can be set for the point clouds to be removed based on the unwanted object area management maps 401, 501, and point clouds located higher than the position of point clouds determined to be road surfaces but lower than the height threshold can be removed. The height threshold can also be set for each grid.

[0079] When the process of step 604 is completed, the process proceeds to step 605, where a three-dimensional map is created (step 605).

[0080] In step 605 of creating a three-dimensional map, a three-dimensional map is created using the point cloud from which the unnecessary objects 404 have been removed. When this process is completed, the three-dimensional map creation process ends. This process is performed at specific time intervals.

[0081] In this first embodiment, a road environment will be described as an example, but in environments where indoor and outdoor environments coexist, such as theme parks and factories, or indoor environments such as retail stores, it is possible to create a three-dimensional map from which unwanted objects 404 have been removed by predetermining the objects to be detected as unwanted objects 404.

[0082] As described above, according to the first embodiment, it is possible to automate the removal of unwanted objects 404 from a three-dimensional map, which has previously been done manually. It is also possible to remove unwanted objects measured by the distance sensor 101, which measures a wider range than the camera 102.

[0083] A 3D map device 7 according to a second embodiment of the present invention will be described below with reference to Figs. 9 and 10. In the second embodiment, a road map 708 and a display unit 709 are added to the configuration of the first embodiment. Elements similar to those in the first embodiment are designated by numbers in the 700 range.

[0084] The three-dimensional map creation device 7 comprises the three-dimensional map creation device 1 of embodiment 1, a road map 708, and a display unit 709, and displays on the road map 708 the positions indicated by the grids managed as unwanted object areas U in the created unwanted object area management maps 401, 501.

[0085] The display unit 709 further displays on the road map 708, as an unmeasured area N, an area that is not classified as either an unwanted object area U or a determined road surface area R, and for which no point cloud was obtained by the distance sensor 701.

[0086] The display unit 709 acquires a road map corresponding to the three-dimensional map created by the three-dimensional map creation unit 707 from the road map 708. A road map is expressed, for example, as a two-dimensional top view of a road observed from above. Similarly, when a three-dimensional map is observed from an aerial perspective, it can be expressed as a two-dimensional top view, and a correspondence can be established between the road map and the three-dimensional map. Similarly, a correspondence can also be established between an unnecessary object area management map, which corresponds to the three-dimensional map. If the coordinate system of the three-dimensional map creation unit 707 matches the coordinate system of the road map 708, the corresponding road map 708 is acquired based on the coordinates, and if the coordinate systems do not match, the road map 708 is acquired using separately specified coordinates.

[0087] 10, the display unit 709 displays, on the road map 708, unmeasured areas N or areas where unnecessary objects exist that are located at a distance from the road 803 on the road map 708 that is shorter than the threshold value. This utilizes the feature that areas far from the road 803 are not necessary for automated driving.

[0088] The display unit 709 associates the unwanted object area management map 801 with the road map 708, and as shown in Fig. 10, among the grids that are not managed as either unwanted object areas or road surface areas R on the unwanted object area management map 801, manages grids 805 that do not contain point clouds on the three-dimensional map as unmeasured areas N. The display unit 709 creates an image in which the positions of the grids 805 managed as unmeasured areas N and the grids 806 managed as unwanted object areas (see Fig. 11) on the road map 708 are indicated in a pre-specified color, and displays the created image as a display image 802. The pre-specified color may be changed for the positions managed as unmeasured areas N and the positions managed as unwanted object areas.

[0089] Display image 802 is an image created using an unwanted object area management map 801, which includes areas managed as unmeasured areas N, and the road map 708. In the example of display image 802 shown in Fig. 10, roads 803 and stationary objects 804 included in the road map 708 are indicated in different colors, as well as the positions of grid points managed as unwanted object areas in the unwanted object area management map 801 and the positions of grid points 805 managed as unmeasured areas N.

[0090] The display unit 709 can also indicate in a specified color only the grids managed as unwanted object areas and the grids 805 managed as unmeasured areas N within a specified range from the road.

[0091] 11, the display unit 709 can also indicate in a specified color when the number of lattices managed as unnecessary object areas U and unmeasured object areas N around a lattice 806 managed as an unnecessary object area U and a lattice 805 managed as an unmeasured object area N is greater than a threshold. In other words, it can also indicate in a specified color a location where lattices managed as unnecessary object areas and lattices managed as unmeasured object areas N are concentrated. This makes it possible to visualize key locations that require remeasurement on the map, thereby making the creation of three-dimensional maps more efficient.

[0092] 11, the display unit 709 can also display on the road map 708 only grids 901 where there are more grids 806 determined to be unwanted object regions U or grids 805 determined to be unmeasured regions N around the grids 806 determined to be unwanted object regions U or grids 805 determined to be unmeasured regions N than a threshold value. Conversely, it is possible not to display grids 902 where there are fewer grids than the threshold value.

[0093] The second embodiment not only achieves the same effects as the first embodiment, but also makes it possible to visualize locations that require remeasurement. Therefore, by utilizing the three-dimensional map and the road map, the creation of the three-dimensional map can be made even more efficient, thereby making autonomous driving even more efficient. [Industrial Applicability]

[0094] Since it is possible to remove unwanted objects outside the angle of view 403 of the camera 102, the efficiency of creating three-dimensional maps can be improved, which contributes to the development of autonomous driving and other technologies, and is of great industrial value. [Explanation of symbols]

[0095] 1. 3D map creation device 7. 3D map creation device 100 Mobile 101 Distance Sensor 102 Camera 103 Unwanted object detection unit 104 Self-position estimation part 105 Unwanted Area Management Map Creation Department 106 Removal section 107 3D Map Creation Department 110 3D point cloud 120 images 130 Projection point cloud 200 Arithmetic circuit 301 Automobiles 302 people 303 Unwanted object detection results 401 Unwanted Area Management Map 403 angle of view 404 Unwanted items 405 3D point cloud 406 Lattice 407 Road 408 Grid 409 point group 505 point group 515 point group 501 Unnecessary object domain management area 502 point group 506 grid 707 Three-dimensional Earth Creation Department 709 Display Department 802 represents portrait 803 Road 804 Stationary Object 805 Grid P own position R Road surface determination area U Don't want the field Time t, t+1

Claims

1. a distance sensor installed on a moving body for acquiring a three-dimensional point cloud representing the surrounding environment of the moving body; a camera for acquiring an image representing the surrounding environment within a field of view; and a calculation means for creating a three-dimensional map based on the three-dimensional point cloud acquired from the distance sensor and the image acquired from the camera, The calculation means an unwanted object detection unit that detects unwanted objects that are not required for the three-dimensional map based on the image and outputs the detected unwanted objects as unwanted object detection results; a self-position estimation unit that estimates the position of the moving object; an unwanted object area management map creation unit that creates an unwanted object area management map that indicates an unwanted object area that is an area where the unwanted object may exist, based on the three-dimensional point cloud acquired in time series and the unwanted object detection result; a removal unit that removes specific points from the three-dimensional point cloud using the unnecessary object area management map; a three-dimensional map creation unit that creates a three-dimensional map using the point cloud that was not removed by the removal unit; A three-dimensional map creation device comprising:

2. The three-dimensional map creation device according to claim 1, characterized in that the unnecessary object area management map divides the space in which the map is to be created into a grid, and manages the grid in the space in which the unnecessary object detection unit determines that unnecessary objects exist as the unnecessary object area.

3. 3. The three-dimensional map creation device according to claim 2, wherein the unnecessary object area management map creation unit changes the number of grids to be managed as the unnecessary object area depending on the type of unnecessary object detected by the unnecessary object detection unit.

4. 3. The three-dimensional map creation device according to claim 2, wherein the removal unit removes a group of points existing on a grid managed as the unnecessary object area in the unnecessary object area management map.

5. The 3D map creation device according to claim 4, characterized in that the removal unit sets a height threshold for the height of the point cloud to be managed in the grid managed as the unnecessary object area, and removes the point cloud included in the range indicated by the height threshold.

6. The three-dimensional map creation device according to claim 2, characterized in that the three-dimensional map creation device further detects a road surface based on the image or the three-dimensional point cloud, and the unnecessary object area management map creation unit manages a grid in which an area detected as a road surface exists as a confirmed road surface area.

7. The three-dimensional map creation device according to claim 2, characterized in that the unnecessary object area management map creation unit further determines a space that the three-dimensional map creation device has passed through based on the time-series self-position estimated by the self-position estimation unit, and manages a grid in which the passed space exists as a determined road surface area.

8. The three-dimensional map creation device according to claim 6, characterized in that the unnecessary object area management map creation unit detects the road surface by determining an area to be judged as a road surface from the three-dimensional point cloud acquired by the distance sensor.

9. The three-dimensional map creation device according to claim 6, characterized in that the removal unit removes a point cloud that exists on a grid managed as the road surface determined area in the unnecessary object area management map and is at a specified height.

10. 2. The 3D map creation device according to claim 1, wherein the unnecessary object detection unit further detects the unnecessary object from the 3D point cloud and outputs the detected unnecessary object as the unnecessary object detection result.

11. 11. The 3D map creation device according to claim 10, wherein the unwanted object detection unit detects unwanted objects from the image within the field of view of the camera, and detects unwanted objects from the 3D point cloud outside the field of view of the camera.

12. The three-dimensional map creation device according to claim 2, characterized in that the unnecessary object area management map creation unit manages the area of the three-dimensional map using a grid, and changes the size of the grid managed as the unnecessary object area depending on the type of unnecessary object detected by the unnecessary object detection unit.

13. The three-dimensional map creation device according to any one of claims 1 to 12, characterized in that the three-dimensional map creation device manages areas of the three-dimensional map using a grid, and further comprises a road map and a display unit, and in the created unnecessary object area management map, the positions indicated by the grid managed as unnecessary object areas are displayed on the road map.

14. The three-dimensional map creation device manages the areas of the three-dimensional map using a grid, and further comprises a road map and a display unit, and in the created unwanted object area management map, the positions indicated by the grid that are managed as unwanted object areas are displayed on the road map, and the display unit displays, as unmeasured areas, areas that are not designated as unwanted object areas or determined road surface areas and for which no point cloud was obtained by the distance sensor, on the road map, as unmeasured areas. A three-dimensional map creation device as described in claim 6 or 7, characterized in that

15. 15. The three-dimensional map creation device according to claim 14, wherein the display unit displays, on the road map, the unmeasured area or the unwanted object area that is at a distance from the road on the road map that is shorter than a threshold value.

16. 15. The three-dimensional map creation device according to claim 14, wherein the display unit displays on the road map a grid in which the number of grids that are determined to be the unnecessary object area or the unmeasured area around the unnecessary object area or the unmeasured area is greater than a threshold value.

17. A method for creating a three-dimensional map, in which a computer executes the following steps: an unwanted object detection step of detecting unwanted objects from an image acquired from the camera and creating an unwanted object detection result; a waste management map creation step of creating a waste management map showing a grid indicating a waste area based on the three-dimensional point cloud acquired from the distance sensor and the waste detection result; an unwanted object point cloud removal step of removing a point cloud indicating unwanted objects from the three-dimensional point cloud acquired from the range sensor by using the unwanted object management map; a three-dimensional map creation step of creating a three-dimensional map using the point cloud from which the point cloud indicating the unnecessary objects has been removed in the unnecessary object point cloud removal step; A three-dimensional map creation method comprising:

Images