Deep learning device and travel body

Abstract

To provide a deep learning device which can stably detect the presence / absence of obstacles on the front side in the travel direction of a travel body and a travel body.SOLUTION: In a surrounding environment recognition unit 30, a vegetation index calculation unit 31 calculates vegetation index data on the basis of the omnidirectional image data from an omnidirectional camera 13 and three-dimensional measurement data from a 3D-LiDAR 12, a three-dimensional point cloud region division unit 32 performs region division processing of the three-dimensional point cloud on the basis of the three-dimensional measurement data from the 3D-LiDAR 12 and the vegetation index data from the vegetation index calculation unit 31, and outputs point cloud region division data, and a control processing unit 33 processes self-position estimation, cost map calculation, path planning and the like on the basis of the three-dimensional measurement data from the 3D-LiDAR 12 and the point cloud region division data from the surrounding environment recognition unit 30 and outputs control signals to control the travel of a mobile robot 1.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a deep learning device and a moving object. [Background technology]

[0002] The mobile robot (vehicle) is equipped with a 3D-LiDAR (Three-Dimensional-Light Detection And Ranging) to measure the 3D point cloud in all directions around it, and compares the shape of the 3D point cloud with a 3D point cloud map measured in advance to estimate its own position while moving. If there is an obstacle ahead while the mobile robot is moving, it will detect the obstacle based on the information measured by the 3D-LiDAR and avoid the obstacle.

[0003] For example, there is a cost map method for detecting obstacles based on information measured by 3D-LiDAR (see Non-Patent Document 1). As in Non-Patent Document 1, an occupancy grid map is created from the position information of the measured 3D point cloud, and a 2D cost map is generated. This increases the cost of areas where objects are located higher than the road surface, and it is possible to detect areas with high costs as obstacles.

[0004] There is also a method for detecting obstacles by dividing a three-dimensional point cloud measured by a 3D-LiDAR into regions and recognizing which region each three-dimensional point belongs to (see Non-Patent Document 2). In three-dimensional point cloud region division, each point is classified into regions such as automobiles, bicycles, roads, parking lots, buildings, vegetation, trunks, and terrain. On the mobile robot side, regions other than drivable regions such as roads and parking lots are detected as obstacles. In addition, by reflecting the obstacle detection results in the cost map of Non-Patent Document 1, it is easy to make the region detected as an obstacle a high-cost area. The method of Non-Patent Document 2 is one of the methods for dividing a three-dimensional point cloud region by deep learning, and since its announcement in 2020, it has maintained a top performance in the field of point cloud region division, where research and development is fiercely competitive. Non-Patent Document 2 targets automatic driving of automobiles and is a technology that is well suited for application to mobile robots.

[0005] In addition, although it is not a method for 3D point clouds, there is a method for detecting obstacles by performing area division on an image taken by a color camera in front of a mobile robot (see Non-Patent Document 3). In image area division, each pixel on the image is classified into areas such as automobiles, bicycles, roads, parking lots, buildings, vegetation, trunks, and terrain. On the mobile robot side, areas other than drivable areas such as roads and parking lots are considered as obstacles, and the three-dimensional position of the obstacle is obtained by converting the camera coordinates into three-dimensional space, and the position of the obstacle is detected. The method in Non-Patent Document 3 is one of the methods for performing image area division using deep learning, and since its announcement in 2017, it has been a representative method in this field. Non-Patent Document 3 targets automatic driving of automobiles, and is a technology that is well suited for application to mobile robots. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yoshitaka Hara: "Autonomous Driving Using ROS", Journal of the Robotics Society of Japan, vol. 35, no. 4, pp286-290, 2017,URL<https: / / www.jstage.jst.go.jp / article / jrsj / 35 / 4 / 35_35_286 / _pdf / -char / ja> [Non-Patent Document 2] C.Tiago, et al., "SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Cloudes for Autonomous Driving", arViv, 2020, URL<https: / / arxiv.org / pdf / 2003.03653.pdf> [Non-Patent Document 3] H. Zhao, et al., "Pyramid Scene Parsing Network", arViv, 2017,URL<https: / / arxiv.org / pdf / 1612.01105.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] However, the method of Non-Patent Document 1 can detect tall objects as obstacles, but has a problem in that it cannot detect vegetation such as grass with little difference in height as an obstacle, such as a sidewalk in a park where the sidewalk is surrounded by grass. If vegetation such as grass with little difference in height cannot be detected as an obstacle, the mobile robot cannot avoid low obstacles, and for example, when traveling in a park, there is a problem that the mobile robot may enter grass where entry is prohibited.

[0008] In addition, while non-patent document 2 makes it possible to distinguish between sidewalks and grass to a certain extent, the problem is that the 3D point cloud alone does not provide enough information for area segmentation, and parts of roads and parking lots are often mistakenly detected as vegetation.

[0009] In addition, the method in Non-Patent Document 3 recognizes the surrounding environment based on color information, and therefore has better area segmentation performance than the method in Non-Patent Document 2. However, there are problems with this method, such as the influence of brightness fluctuations and the possibility of false detection of a different object when the sidewalk is colored in various colors such as green or blue.

[0010] In addition, in Non-Patent Document 2 or Non-Patent Document 3, if a mobile robot mistakenly detects an obstacle that does not exist ahead, it may take abnormal evasive action in an attempt to avoid the obstacle, or it may be unable to find an avoidance route and become unable to move.

[0011] Therefore, an objective of the present invention is to provide a deep learning device and a running body that can stably detect the presence or absence of an obstacle ahead in the direction of travel of the running body. [Means for solving the problem]

[0012] The deep learning device of the first invention for solving the above problem is characterized in that, in the first invention, it comprises a memory unit that stores three-dimensional measurement data including three-dimensional information of the surrounding environment of a moving body, omnidirectional image data including an image of the surrounding environment in all directions, and point cloud area label data created based on the three-dimensional measurement data, a vegetation index calculation unit that calculates vegetation index data indicating the state of vegetation in the surrounding environment based on the three-dimensional measurement data and the omnidirectional image data, and a three-dimensional point cloud area segmentation learning unit that performs deep learning using the three-dimensional measurement data, the vegetation index data, and the point cloud area label data as learning datasets, and creates learning result data.

[0013] In addition, a deep learning device according to a second invention for solving the above problem is characterized in that the memory unit stores the three-dimensional measurement data, the omnidirectional image data, and image region label data created based on the omnidirectional image data in place of the point cloud region label data, and further includes a point cloud region label creation unit that creates the point cloud region label data based on the three-dimensional measurement data and the image region label data.

[0014] In addition, a third invention for solving the above-mentioned problems provides a moving body that recognizes its surrounding environment and moves by a moving means, and is characterized in that it comprises a three-dimensional measurement unit that acquires three-dimensional measurement data including three-dimensional information of the surrounding environment, an imaging unit that acquires image data of the surrounding environment, a vegetation index calculation unit that calculates vegetation index data indicating the state of vegetation in the surrounding environment based on the three-dimensional measurement data and the image data, a three-dimensional point cloud area division unit that generates point cloud area division data based on the three-dimensional measurement data and the vegetation index data, and a control processing unit that calculates a cost map of the surrounding environment based on the three-dimensional measurement data and the point cloud area division data, and controls the moving of the moving means based on the cost map.

[0015] In addition, the running body of the fourth invention for solving the above-mentioned problems is characterized in that, in the third invention, the imaging unit is an omnidirectional camera that captures images of the surrounding environment in all directions, and the image data is omnidirectional image data captured by the omnidirectional camera.

[0016] In addition, the moving body of the fifth invention for solving the above problem is characterized in that, in the fourth invention, the omnidirectional cameras are fisheye cameras installed both before and after the moving body in the direction of travel.

[0017] In addition, the running body of the sixth invention for solving the above-mentioned problems is characterized in that, in the third invention, the imaging unit is a color camera that captures an image ahead in the direction of travel of the running body, and the image data is color image data captured by the color camera.

[0018] In addition, the moving body of the seventh invention for solving the above-mentioned problems is characterized in that, in the third invention, the imaging unit is a monochrome camera equipped with a red transmission filter on the front lens and capturing images of the area ahead in the direction of travel of the moving body, and the image data is gray image data captured by the monochrome camera. Effect of the Invention

[0019] According to the present invention, it is possible to stably detect the presence or absence of an obstacle ahead in the traveling direction of the traveling body. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram showing an example of the configuration of a mobile robot 1 according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a block diagram showing a typical configuration example for performing control processing of a mobile robot based only on three-dimensional measurement data from a 3D-LiDAR. [Diagram 3] 2 is a block diagram showing an example of the configuration of a controller 11 according to the first embodiment. FIG. [Figure 4] 4 is a flowchart for explaining the overall operation (surrounding environment recognition processing) of the surrounding environment recognition unit 30 according to the first embodiment. [Diagram 5] 5 is a flowchart for explaining the operation of a vegetation index calculation unit 31 according to the first embodiment. [Figure 6] FIG. 2 is a schematic diagram showing the relationship between a world coordinate system and an image coordinate system in a fisheye projection model in the first embodiment. [Figure 7] FIG. 11 is a conceptual diagram for explaining the coordinate conversion calculation (S24) of a three-dimensional point according to the first embodiment. [Figure 8] FIG. 2 is a conceptual diagram for explaining storage of vegetation index data according to the first embodiment. [Figure 9] 10 is a flowchart for explaining the operation of a three-dimensional point cloud area dividing unit 32 according to the first embodiment. [Figure 10] FIG. 1 is a conceptual diagram for explaining a general area division calculation. [Figure 11] FIG. 2 is a conceptual diagram for explaining region division calculation according to the first embodiment. [Figure 12] FIG. 2 is a conceptual diagram for explaining point cloud region division data creation according to the first embodiment. [Figure 13] FIG. 4 is a conceptual diagram for explaining an example of a cost map used in the first embodiment. [Figure 14] 11 is a schematic diagram showing a configuration example of a mobile robot 1 according to the second embodiment. FIG. [Figure 15] 11 is a block diagram showing an example of the configuration of a controller 11 according to the second embodiment. FIG. [Figure 16] 10 is a flowchart for explaining the operation of a vegetation index calculation unit 34 according to the second embodiment. [Figure 17] FIG. 11 is a conceptual diagram showing the relationship between a world coordinate system and an image coordinate system in a perspective projection camera model in the second embodiment. [Figure 18] 11 is a schematic diagram showing a configuration example of a mobile robot 1 according to the third embodiment. FIG. [Figure 19]13 is a block diagram showing an example of the configuration of a controller 11 according to the third embodiment. FIG. [Figure 20] 10 is a flowchart for explaining the operation of a vegetation index calculation unit 35 according to the third embodiment. [Figure 21] FIG. 13 is a conceptual diagram showing an overview of storage of vegetation index data in the third embodiment. [Figure 22] FIG. 11 is a block diagram showing the configuration of a three-dimensional point cloud region segmentation learning device 80 according to the fourth embodiment. [Diagram 23] 13 is a flowchart for explaining the overall operation of a three-dimensional point cloud region segmentation learning device 80 according to the fourth embodiment. [Figure 24] 13 is a flowchart for explaining the operation of a three-dimensional point cloud region division learning unit 83 according to the fourth embodiment. [Diagram 25] FIG. 13 is a conceptual diagram for explaining learning by a deep learning network according to the fourth embodiment. [Figure 26] FIG. 13 is a block diagram showing the configuration of a three-dimensional point cloud region segmentation learning device 100 according to the fifth embodiment. [Figure 27] 13 is a flowchart for explaining the overall operation of the three-dimensional point cloud region segmentation learning device 100 according to the fifth embodiment. [Figure 28] 13 is a flowchart for explaining the operation of a point cloud area label creation unit 103 according to the fifth embodiment. [Figure 29] FIG. 13 is a conceptual diagram showing an overview of storing point cloud area label data in the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

[0022] A. First embodiment The mobile robot according to the first embodiment basically moves by measuring the surrounding environment in three dimensions and estimating its own position (see Non-Patent Document 1). In addition, in the first embodiment, three-dimensional point cloud area division is used to detect obstacles with little height difference (see Non-Patent Document 2).

[0023] Furthermore, in this first embodiment, in order to improve the performance of the above three-dimensional point cloud region segmentation, visible light information is used to assist the region segmentation process. If color information is simply added, the region segmentation results will be unstable due to the influence of brightness fluctuations, shadows, road surface coloring, etc. Therefore, the normalized difference vegetation index (NDVI) calculated from information in the near-infrared region and visible light region is used. Note that hereinafter, this will also be referred to simply as vegetation index or vegetation index data. The normalized difference vegetation index NDVI is explained on the Geospatial Information Authority of Japan website (Reference 1: Geospatial Information Authority of Japan, "About Vegetation Index Data", URL<https: / / www.gsi.go.jp / kankyochiri / ndvi.html> ) The Normalized Difference Vegetation Index (NDVI) is a representative index that utilizes the characteristics of light reflection by plants to grasp the condition of vegetation, such as the amount of vegetation and its vitality.

[0024] Fig. 1 is a schematic diagram showing an example of a configuration of a mobile robot 1 according to the first embodiment. In Fig. 1, the mobile robot (running body) 1 includes a mobile dolly 10, a controller 11, a 3D-LiDAR (three-dimensional measurement unit) 12, and an omnidirectional camera (imaging unit) 13. The mobile dolly 10 includes a dolly unit 15 including running means 14 including wheels or the like, and a vertical unit 16 extending vertically from the dolly unit 15 to a predetermined height. The dolly unit 15 includes a controller 11, and the vertical unit 16 includes a 3D-LiDAR 12 and an omnidirectional camera 13. The mobile dolly 10 drives the running means 14 in response to a control signal from the controller 11 to run.

[0025] The controller 11 recognizes the surrounding environment based on information measured by the 3D-LiDAR 12 and the omnidirectional camera 13, and controls the movement of the mobile robot 1 based on the recognition result. The control process is equipped with various functions required for controlling the mobile robot, such as self-position estimation, cost map calculation, and path planning. The 3D-LiDAR 12 performs omnidirectional three-dimensional measurement of the surrounding environment of the mobile robot 1, and outputs three-dimensional measurement data including three-dimensional information of the surrounding environment (including distance to objects, shape, etc.) to the controller 11. The omnidirectional camera 13 captures omnidirectional images of the surroundings of the mobile robot 1, and outputs the omnidirectional image data to the controller 11.

[0026] Fig. 2 is a block diagram showing a typical configuration example in which control processing of a mobile robot is performed only from three-dimensional measurement data of a 3D-LiDAR. Fig. 3 is a block diagram showing a typical configuration example of a controller 11 according to the first embodiment. As shown in Fig. 2, in the typical configuration, a control processing unit 21 in a controller 20 processes self-position estimation, cost map calculation, route planning, and other basic functions based on three-dimensional measurement data from a 3D-LiDAR 12, and generates and outputs control signals for driving a traveling means of a mobile robot 1.

[0027] As one of its functions, the controller 11 according to the first embodiment is equipped with a surrounding environment recognition unit 30 as shown in Fig. 3 in addition to the general configuration shown in Fig. 2. The surrounding environment recognition unit 30 includes a vegetation index calculation unit 31 and a three-dimensional point cloud region division unit 32. The vegetation index calculation unit 31 calculates a vegetation index based on omnidirectional image data from the omnidirectional camera 13 and three-dimensional measurement data from the 3D-LiDAR 12, and supplies the vegetation index data to the three-dimensional point cloud region division unit 32. The three-dimensional point cloud region division unit 32 performs region division processing of the three-dimensional point cloud based on the three-dimensional measurement data from the 3D-LiDAR 12 and the vegetation index data from the vegetation index calculation unit 31, and outputs the point cloud region division data. The control processing unit 33 in this first embodiment is basically equivalent to a general control processing unit 21, but differs in function in that it processes self-position estimation, cost map calculation, route planning, and other basic functions based on both the three-dimensional measurement data from the 3D-LiDAR 12 and the point cloud area segmentation data from the surrounding environment recognition unit 30.

[0028] Next, the operation of the surrounding environment recognition unit 30 according to the first embodiment will be described. 4 is a flowchart for explaining the overall operation (surrounding environment recognition processing) of the surrounding environment recognition unit 30 according to the first embodiment. In the surrounding environment recognition unit 30, the vegetation index calculation unit 31 executes a vegetation index calculation processing for calculating vegetation index data based on the omnidirectional image data from the omnidirectional camera 13 and the three-dimensional measurement data from the 3D-LiDAR 12 (step S10). Next, the three-dimensional point cloud region division unit 32 executes a three-dimensional point cloud region division processing for obtaining point cloud region division data based on the three-dimensional measurement data from the 3D-LiDAR 12 and the vegetation index data (step S12).

[0029] Next, the above-mentioned vegetation index calculation process (S10) will be described. FIG. 5 is a flowchart for explaining the operation of the vegetation index calculation unit 31 according to the first embodiment. The vegetation index calculation unit 31 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S20), and inputs omnidirectional image data from the omnidirectional camera 13 (step S22). Next, the vegetation index calculation unit 31 performs three-dimensional point coordinate conversion calculation for all three-dimensional points obtained from the input three-dimensional measurement data (step S24). Next, the vegetation index calculation unit 31 performs three-dimensional point:pixel correspondence calculation based on the omnidirectional image data (step S26), and extracts a red luminance value (step S28). Next, the vegetation index calculation unit 31 calculates a vegetation index value based on the red luminance value (step S30), and saves the vegetation index data (step S32). In this way, the vegetation index calculation unit 31 creates vegetation index data by sequentially performing a series of processes. Thereafter, it is determined whether or not all three-dimensional points have been processed (step S34), and if there are unprocessed three-dimensional points (NO in step S34), the process returns to step S24 and repeats the above-mentioned process. On the other hand, if all three-dimensional points have been processed (YES in step S34), the process ends.

[0030] Next, the calculation of the vegetation index value (S30) described above will be explained. In the present first embodiment, a normalized difference vegetation index (NDVI) is used as the vegetation index value. The normalized difference vegetation index (NDVI) is described in detail in the above-mentioned Reference 1. The normalized difference vegetation index (NDVI) can be calculated from a reflectance IR in the near-infrared region and a reflectance R in the visible red region. In the present first embodiment, the laser reflection intensity of the 3D-LiDAR 12 is used as the reflectance IR in the near-infrared region. The laser of the 3D-LiDAR 12 uses light with a wavelength in the near-infrared region. In addition, the red luminance value of the color information of the pixel of the omnidirectional image data by the omnidirectional camera 13 is used as the reflectance R in the visible red region.

[0031] The normalized difference vegetation index (NDVI) can be calculated using the following formula:

[0032]

number

[0033] The normalized difference vegetation index (NDVI) is a value between -1 and +1. This value is converted to an integer using the following formula.

[0034]

number

[0035] The integer value of the normalized difference vegetation index (NDVI) ranges from 0 to 200. A higher normalized difference vegetation index (NDVI) value indicates more vegetation.

[0036] Next, the above-mentioned 3D point:pixel correspondence calculation (S26) will be described. In this first embodiment, in order to calculate the Normalized Difference Vegetation Index NDVI, it is necessary to find the pixel on the omnidirectional image that corresponds to the 3D point and extract the red luminance value R from the BGR color information of the pixel. Therefore, in this first embodiment, fisheye cameras (cameras equipped with fisheye lenses) are installed on both the front and rear of the traveling direction as the omnidirectional camera 13 to capture the omnidirectional image that corresponds to the 3D point (fisheye projection model).

[0037] FIG. 6 is a schematic diagram showing the relationship between the world coordinate system and the image coordinate system in the fisheye projection model in the first embodiment. In the fisheye projection model, a three-dimensional point P(xw, yw, zw) in the world coordinate system (Xw, Yw, Zw) is projected onto a hemisphere, and the projected point is further projected onto a point p(xi, yi) on the image plane in the image coordinate system (Xi, Yi). This procedure makes it possible to obtain a pixel on the omnidirectional image corresponding to the three-dimensional point. The fisheye projection model is described in detail in Reference 2 (Yamauchi et al., "Basic Experiments of an Omnidirectional Obstacle Detection System for Mobile Robots Using a Fisheye Camera," Tokyo National College of Technology Research Report No. 45(2), 2014).

[0038] Next, the above-mentioned three-dimensional point coordinate conversion calculation (S24) will be described. The 3D-LiDAR 12 that measures the three-dimensional measurement data and the omnidirectional camera 13 that measures the omnidirectional image data are installed at different positions on the moving dolly 10. Therefore, in the three-dimensional point coordinate conversion calculation, the three-dimensional measurement data measured by the 3D-LiDAR 12 is converted from the coordinate system of the 3D-LiDAR 12 to the coordinate system of the omnidirectional camera 13, and input to the three-dimensional point:pixel correspondence calculation (S26).

[0039] FIG. 7 is a conceptual diagram for explaining the coordinate conversion calculation (S24) of a three-dimensional point according to the first embodiment. The coordinate system of the 3D-LiDAR 12 generally has a Z axis in a vertical upward direction when installed on the mobile robot 1, and an X axis in the moving direction of the mobile robot 1. The coordinate system of the omnidirectional camera 13 generally has a Z axis on the optical axis. For simplicity, FIG. 7 shows only the front side of the fisheye camera constituting the omnidirectional camera 13, but the same idea can be applied to the rear side. For a point P in a three-dimensional space, three-dimensional position data obtained by measurement using the 3D-LiDAR 12 can be converted into three-dimensional position data in the coordinate system (Xc, Yc, Zc) of the omnidirectional camera 13 by a conversion matrix T from the coordinate system (XI, YI, ZI) of the 3D-LiDAR 12 to the coordinate system (Xc, Yc, Zc) of the omnidirectional camera 13. Coordinate transformation using transformation matrix T is performed by using a homogeneous transformation matrix that includes a rotation transformation indicating rotation between coordinate systems and a translation transformation indicating translation. For more information on transformation matrix T, see, for example, Reference 3 ("Homogeneous Transformation Matrix", URL<http: / / www.thothchildren.com / chapter / 5b46226e103f2f316870c401> ) for more details.

[0040] In addition, the values ​​of each component of the homogeneous transformation matrix and the camera parameters required for the coordinate transformation calculation of 3D points and the correspondence calculation of 3D points:pixels in this first embodiment are assumed to be obtained in advance offline by measuring the installation position, calibration, etc.

[0041] Next, the above-mentioned storage of vegetation index data (S32) will be described. Fig. 8 is a conceptual diagram for explaining the storage of vegetation index data according to the first embodiment. In the first embodiment, as shown in Fig. 8, point position information (x, y, z) and reflection intensity are sequentially extracted from the data string of three-dimensional measurement data, and a coordinate conversion calculation of the three-dimensional point and a correspondence calculation of the three-dimensional point:pixel are performed for the point position information (x, y, z) to obtain pixels on the image plane. Furthermore, a red luminance value (R) is extracted from the color information of the pixel, and a normalized difference vegetation index (NDVI) is obtained from the reflection intensity and the red luminance value (R) by a vegetation index value calculation. The obtained normalized difference vegetation index (NDVI) is then stored in the position of the vegetation index data in the same order as the three-dimensional measurement data.

[0042] Next, the above-mentioned three-dimensional point cloud region division process (S12) will be described. FIG. 9 is a flowchart for explaining the operation of the three-dimensional point cloud region division unit 32 according to the first embodiment. In the three-dimensional point cloud region division process, the three-dimensional point cloud region division unit 32 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S40), and inputs vegetation index data from the vegetation index calculation unit 31 (step S42). Next, the three-dimensional point cloud region division unit 32 shapes the three-dimensional measurement data and the vegetation index data into input data for region division (step S44), and performs region division calculation using the formed input data for region division (step S46). After that, the three-dimensional point cloud region division unit 32 creates and outputs point cloud region division data from the formed input data for region division (step S48).

[0043] In the region division calculation (S46) according to the first embodiment, three-dimensional point cloud region division is used. For example, this is a method such as SalsaNext in the above-mentioned non-patent document 2. In the three-dimensional point cloud region division, each point is classified into regions such as automobiles, bicycles, roads, parking lots, buildings, vegetation, trunks, and topography. In this method, it is common to form three-dimensional measurement data into input data for region division, input it to a deep learning network, and output region label data that is the result of region classification. The input data for region division includes "distance" that can be calculated based on three-dimensional measurement data and calculated from position data, "x position", "y position", and "z position" that are the position data themselves, and "reflection intensity" included in the three-dimensional measurement data.

[0044] FIG. 10 is a conceptual diagram for explaining a general region division calculation. FIG. 11 is a conceptual diagram for explaining a region division calculation according to the first embodiment. In a general region division calculation, as shown in FIG. 10, "distance", "x position", "y position", "z position", and "reflection intensity" are input to a three-dimensional point cloud region division (deep learning network) 41 as region division input data 40, and region label data 42 is obtained. In contrast, in a region division calculation according to the first embodiment, as shown in FIG. 11, in addition to the region division input data "distance", "x position", "y position", "z position", and "reflection intensity" of the general region division calculation, normalized vegetation index (NDVI) formed from vegetation index data is input to a three-dimensional point cloud region division (deep learning network) 51 as region division input data 50, and region label data 52 is obtained.

[0045] The deep learning network 51 that performs the 3D point cloud region segmentation is assumed to have been trained offline using a training data set. The training method will be described in detail in the fourth and fifth embodiments described later.

[0046] Next, the above-mentioned point cloud region division data creation (S48) will be described. Fig. 12 is a conceptual diagram for explaining the creation of point cloud area division data according to the first embodiment. In the creation of point cloud area division data according to the first embodiment, as shown in Fig. 12, the area label data calculated by the area division calculation is arranged and stored in an order corresponding to the point position information (x, y, z) stored in the three-dimensional measurement data to create point cloud area division data. By this process, it is possible to obtain point cloud area division data corresponding to the order of points stored in the three-dimensional point cloud data.

[0047] Next, a cost map calculation on the control processing unit 33 side using the point cloud region division data output by the surrounding environment recognition unit 30 according to the first embodiment will be described. A cost map using three-dimensional measurement data is created from the position information of the three-dimensional point cloud in the same manner as in Non-Patent Document 1. In the first embodiment, this is called a "position information cost map." Also, in the first embodiment, a cost map is also created from the point cloud region division data. This is called an "region division cost map."

[0048] Point cloud region segmentation data has region labels for each point, such as cars, bicycles, roads, parking lots, buildings, vegetation, trunks, and terrain. For example, the region labels in which a mobile robot may travel may be "roads, parking lots, and sidewalks." In this case, the cost value for region labels of "roads, parking lots, and sidewalks" is set to 0, and the cost value for other regions is set to c. c is set to a value according to the settings. The region segmentation cost map is calculated by projecting the cost values ​​assigned to the region labels onto a horizontal plane and accumulating them.

[0049] FIG. 13 is a conceptual diagram for explaining an example of a cost map used in the first embodiment. For example, as shown in FIG. 13, consider a situation in which an automobile 62 is parked beside a road 61 that passes through a lawn 60. In the location information cost map 63, a cost map with a high cost value is generated around the automobile 62, which is high from the ground. In the figure, the darker the shade, the higher the cost value. In the area division cost map 64, a cost value is assigned to an area other than "road, parking lot, and sidewalk," and the cost value is high near the automobile 62 where the point clouds overlap in the height direction. A cost value is also assigned to the grass 60. Finally, the location information cost map 63 and the area division cost map 64 are combined to create a cost map 65. A method of adding cost maps is used for the combination.

[0050] According to the first embodiment described above, since the region division process is assisted by using vegetation indices, it is possible to stably distinguish between artificial objects such as "roads, parking lots, and sidewalks" and vegetation such as grass. Therefore, for example, when a mobile robot travels through a park, it will not enter the grass, which is prohibited. Furthermore, even if there is some kind of plant on the "roads, parking lots, and sidewalks" in the region division result, it will not be mistakenly detected as an obstacle, so the robot can travel smoothly without taking abnormal avoidance actions or becoming unable to travel because it cannot find an avoidance route.

[0051] B. Second embodiment Next, a second embodiment of the present invention will be described. When training a deep learning network that performs 3D point cloud region segmentation, it is preferable to obtain omnidirectional image data that covers the entire range of omnidirectional 3D measurement data around the mobile robot 1 measured by the 3D-LiDAR 12. In this case, both the 3D-LiDAR 12 and the omnidirectional camera 13 need to be installed at locations that overlook the surroundings of the mobile robot 1.

[0052] On the other hand, when the mobile robot 1 is actually made to travel automatically, it is sufficient if the area ahead of the mobile robot 1 can be correctly divided in the direction of travel. Therefore, when the mobile robot 1 travels automatically, it is possible to use a color camera (imaging unit) 70 that captures an image ahead of the mobile robot 1 in the direction of travel instead of the omnidirectional camera 13. This increases the degree of freedom in the location where the color camera 70 is installed. For example, the color camera 70 may be installed in front of the mobile robot 1.

[0053] Fig. 14 is a schematic diagram showing one configuration example of the mobile robot 1 according to the second embodiment. Parts corresponding to those in Fig. 1 are given the same reference numerals and their explanations are omitted. In Fig. 14, a color camera 70 is mounted on a vertical section 16 extending vertically from the cart section 15, instead of the omnidirectional camera 13. The color camera 70 captures an image of the area ahead in the traveling direction of the mobile robot 1, and supplies color image data to the controller 11.

[0054] Fig. 15 is a block diagram showing an example of the configuration of the controller 11 according to the second embodiment. The parts corresponding to those in Fig. 3 are given the same reference numerals and the description thereof will be omitted. The surrounding environment recognition unit 30 is configured to input color image data from a color camera 70 to a vegetation index calculation unit 34 instead of omnidirectional image data. The vegetation index calculation unit 34 calculates a vegetation index based on the color image data from the color camera 70 and the three-dimensional measurement data from the 3D-LiDAR 12, and supplies the vegetation index data to the three-dimensional point cloud area division unit 32.

[0055] Next, the operation of the surrounding environment recognition unit 30 according to the second embodiment will be described. The overall operation (surrounding environment recognition processing) of the surrounding environment recognition unit 30 in the second embodiment is basically the same as the flowchart shown in FIG. 4 described in the first embodiment above. However, the content of the vegetation index calculation processing in step S10 is different because a color camera 70 is installed instead of the omnidirectional camera 13.

[0056] Next, the vegetation index calculation process according to the second embodiment will be described. FIG. 16 is a flowchart for explaining the operation of the vegetation index calculation unit 34 according to the second embodiment. The vegetation index calculation unit 34 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S60), and inputs color image data from the color camera 70 (step S62). Next, the vegetation index calculation unit 34 performs coordinate conversion calculation of three-dimensional points for all three-dimensional points obtained from the input three-dimensional measurement data (step S64). Next, the vegetation index calculation unit 34 performs a correspondence calculation of three-dimensional points: pixels based on the color image data (step S66), and extracts a red luminance value (step S68). Next, the vegetation index calculation unit 34 calculates a vegetation index value based on the red luminance value (step S70), and saves the vegetation index data (step S72). In this way, the vegetation index calculation unit 34 creates vegetation index data by sequentially performing a series of processes. Thereafter, it is determined whether or not all three-dimensional points have been processed (step S74), and if there are unprocessed three-dimensional points (NO in step S74), the process returns to step S64 and repeats the above-mentioned process. On the other hand, if all three-dimensional points have been processed (YES in step S74), the process ends.

[0057] In the second embodiment, the 3D point:pixel correspondence calculation (S66) is replaced with a calculation method compatible with the color camera 70. A perspective projection camera model is used as a method for determining pixels on the color camera 70 that correspond to 3D points. For more information on the perspective projection camera model, see Reference 4 ("Camera Model and Perspective Projection"), URL:<https: / / cvml-expertguide.net / terms / cv / camera-geometry / camera-model / > ) as explained in

[0058] 17 is a conceptual diagram showing the relationship between the world coordinate system and the image coordinate system in the perspective projection camera model in the second embodiment. In the perspective projection camera model, a three-dimensional point P(xw,yw,zw) in the world coordinate system (Xw,Yw,Zw) is the camera origin, and is projected to a point p(xi,yi) on the image plane perspectively projected by a straight line toward the world coordinate origin Ow. This method makes it possible to obtain the pixel on the image corresponding to the three-dimensional point.

[0059] After the vegetation index calculation process, a three-dimensional point cloud region division process is performed. However, the three-dimensional point cloud region division process is similar to that in the first embodiment described above, and therefore a description thereof will be omitted.

[0060] According to the second embodiment described above, by mounting a color camera that captures only what is ahead in the direction of travel when the running robot is actually running, it is possible to increase the degree of freedom in the location where the camera is installed.

[0061] C. Third embodiment Next, a third embodiment of the present invention will be described. The above-mentioned vegetation index can be calculated if there is brightness information in the red region of the visible light region. Therefore, in the third embodiment, a monochrome camera with a red transmission filter installed in front of the lens is used as the camera that captures the forward direction. In this way, the color information of the image becomes monochromatic compared to the BGR color image, and the memory amount can be reduced to one-third, and the effort of extracting the R value from the color information BGR can be eliminated.

[0062] FIG. 18 is a schematic diagram showing an example of the configuration of a mobile robot 1 according to the third embodiment. The parts corresponding to those in FIG. 1 or FIG. 14 are given the same reference numerals and their explanations are omitted. In FIG. 18, a monochrome camera (imaging unit) 71 is mounted on a vertical unit 16 extending vertically from a cart unit 15, instead of the omnidirectional camera 13 or a color camera 70. The monochrome camera 71 is provided with a red transmission filter 72 on the front surface of the lens. The monochrome camera 71 captures an image of the area ahead in the traveling direction of the mobile robot 1, and supplies gray image data corresponding to red to the controller 11.

[0063] Fig. 19 is a block diagram showing an example of the configuration of the controller 11 according to the third embodiment. The same reference numerals are used for the parts corresponding to Fig. 3 or Fig. 15, and the description thereof will be omitted. The surrounding environment recognition unit 30 is configured to input gray image data from the monochrome camera 71 to the vegetation index calculation unit 35, instead of the omnidirectional image data or color image data. The vegetation index calculation unit 35 calculates vegetation index data based on the gray image data from the monochrome camera 71 and the three-dimensional measurement data from the 3D-LiDAR 12, and supplies the vegetation index data to the three-dimensional point cloud area division unit 32.

[0064] Next, the operation of the surrounding environment recognition unit 30 according to the third embodiment will be described. The overall operation (surrounding environment recognition processing) of the surrounding environment recognition unit 30 according to the third embodiment is basically the same as the flowchart shown in FIG. 4 described in the first embodiment above. However, the incorporation of the monochrome camera 71 results in a difference in the content of the vegetation index calculation processing in step S10.

[0065] Next, the vegetation index calculation process according to the third embodiment will be described. FIG. 20 is a flowchart for explaining the operation of the vegetation index calculation unit 35 according to the third embodiment. The vegetation index calculation unit 35 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S80), and inputs gray image data from the monochrome camera 71 (step S82). Next, the vegetation index calculation unit 35 performs three-dimensional point coordinate conversion calculation for all three-dimensional points obtained from the input three-dimensional measurement data (step S84). Next, the vegetation index calculation unit 35 performs three-dimensional point:pixel correspondence calculation based on the gray image data (step S86), and extracts a red luminance value (step S88). Note that, in the red luminance value extraction, it is sufficient to take out the luminance value of the gray image data at the corresponding pixel on the image plane.

[0066] Next, the vegetation index calculation unit 35 calculates the vegetation index value based on the red luminance value (step S90) and stores the vegetation index data (step S92). In this way, the vegetation index calculation unit 35 creates vegetation index data by sequentially performing a series of processes. Thereafter, it is determined whether or not the process has been performed for all three-dimensional points (step S94), and if there are unprocessed three-dimensional points (NO in step S94), the process returns to step S84 and repeats the above-mentioned process. On the other hand, if the process has been performed for all three-dimensional points (YES in step S94), the process ends. Note that the three-dimensional point:pixel correspondence calculation (S86) in this third embodiment uses a perspective projection camera model similar to that in the second embodiment described above.

[0067] Next, the storage of vegetation index data (S92) in the third embodiment will be described. Fig. 21 is a conceptual diagram showing an overview of vegetation index data storage in the third embodiment. As shown in Fig. 21, point position information (x, y, z) and reflection intensity are sequentially extracted from the data string of three-dimensional measurement data, and a coordinate conversion calculation of the three-dimensional point and a correspondence calculation of the three-dimensional point:pixel are performed for the point position information (x, y, z) to obtain pixels on the image plane. Furthermore, the luminance value of the pixel is extracted (this becomes the red luminance value (R)), and a normalized difference vegetation index (NDVI) is obtained from the reflection intensity and the red luminance value (R) by a vegetation index value calculation. The obtained normalized difference vegetation index (NDVI) is then stored in the position of the vegetation index data in the same order as the three-dimensional measurement data.

[0068] After the vegetation index calculation process, a three-dimensional point cloud region division process is performed. However, the three-dimensional point cloud region division process is similar to that of the first or second embodiment described above, so a description thereof will be omitted.

[0069] According to the third embodiment described above, when the running robot actually runs, a monochrome camera is installed that captures only the image ahead in the direction of travel, and a red transparent filter is installed in front of the lens of the monochrome camera. This makes the color information of the image monochromatic compared to the BGR color image, reducing the memory size by one-third and eliminating the need to extract the R value from the BGR color information.

[0070] D. Fourth embodiment Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, a deep learning network (deep learning device) that performs three-dimensional point cloud region segmentation will be described. The deep learning network is assumed to be trained offline in advance using a training data set. In three-dimensional point cloud region segmentation, a method of creating region labels for a three-dimensional point cloud measured by a 3D-LiDAR is generally used. The creation of point cloud region labels for a three-dimensional point cloud is described in, for example, Reference 5 ("Point Cloud Annotation", URL<https: / / docs.fastlabel.ai / docs / 3d-point-cloud-annotation> ) to display the 3D point cloud and manually surround areas, or to specify each point individually, to specify region labels for the points and create point cloud region label data.

[0071] In the fourth embodiment, a set of three-dimensional measurement data measured by the 3D-LiDAR 12, omnidirectional image data captured by the omnidirectional camera 13, and point cloud region label data created based on the three-dimensional measurement data are used as a training data set. A deep learning network that performs three-dimensional point cloud region division is trained offline using this training data set. The results of the training are saved as training result data. The training result data is what is called a trained model of the deep learning network.

[0072] The measurement of the three-dimensional measurement data and the capture of the omnidirectional image data required to create a learning data set are performed using the mobile robot 1 of the first embodiment shown in Fig. 1. The three-dimensional measurement data and the omnidirectional image data are temporarily stored in the controller 11 of the mobile robot 1 and are retrieved offline via a network or via storage.

[0073] The creation of point cloud region label data for 3D point clouds is performed offline using a work terminal equipped with an annotation tool such as that described in Reference 5. The point cloud region label data is temporarily stored in the work terminal and then retrieved offline via a network or via storage.

[0074] FIG. 22 is a block diagram showing the configuration of a three-dimensional point cloud region segmentation learning device (deep learning device) 80 according to the fourth embodiment. The three-dimensional point cloud region segmentation learning device 80 includes a storage unit 81, a vegetation index calculation unit 82, and a three-dimensional point cloud region segmentation learning unit 83. The three-dimensional point cloud region segmentation learning device 80 inputs three-dimensional measurement data, omnidirectional image data, and point cloud region label data, and temporarily stores them in the storage unit 81. The vegetation index calculation unit 82 reads out the three-dimensional measurement data and omnidirectional image data from the storage unit 81, creates vegetation index data, and stores it in the storage unit 81. The three-dimensional point cloud region segmentation learning unit 83 reads out the point cloud region label data, the three-dimensional measurement data, and the vegetation index data from the storage unit 81, learns them, and then stores the learning result data in the storage unit 81. The learning result data is taken out offline via a network or via a storage device.

[0075] Next, the operation of the three-dimensional point cloud region division learning process according to the fourth embodiment will be described. FIG. 23 is a flowchart for explaining the overall operation of the three-dimensional point cloud region division learning device 80 according to the fourth embodiment. In the three-dimensional point cloud region division learning device 80, the vegetation index calculation unit 82 executes a vegetation index calculation process for calculating vegetation index data based on the omnidirectional image data from the omnidirectional camera 13 and the three-dimensional measurement data from the 3D-LiDAR 12 (step S100). The vegetation index calculation process uses a method similar to the basic method (FIG. 5) described in the first embodiment above. Next, the three-dimensional point cloud region division learning unit 83 executes a deep learning network learning process for learning based on the point cloud region label data, the three-dimensional measurement data, and the vegetation index data to obtain learning result data (step S102).

[0076] Next, the deep learning network learning process (S102) according to the fourth embodiment will be described. FIG. 24 is a flowchart for explaining the operation of the three-dimensional point cloud region division learning unit 83 according to the fourth embodiment. The three-dimensional point cloud region division learning unit 83 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S110), inputs vegetation index data calculated by the vegetation index calculation unit 82 (step S112), and inputs point cloud region label data (step S114). Next, the three-dimensional point cloud region division learning unit 83 forms the three-dimensional measurement data and the vegetation index data into region division input data (step S116), and forms the point cloud region label data into region label data (step S118). Next, the three-dimensional point cloud region division learning unit 83 performs learning by a deep learning network based on the region division input data and the region label data (step S120), creates learning result data, and outputs the learning result data (step S122).

[0077] The learning in the fourth embodiment is performed by a method similar to the three-dimensional point cloud region segmentation method used in the region segmentation calculation (for example, a method such as SalsaNext in Non-Patent Document 2). FIG. 25 is a conceptual diagram for explaining learning by a deep learning network according to the fourth embodiment. As input data 90 for area division, in addition to "distance" that can be calculated from position data of three-dimensional measurement data, "x position", "y position", and "z position" which are the position data themselves, and "reflection intensity" included in the three-dimensional measurement data, "normalized difference vegetation index (NDVI)" formed from vegetation index data is used. In addition, area label data 91 formed from point cloud area label data is input. A deep learning network 92 learns by repeated calculations using pairs of the formed input data for area division and area label data, and finally outputs learning result data 93.

[0078] According to the above-mentioned fourth embodiment, the three-dimensional measurement data measured by the 3D-LiDAR 12, the omnidirectional image data captured by the omnidirectional camera 13, and a set of point cloud area label data created based on the three-dimensional measurement data are used as a learning dataset for the deep learning network, so that it is possible to stably distinguish between man-made objects such as "roads, parking lots, and sidewalks" and vegetation such as grass.

[0079] E. Fifth embodiment Next, a fifth embodiment of the present invention will be described. In the above-mentioned fourth embodiment, a point cloud region label for a three-dimensional point cloud is created by displaying a three-dimensional point cloud and manually surrounding the region, or by individually specifying each point, using an annotation tool. However, it is difficult to actually carry out this. A car or a person that exists alone on a road can be distinguished to some extent by displaying three-dimensional points. However, it is often difficult to easily distinguish between grass and roads, roads and sidewalks, buildings and fences, etc., just by displaying three-dimensional points. Therefore, in the present fifth embodiment, omnidirectional image data captured by the omnidirectional camera 13 paired with the three-dimensional measurement data measured by the 3D-LiDAR 12 is used to create a point cloud region label. It is easier for an operator to distinguish between regions when displaying an omnidirectional image than when displaying three-dimensional points.

[0080] In the fifth embodiment, region labels are generated for omnidirectional image data.<https: / / www.gigafree.net / tool / paint / gimp.html> ), a paint tool such as the one shown in FIG. 1 is used to display the omnidirectional image and manually color the regions to create image region labels for the omnidirectional image data. In the fifth embodiment, a set of three-dimensional measurement data measured by the 3D-LiDAR 12, omnidirectional image data captured by the omnidirectional camera 13, and image region label data created based on the omnidirectional image data are used as a learning dataset.

[0081] The measurement of the three-dimensional measurement data and the capture of the omnidirectional image data required to create a learning data set are performed using the mobile robot 1 having the configuration shown in Fig. 1 of the first embodiment. The three-dimensional measurement data from the 3D-LiDAR 12 and the omnidirectional image data from the omnidirectional camera 13 are temporarily stored in the controller 11 of the mobile robot 1 and are then retrieved offline via a network or via storage.

[0082] To create image region labels for 3D point clouds, a work terminal equipped with a paint tool such as that described in Reference 6 is prepared, and image region label data is created offline. The image region label data is temporarily stored in the work terminal, and then retrieved offline via a network or via storage.

[0083] FIG. 26 is a block diagram showing the configuration of a three-dimensional point cloud region segmentation learning device (deep learning device) 100 according to the fifth embodiment. Note that the same reference numerals are used for the parts corresponding to FIG. 22, and the description thereof will be omitted. The three-dimensional point cloud region segmentation learning device 100 according to the fifth embodiment inputs image region label data instead of the point cloud region label data in the three-dimensional point cloud region segmentation learning device 80 according to the fourth embodiment described above, and temporarily stores the image region label data in the storage unit 101. The three-dimensional point cloud region segmentation learning device 100 according to the fifth embodiment further includes a point cloud region label creation unit 103 in addition to the configuration of the three-dimensional point cloud region segmentation learning device 80 according to the fourth embodiment described above. The point cloud region label creation unit 103 reads out three-dimensional measurement data and image region label data from the storage unit 101, creates point cloud region label data, and stores it in the storage unit 101. The learning result data is taken out offline via a network or via storage.

[0084] Next, the operation of the three-dimensional point cloud region division learning process according to the fifth embodiment will be described. FIG. 27 is a flowchart for explaining the overall operation of the three-dimensional point cloud region division learning device 100 according to the fifth embodiment. In the three-dimensional point cloud region division learning device 100, the vegetation index calculation unit 102 executes a vegetation index calculation process for calculating a vegetation index based on the omnidirectional image data from the omnidirectional camera 13 and the three-dimensional measurement data from the 3D-LiDAR 12 (step S130). The vegetation index calculation process uses a method similar to the basic method (FIG. 5) described in the first embodiment above. Next, the point cloud region label creation unit 103 executes a point cloud region label creation process for creating point cloud region label data based on the three-dimensional measurement data and the image region label data (step S132). Next, the three-dimensional point cloud region division learning unit 104 executes a deep learning network learning process for learning based on the point cloud region label data, the three-dimensional measurement data, and the vegetation index data to obtain learning result data (step S134).

[0085] Next, the point cloud region label creation process (S132) according to the fifth embodiment will be described. FIG. 28 is a flowchart for explaining the operation of the point cloud region label creation unit 103 according to the fifth embodiment. The point cloud region label creation unit 103 first inputs three-dimensional measurement data from the 3D-LiDAR 12 (step S140), and inputs image region label data (step S142). Next, the point cloud region label creation unit 103 performs three-dimensional point coordinate conversion calculation for all three-dimensional points obtained from the input three-dimensional measurement data (step S144), and performs three-dimensional point: pixel correspondence calculation (step S146). Note that the three-dimensional point coordinate conversion calculation and three-dimensional point: pixel correspondence calculation in the fifth embodiment use a method similar to the basic method (steps S24 and S26 in FIG. 5) described in the first embodiment above.

[0086] Next, the point cloud region label creation unit 103 extracts the region label (step S148), creates and saves the point cloud region label data (step S150). After that, it is determined whether or not the process has been performed for all the 3D points (step S152). If there are unprocessed 3D points (NO in step S152), the process returns to step S144 and repeats the above-mentioned process. On the other hand, if the process has been performed for all the 3D points (YES in step S152), the process ends.

[0087] Next, the storage of point cloud area label data (S150) in the fifth embodiment will be described. FIG. 29 is a conceptual diagram showing an overview of the storage of point cloud area label data in the fifth embodiment. Image area label data is a data structure in which an area label is arranged in each pixel of omnidirectional image data. Therefore, it is sufficient to extract the area label in the corresponding pixel in the image plane. As shown in FIG. 29, point position information (x, y, z) and reflection intensity are sequentially extracted from the data string of the three-dimensional measurement data, and a three-dimensional point coordinate conversion calculation and a three-dimensional point:pixel correspondence calculation are performed on the point position information (x, y, z) to obtain the pixel on the image plane. Furthermore, the area label stored in the pixel position is extracted and stored in the position of the point cloud area label data in the same order as the three-dimensional measurement data.

[0088] The learning process of the deep learning network according to the fifth embodiment is similar to that of the fourth embodiment described above, so a description thereof will be omitted.

[0089] According to the above-mentioned fifth embodiment, the three-dimensional measurement data measured by the 3D-LiDAR 12, the omnidirectional image data captured by the omnidirectional camera 13, and a set of image region label data created based on the omnidirectional image data are used as a learning dataset for the deep learning network, so that man-made objects such as "roads, parking lots, and sidewalks" can be stably distinguished from vegetation such as grass.

[0090] In addition, it is possible to select and discard the configurations given in the above embodiment, or to change them to other configurations as appropriate, without departing from the spirit of the present invention. [Explanation of symbols]

[0091] 1 Mobile robot (running body) 10 Mobile stand 11 Controller 12 3D LiDAR 13 Omnidirectional camera 14 Means of transportation 16 Vertical section 30 Surrounding Environment Recognition Unit 31, 34, 35 Vegetation index calculation section 32 3D point cloud region segmentation part 33 Control processing section 50, 90 Input data for region division 51,92 3D point cloud segmentation (deep learning network) 52, 91 Region label data 70 Color Camera 71 Monochrome Camera 72 Red transmission filter 80, 100 3D point cloud segmentation learning device (deep learning device) 81, 101 Storage section 82, 102 Vegetation index calculation section 83, 104 3D point cloud segmentation learning part 103 Point cloud area label creation unit

Claims

1. a storage unit that stores three-dimensional measurement data including three-dimensional information on a surrounding environment of a moving object, omnidirectional image data including an image of the surrounding environment in all directions, and point cloud area label data created based on the three-dimensional measurement data; a vegetation index calculation unit that calculates vegetation index data indicating a state of vegetation in the surrounding environment based on the three-dimensional measurement data and the omnidirectional image data; a three-dimensional point cloud region segmentation learning unit that performs deep learning using the three-dimensional measurement data, the vegetation index data, and the point cloud region label data as a learning data set, and creates learning result data; A deep learning device comprising:

2. the storage unit stores the three-dimensional measurement data, the omnidirectional image data, and image region label data created based on the omnidirectional image data in place of the point cloud region label data; The deep learning device according to claim 1 , further comprising a point cloud region label creation unit that creates the point cloud region label data based on the three-dimensional measurement data and the image region label data.

3. A traveling object that recognizes the surrounding environment and travels by a traveling means, a three-dimensional measurement unit for acquiring three-dimensional measurement data including three-dimensional information of the surrounding environment; an imaging unit for acquiring image data of the surrounding environment; a vegetation index calculation unit that calculates vegetation index data indicating a state of vegetation in the surrounding environment based on the three-dimensional measurement data and the image data; a three-dimensional point cloud region dividing unit that generates point cloud region division data based on the three-dimensional measurement data and the vegetation index data; a control processing unit that calculates a cost map of the surrounding environment based on the three-dimensional measurement data and the point cloud area division data, and controls traveling of the traveling means based on the cost map; A traveling body comprising:

4. the imaging unit is an omnidirectional camera that captures images of the surrounding environment in all directions, The image data is omnidirectional image data captured by the omnidirectional camera.

4. The vehicle according to claim 3.

5. The omnidirectional camera is a fisheye camera installed on both the front and rear of the traveling direction of the traveling body.

5. The vehicle according to claim 4.

6. The imaging unit is a color camera that captures an image of a forward direction of the traveling object, The image data is color image data captured by the color camera.

4. The vehicle according to claim 3.

7. the imaging unit is a monochrome camera having a red transmission filter on a front surface of a lens and capturing an image of a forward direction in a traveling direction of the traveling object, The image data is gray image data captured by the monochrome camera.

4. The vehicle according to claim 3.

Images