Mobile body control system, method for controlling same, program, and storage medium

Abstract

This mobile body control system comprises: point group generation means for generating information on a point group indicating an object surface in a three-dimensional space around a mobile body on the basis of a captured image obtained by imaging the periphery of the mobile body; recess determination means for determining the presence of a recess on a travel path on the basis of the height difference and the height of the point group in a divided region for individual divided regions in the periphery of the mobile body; and map generation means for generating an occupancy map indicating whether an obstacle is occupying each divided region on the basis of the determination result by the recess determination means.

Description

Mobile object control system, control method thereof, program, and storage medium

[0001] The present invention relates to a mobile object control system, a control method therefor, a program, and a storage medium.

[0002] In recent years, autonomously moving bodies, such as vehicles, have become known that use captured images of their surroundings to recognize the conditions of the surrounding road and generate a route for themselves to travel. The travel of such bodies can be restricted by objects that protrude convexly from the plane on which they travel, but can also be restricted by areas where the plane on which they travel is recessed.

[0003] To detect concave areas, Patent Document 1 discloses a technology in which three-dimensional coordinates corresponding to each pixel are calculated using images from a stereo camera, and areas in which pixel groups whose three-dimensional coordinates are lower than the ground are clustered are detected as road shoulders (road structures lower than the ground).

[0004] JP 2011-138244 A

[0005] However, when estimating three-dimensional coordinates using images, there are cases where the estimation accuracy is reduced due to noise caused by blind spots (occlusions) and textures. That is, in the technology proposed in Patent Document 1, there are cases where the accuracy of detecting a recessed area is reduced when noise is present in the height information of the three-dimensional coordinates.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a technique that can accurately detect a recessed area around a moving object.

[0007] According to the present invention, there is provided a mobile body control system comprising: a point cloud generation means for generating point cloud information indicating the surface of an object in three-dimensional space around a mobile body based on an image of the area around the mobile body; a depression determination means for determining, for each divided area around the mobile body, whether or not a depression exists on the road based on the elevation difference and height of the point cloud within the divided area; and a map generation means for generating an occupancy map indicating the occupation of an obstacle for each divided area based on the determination result by the depression determination means.

[0008] According to the present invention, it is possible to accurately detect a recessed area around a moving object.

[0009] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0010] The accompanying drawings are included in and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description are used to explain the principles of the present invention. Block diagram showing an example of the hardware configuration of a mobile body according to the present embodiment (1) Block diagram showing an example of the hardware configuration of a mobile body according to the present embodiment (2) Block diagram showing the control configuration of a mobile body according to the present embodiment Block diagram showing the functional configuration of a control unit according to the present embodiment Diagram showing an occupancy grid map according to the present embodiment Diagram showing a method for generating an occupancy grid map according to the present embodiment Diagram explaining depression evaluation according to the present embodiment Diagram showing an example of depression evaluation according to the present embodiment Diagram explaining filtering of point clouds according to the present embodiment (1) Diagram explaining filtering of point clouds according to the present embodiment (2) Diagram showing an example of reflecting a depressed area in an occupancy grid map according to the present embodiment Diagram showing global routes and local routes according to the present embodiment Flowchart showing a processing procedure for controlling the travel of a mobile body according to the present embodiment Flowchart showing a processing procedure for generating an occupancy grid map according to the present embodiment

[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0012] <Configuration of Mobile Body> The configuration of a mobile body 100 according to this embodiment will be described with reference to Figures 1A and 1B. Figure 1A shows a side view of the mobile body 100 according to this embodiment, and Figure 1B shows the internal configuration of the mobile body 100. In the figures, arrow X indicates the front-to-rear direction of the mobile body 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the mobile body 100.

[0013] The mobile body 100 is equipped with a battery 113 and is, for example, an ultra-compact mobility vehicle that moves primarily by motor power. An ultra-compact mobility vehicle is more compact than a typical automobile and has a passenger capacity of approximately one or two people. In this embodiment, a three-wheeled ultra-compact mobility vehicle is described as an example of the mobile body 100. However, this is not intended to limit the present invention, and the mobile body 100 may also be, for example, a four-wheeled vehicle or a saddle-type vehicle. Furthermore, the mobile body of the present invention is not limited to a vehicle, but may also be a vehicle that carries luggage and runs alongside a person walking, or a vehicle that leads a person. Furthermore, the mobile body control system of this embodiment may be a mobile body, a control device such as an ECU included in the mobile body, or a cloud-based information processing server for controlling the mobile body. In other words, some or all of the processing described below according to this embodiment may be executed in the mobile body or in a cloud-based information processing server. Furthermore, the present invention is not limited to four-wheeled or two-wheeled vehicles, but may also be applied to autonomously moving robots and the like.

[0014] The mobile body 100 is an electric autonomous vehicle equipped with a propulsion unit 112 and using a battery 113 as its main power source. The battery 113 is, for example, a secondary battery such as a lithium-ion battery, and the mobile body 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 is equipped with, for example, a pair of left and right front wheels 120 and a tail wheel (driven wheel) 121. The propulsion unit 112 may be in another form, such as a four-wheeled vehicle. The mobile body 100 is equipped with a seat 111 for one or two people.

[0015] The traveling unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that uses motors 122a, 122b as drive sources to change the steering angle of the pair of front wheels 120. Changing the steering angle of the pair of front wheels 120 can change the traveling direction of the mobile body 100. The tail wheel 121 does not have its own drive source, but is a driven wheel that operates in response to the drive of the pair of front wheels 120. The tail wheel 121 is also connected to the body of the mobile body 100 via a swivel. The swivel rotates so that the direction of the tail wheel 121 changes separately from the rotation of the tail wheel 121. In this way, the mobile body 100 according to this embodiment employs a differential two-wheel mobility with a tail wheel, but is not limited to this.

[0016] The moving body 100 includes a detection unit 114 that recognizes a plane in front of the moving body 100. The detection unit 114 is an external sensor that monitors the area in front of the moving body 100. In this embodiment, the detection unit 114 is an imaging device that captures an image in front of the moving body 100. In this embodiment, the detection unit 114 includes, for example, a stereo camera having optical systems such as two lenses and respective image sensors, and a monocular camera. However, it is also possible to employ radar or lidar (light detection and ranging) instead of or in addition to the imaging device. Furthermore, in this embodiment, an example in which the detection unit 114 is provided only in front of the moving body 100 is described, but this is not intended to limit the present invention, and the detection unit 114 may be provided behind, on the left or right sides of the moving body 100. Furthermore, instead of using a monocular camera, an image captured by one of the stereo cameras may be used.

[0017] The mobile body 100 according to this embodiment captures an image of the area ahead of the mobile body 100 using the detection unit 114 and detects obstacles from the captured image. Furthermore, the mobile body 100 divides the area around the mobile body 100 into a grid and controls its travel while generating an occupancy grid map in which obstacle information is accumulated for each divided area (hereinafter also referred to as a grid). The occupancy grid map will be described in detail later.

[0018] <Control Configuration of the Mobile Body> FIG. 2 is a block diagram of a control system of the mobile body 100 according to this embodiment. Here, the configuration necessary for implementing the present invention will be mainly described. Therefore, other configurations may be included in addition to the configurations described below. Furthermore, in this embodiment, the mobile body 100 is described as including each of the components described below, but this is not intended to limit the present invention. A mobile body control system including multiple devices may be realized. For example, some functions of the control unit 130 may be implemented by an information processing server connected to the mobile body 100 in a communicable manner, or the detection unit 114 and the GNSS sensor 134 may be provided as external devices. The mobile body 100 includes a control unit (ECU) 130. The control unit 130 includes a processor, such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used by the processor for processing, and the like. Multiple sets of processors, storage devices, and interfaces may be provided for different functions of the mobile body 100 and configured to communicate with each other.

[0019] The control unit 130 acquires the detection results of the detection unit 114, input information from the operation panel 131, audio information input from the audio input device 133, position information from the GNSS sensor 134, and information received via the communication unit 136, and executes corresponding processing. The control unit 130 controls the motors 122a and 122b (travel control of the traveling unit 112), controls the display on the operation panel 131, and notifies and outputs information to the occupants of the mobile object 100 by audio from the speaker 132.

[0020] The voice input device 133 collects the voices of the occupants of the mobile body 100. The control unit 130 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals to detect the current position of the mobile body 100. The storage device 135 is a storage device that stores images captured by the detection unit 114, obstacle information, previously generated routes, and occupancy grid maps. The storage device 135 may also store programs executed by the processor, data used for processing by the processor, and the like. The storage device 135 may also store various parameters (e.g., learned parameters and hyperparameters of a deep neural network) of machine learning models for voice recognition and image recognition executed by the control unit 130.

[0021] The communication unit 136 communicates with a communication device 140, which is an external device, via wireless communication such as Wi-Fi or fifth-generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, or the like. The communication device 140 connects to a network via wireless communication such as Wi-Fi or fifth-generation mobile communication.

[0022] A user who owns the communication device 140 can issue instructions to the mobile body 100 via the communication device 140. The instructions include, for example, an instruction to call the mobile body 100 to a location desired by the user and join the mobile body 100. Upon receiving the instruction, the mobile body 100 sets a target position based on the position information included in the instruction. In addition to such instructions, the mobile body 100 can also set a target position from an image captured by the detection unit 114 or based on an instruction from a user riding in the mobile body 100 via the operation panel 131. When setting a target position from a captured image, for example, a person raising their hand toward the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target position.

[0023] <Functional Configuration of Mobile Body> Next, the functional configuration of the mobile body 100 according to this embodiment will be described with reference to Fig. 3. The functional configuration described here is realized by a processor, such as a CPU, in the control unit 130 reading a program stored in a memory, such as a ROM, into a RAM and executing it. Note that the functional configuration described below describes only the functions necessary for explaining the present invention, and does not describe all of the functional configuration actually included in the mobile body 100. In other words, the functional configuration of the mobile body 100 according to the present invention is not limited to the functional configuration described below.

[0024] The user instruction acquisition unit 301 has a function of accepting instructions from the user, and can accept user instructions via the operation panel 131, user instructions from an external device such as the communication device 140 via the communication unit 136, and user spoken instructions via the voice input device 133. As described above, user instructions include an instruction to set a target position (also referred to as a destination) of the mobile body 100 and instructions related to driving control of the mobile body 100.

[0025] The image information processing unit 302 processes the captured images acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image from the stereo images acquired by the detection unit 114 and converts it into a three-dimensional point cloud. The three-dimensional point cloud image data (also referred to as depth information) is used to detect obstacles that impede the travel of the mobile object 100. In this way, the image information processing unit 302 includes a point cloud generation means that generates point cloud information indicating the surface of an object in three-dimensional space based on the captured image information.

[0026] The image information processing unit 302 may also include a machine learning model that processes image information and may execute a learning stage process or an inference stage process of the machine learning model. For example, the image information processing unit 302 may identify obstacles included in images captured by a monocular camera. For example, the image information processing unit 302 may use the machine learning model to identify obstacle types, and may distinguish between predetermined dynamic obstacles and static obstacles for each type of obstacle. Dynamic obstacles are obstacles that can move autonomously, and include, for example, vehicles or other traffic participants such as pedestrians and bicycles. Static obstacles are obstacles that do not move autonomously, and include, for example, objects such as signs and guardrails.

[0027] The grid map generator 303 creates a grid map of a predetermined size (e.g., each cell is 10 cm x 10 cm in a 20 m x 20 m area) based on the 3D point cloud data. This is done to reduce the size of the grid, since the amount of 3D point cloud data is large and real-time processing is difficult. The grid map includes, for example, a grid map showing the difference between the maximum and minimum heights of the point cloud within the grid (representing whether the cell is a step) and a grid map showing the maximum height of the point cloud within the grid from a reference point (representing the topographical shape of the cell). The grid map generator 303 removes spike noise and white noise from the generated grid map, detects obstacles with a height above a predetermined level, and generates an occupancy grid map indicating whether or not there are any three-dimensional obstacles in each grid.

[0028] Furthermore, the grid map generation unit 303 determines whether or not a gutter that constitutes an obstacle exists for each grid based on the determination result of the gutter determination unit 306 (described later) as to whether or not a gutter exists. Gutters include, for example, dented ditches that exist on the side of a road. Note that this embodiment will be described taking as an example a case where a gutter that is a dented ditch on a road is detected, but this embodiment is not limited to guttering and can also be applied to determining whether or not a depression exists on a road.

[0029] The gutter determination unit 306 determines whether or not a gutter exists for each grid around the moving object based on the elevation difference and height of the point cloud within the grid. Details of the determination process by the gutter determination unit 306 will be described later.

[0030] The path generation unit 304 generates a travel path for the mobile object 100 to the target position set by the user instruction acquisition unit 301. Specifically, the path generation unit 304 generates a path using an occupancy grid map generated by the grid map generation unit 303 from images captured by the detection unit 114, without requiring obstacle information from a high-precision map. Note that because the detection unit 114 is a stereo camera that captures images of the area ahead of the mobile object 100, it cannot recognize obstacles in other directions. Therefore, it is desirable for the mobile object 100 to store detected obstacle information for a predetermined period of time in order to avoid colliding with obstacles outside the field of view or getting stuck in a dead end. This allows the mobile object 100 to generate a path taking into account both obstacles detected in the past and obstacles detected in real time.

[0031] The path generation unit 304 periodically generates a global path using the occupancy grid map, and also periodically generates local paths that follow the global path. In other words, the target position of the local path is determined by the global path. In this embodiment, the generation period for each path is 100 ms for the global path and 50 ms for the local path, but this is not a limitation of the present invention. Various algorithms are known for generating global paths, such as Rapid-Exploring Random Tree (RRT), Probabilistic Road Map (PRM), and A*. The path generation unit 304 can determine the global path using information on grids in the occupancy grid map where static obstacles exist. On the other hand, when generating local paths, the path generation unit 304 can use information on grids in the occupancy grid map where static obstacles exist and grids in the occupancy grid map where dynamic obstacles exist. In this way, a global route can be generated using information on stable obstacles, and a local route can be generated that is drivable and avoids approaching dynamic obstacles.

[0032] The travel control unit 305 controls the travel of the mobile body 100 according to the local route. Specifically, the travel control unit 305 controls the travel unit 112 according to the local route to control the speed and angular velocity of the mobile body 100. Furthermore, the travel control unit 305 controls the travel in accordance with various operations by the driver. When a deviation occurs in the driving plan of the local route due to an operation by the driver, the travel control unit 305 may again acquire a new local route generated by the route generation unit 304 and control the travel, or may control the speed and angular velocity of the mobile body 100 so as to eliminate the deviation from the local route currently in use.

[0033] <Occupancy Grid Map> Figure 4 shows an occupancy grid map 400 including obstacle information according to this embodiment. Since the mobile body 100 according to this embodiment travels without relying on obstacle information from a high-precision map, all obstacle information is obtained from the recognition results of the detection unit 114. At this time, it is necessary to store obstacle information to avoid collisions with obstacles outside the field of view or getting stuck in dead ends. In this embodiment, an occupancy grid map is used as a method of storing obstacle information from the perspectives of reducing the amount of information in the 3D point cloud of stereo images and making it easier to handle in route planning.

[0034] As described above, the grid map generator 303 generates an occupancy grid map. Here, an example is described in which a predetermined area centered on the mobile object 100 is divided into a grid. However, the area may be divided into other shapes instead of a grid, and an occupancy map indicating the presence or absence of obstacles in each divided area may be generated. The occupancy grid map 400 defines a peripheral area, for example, an area 40 m x 40 m or 20 m x 20 m around the mobile object 100, as a 20 cm x 20 cm or 10 cm x 10 cm grid. The grid information is dynamically set in accordance with the movement of the mobile object 100. In other words, the occupancy grid map 400 is a real-time changing area that is shifted so that the mobile object 100 is always at the center as the mobile object 100 moves. The size of the area can be set arbitrarily depending on the hardware resources of the mobile object 100.

[0035] Furthermore, the occupancy grid map 400 defines, for each divided region (grid), information on the presence or absence of obstacles detected from the image captured by the detection unit 114. For example, the presence or absence information may be defined as "0" for a drivable region and "1" for an undrivable region (i.e., the presence of an obstacle). Note that the occupancy grid map 400 may also set obstacle type information (e.g., a moving obstacle or a static obstacle) for each grid. In FIG. 4 , 401 indicates a grid in which an obstacle exists. An obstacle-existing region includes, for example, a region in which a solid object of 5 cm or more exists, or a region with a depression with a negative height, such as a roadside gutter. Therefore, the mobile body 100 generates a route to avoid these obstacles 401.

[0036] <Storage of Obstacle Information> With reference to FIG. 5 , the storage of obstacle information in an occupancy grid map according to this embodiment will be described. 500 indicates a local map that moves in accordance with the movement of the moving object 100. The local map 500 is shifted in accordance with the movement of the moving object 100 in the x-axis direction and the y-axis direction on the grid map. The local map 500 shows how a dotted line area 501 is deleted and a solid line area 502 is added in accordance with, for example, the amount of movement Δx of the moving object 100 in the x-axis direction. The deleted area is an area opposite to the traveling direction of the moving object 100, and the added area is an area in the traveling direction. Similarly, areas are deleted and added in the y-axis direction in accordance with the movement of the moving object 100.

[0037] The local map 500 also stores information about obstacles that have been detected in the past. If an obstacle is present in a grid included in the deletion area, the obstacle information is deleted from the local map 500, but it is desirable to store the information separately from the local map 500 for a certain period of time. This information is useful, for example, when the moving body 100 changes its course and the deletion area is again included in the local map 500, and can improve the accuracy of the moving body 100 in avoiding obstacles. Furthermore, by using the stored information, there is no need to detect obstacles again, and the processing load can be reduced.

[0038] Furthermore, before the local map 500 is added to the obstacle detection map 510 (described later), a forgetting process is performed on the local map 500 according to a forgetting rate set for each grid. When a dynamic obstacle is detected, if previously detected obstacle information stored in grids is retained, a false detection that the obstacle exists in all grids along the obstacle's movement trajectory may occur. Therefore, to avoid erroneously determining that an obstacle exists in a grid that the obstacle has already passed, it is necessary to forget the accumulated obstacle information after a certain period of time has elapsed. In this embodiment, a forgetting rate is set individually for each grid. Here, the forgetting rate indicates the extent to which the accumulated obstacle information is retained. For example, according to this embodiment, the occupancy grid map is generated periodically, and the forgetting rate indicates the number of cycles for which obstacle information is stored.

[0039] The forgetting rate set for each grid can be set in various ways. For example, in this embodiment, the forgetting rate for forgetting accumulated obstacle information can be set to different values ​​depending on the type of obstacle. As an example, the forgetting rate is set to different values ​​for each grid whose obstacle type is a dynamic obstacle and each grid whose obstacle type is a static obstacle. Since dynamic obstacles are moving or may move, the forgetting rate may be set higher than that for static obstacles (i.e., they are more likely to be forgotten than static obstacles) in order to follow their movement.

[0040] Furthermore, the forgetting rate for forgetting accumulated obstacle information may be set to a different value for each grid included in the field of view range 511 and for grids not included in the field of view range 511. For example, a forgetting rate higher than the forgetting rate for grids overlapping the field of view range 511 may be set. Here, a grid overlapping the field of view range 511 is a grid where at least a predetermined area of ​​the grid overlaps the field of view range 511. The size of the predetermined area is arbitrary and can be set, for example, between 1 and 100%.

[0041] Reference numeral 510 denotes an obstacle detection map showing detection information of obstacles present in the forward vicinity of the moving body 100 from captured images captured by the detection unit 114 of the moving body 100. The obstacle detection map 510 shows real-time information and is periodically generated according to the captured images acquired by the detection unit 114. In the forward region of the moving body 100, within the field of view 511 of the detection unit 114, previously detected obstacles are preferably not stored in a fixed form but are updated using the periodically generated obstacle detection map 510. This allows moving obstacles to be recognized and prevents the generation of unnecessary detours. The obstacle detection map 510 may include obstacle type information (e.g., moving obstacle or stationary obstacle) set for each grid. Meanwhile, in the rear region of the moving body 100 (strictly speaking, outside the field of view of the detection unit 114), information on previously detected obstacles is stored as shown in the local map 500. This allows, for example, when an obstacle is detected in the forward region and a detour route is generated, to easily generate a route that avoids collision with the obstacle that has passed through.

[0042] Reference numeral 520 denotes an occupancy grid map generated by adding together the local map 500 and the obstacle detection map 510. In this way, the occupancy grid map 520 is generated as a grid map that combines the local map and obstacle detection information that change in real time with obstacle information that has been detected and accumulated in the past.

[0043] <Determining the Presence or Absence of a Gutter> The determination of the presence or absence of a gutter (gutter determination process) will be described with reference to Figures 6A to 6E. The gutter determination process is a process for determining whether or not a gutter exists, and for setting the occupancy of an obstacle for each grid based on the determination result. However, as mentioned above, this process is not limited to guttering, and can also be applied to determining whether or not a depression exists on a road.

[0044] 6A shows an example of an image captured when a roadway has a ditch, and may be, for example, one of the stereo images acquired from the detection unit 114. In the example of image 601, a roadway has a ditch and a wall on the right side of the roadway.

[0045] The image information processing unit 302 generates data of a point cloud 613 existing in three-dimensional space by creating a three-dimensional point cloud based on the image acquired by the detection unit 114. The point cloud 613 corresponds to points on the surface of an object, and is therefore generated, for example, on the surface of the asphalt, the surface of the gutter 602, and the wall surface to the right of the gutter 602. When an XY plane orthogonal to the height direction Z is divided into grids, the point cloud 613 includes point clouds at different heights within the divided grids.

[0046] For example, reference numeral 621 schematically shows a point cloud 611 that exists on the side surface of a gutter 602 and is included in a grid 622. The grid 622 includes a point cloud whose height is equal to or less than 0. In this embodiment, in order to determine whether or not a depression such as a gutter exists, the elevation difference of the point cloud is evaluated in addition to the minimum height of the point cloud.

[0047] Because the detection unit 114 attached to the mobile object 100 is used, the minimum height of the point cloud 611 may be less than zero (the height has a negative value) depending on the attitude (e.g., tilt) of the mobile object 100, even though the minimum height of the point cloud 611 is actually equal to or greater than zero. Therefore, if the presence or absence of a gutter is determined by simply determining whether the minimum height of the point cloud is less than zero (the height has a negative value), the determination result may vary due to fluctuations in the minimum height. Therefore, in this embodiment, in addition to the minimum height of the point cloud, it is also determined whether the elevation difference of the point cloud is equal to or greater than a predetermined value. Because the elevation difference of the point cloud is a relative relationship between points, it is less affected by the attitude of the mobile object 100. Therefore, in this embodiment, the gutter determination unit 306 determines the presence of a gutter (depression) when the minimum height of the point cloud 611 included in the grid 622 is less than zero and the elevation difference of the point cloud 611 included in the grid 622 is greater than a predetermined value.

[0048] FIG. 6B shows a specific example of the above-described determination of the presence or absence of a gutter. Reference numeral 612 denotes the position of the detection unit, and reference numeral 634 schematically denotes the location of a gutter. Image 631 indicates the position of a point cloud on the X-Y plane where the elevation difference is equal to or greater than a predetermined value using black dots (points 635). Because there is a wall to the right of the gutter in image 601, image 631 shows that a point cloud with an elevation difference equal to or greater than a predetermined value is distributed inside and to the right of gutter 634. Meanwhile, image 632 indicates the position of a point cloud on the X-Y plane where the minimum elevation of the point cloud is less than 0 using black dots (points 636). Image 633 then shows point 637, obtained by taking the intersection of points 635 and 636, in black, while other points are shown in gray. In the example shown in Figure 6B, almost all of points 636 are located in the gutter, but if some of points 636 are also located on the asphalt, the points located on the asphalt will be excluded from points 637 by taking into account the elevation difference of the points as described above.

[0049] Next, the filtering process applied by the grid map generation unit 303 will be described with reference to FIGS. 6C and 6D . Before the gutter determination unit 306 determines whether a gutter exists, the grid map generation unit 303 performs a filtering process to remove outliers in the height direction from the generated point cloud. The filtering process is performed to remove exceptional point clouds (or points) generated in the depth image generation and point cloud generation processes in the image information processing unit 302 from the processing target. In this embodiment, the strength of the filter is different when evaluating the point cloud in the grid to determine whether a gutter (depression) exists and when evaluating the point cloud in the grid for purposes other than gutter determination (e.g., determining an obstacle with a positive height). That is, the first filter used when evaluating the point cloud in the grid to determine a gutter (depression) has different filter characteristics in the height direction from the second filter used when evaluating the point cloud in the grid for purposes other than gutter (depression) determination.

[0050] In FIG. 6C , point cloud 630, among the point clouds within grid 622, is a point cloud whose variance in the height direction of the point cloud within grid 622 is greater than a predetermined value. When a first filter (for gutter determination) is applied, point clouds whose variance in the height direction of the point cloud is greater than a first threshold are filtered (removed). Point cloud 630 whose variance in the height direction of the point cloud is greater than the predetermined value but less than the first threshold is not removed. In this case, as shown in FIG. 6D , the gutter determination unit 306 treats point cloud 630 in the same way as point cloud 611 and calculates the minimum height and the height difference in the point cloud including point cloud 611 and point cloud 630. This is because gutter (or depressions on a road) have similar brightness and color in the image of the step portion, and a sparse point cloud like point cloud 630 may be generated during the depth image generation and point cloud generation processes in the image information processing unit 302. If a sparse point cloud like point cloud 630 is filtered (as with other processes), point cloud 611 may remain only in shallow areas in the Z direction, which may reduce the accuracy of gutter detection. For this reason, when performing gutter detection processing, the gutter detection unit 306 calculates the minimum height and height difference in the point cloud, while allowing for a certain degree of sparseness in the height direction. This can improve the accuracy of gutter detection.

[0051] On the other hand, when a second filter (e.g., for determining obstacles with a positive height direction) is applied, point clouds whose variance in the height direction of the point cloud is greater than a second threshold value are filtered. In this case, the second threshold value is smaller than the first threshold value. That is, the first filter results in a larger variance in the height direction of the point cloud in the grid after application than when the second filter is applied.

[0052] The gutter determination unit 306 determines whether a gutter exists in each grid. For example, the gutter determination unit 306 determines that a gutter (depression) exists when, for example, the elevation difference between the points in the grid is equal to or greater than a predetermined elevation difference threshold and the height of at least one of the points in the grid is lower than a predetermined height threshold. The grid map generation unit 303 then sets the obstacle occupancy for each grid based on the determination result of the gutter determination unit 306 as to whether a gutter exists. For example, in FIG. 6E , the grid map generation unit 303 sets the obstacle occupancy for the grid indicated by 641 based on the determination result that a gutter exists in the area indicated by 641. The grid map generation unit 303 may reflect the grid in area 641 as an area where an obstacle exists, similar to the area indicated by the obstacle 401. Alternatively, the grid map generation unit 303 may distinguish the grid in area 641 from the area indicated by the tall obstacle 401 by determining that a depression exists.

[0053] <Route Generation> A travel route generated in the mobile body 100 according to this embodiment will be described with reference to Fig. 7. The route generation unit 304 according to this embodiment periodically generates a global path 702 using an occupancy grid map in accordance with a set target position 701, and further periodically generates a local path 703 so as to follow the global path.

[0054] The target position 701 is set based on various instructions. For example, these include instructions from a passenger aboard the vehicle 100 or instructions from a user outside the vehicle 100. Instructions from the passenger are given via the operation panel 131 or the voice input device 133. Instructions given via the operation panel 131 may be given by specifying a specific grid on a grid map displayed on the operation panel 131. In this case, the size of each grid may be set large so that a wider area of ​​the map can be selected. Instructions given via the voice input device 133 may be given using nearby landmarks as landmarks. The landmarks may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like, which are included in the spoken information. Upon receiving an instruction via the voice input device 133, the path generation unit 304 detects the specified landmark from the captured image acquired by the detection unit 114 and sets it as the target position.

[0055] Machine learning models are used for these speech recognition and image recognition. The machine learning model, for example, performs calculations using a deep learning algorithm using a deep neural network (DNN) to recognize place names, landmark names such as buildings, store names, and target names contained in speech information and image information. The speech recognition DNN becomes trained by performing a learning stage process, and by inputting new speech information into the trained DNN, recognition processing (inference stage processing) for the new speech information can be performed. In addition, the image recognition DNN can recognize passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, and motorcycles contained in images.

[0056] Furthermore, instructions from a user outside the vehicle 100 can be sent via the user's own communication device 140 to the vehicle 100 via the communication unit 136, or the user can call the vehicle 100 by raising their hand toward the vehicle 100 as shown in Fig. 7. Instructions from the communication device 140 are given by operation input or voice input, similar to the instructions from the occupant.

[0057] Once the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupancy grid map. As described above, various algorithms such as RRT, PRM, and A* are known as methods for generating a global path, but any method may be used. Next, the path generation unit 304 generates a local path 703 so as to follow the generated global path 702. There are various methods for planning a local path, such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0058] 8 is a flowchart showing the basic control of the mobile body 100 according to this embodiment. The processing described below is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing the program.

[0059] In S101, the control unit 130 sets a target position of the moving object 100 based on a user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be received in various ways. Next, in S102, the control unit 130 acquires a captured image and depth information. Specifically, the control unit 130 captures an image of the area in front of the moving object 100 using the detection unit 114 and acquires the captured image. The acquired captured image is processed by the image information processing unit 302, and a depth image is created and converted into a three-dimensional point cloud (depth information is generated).

[0060] In S103, the control unit 130 recognizes a specific target object using the image information processing unit 302 and identifies the recognized target object as an obstacle. The obstacle can be identified by any method, but for example, the target object is recognized using a DNN from an image captured by a monocular camera, and whether the target object is a static or dynamic obstacle is identified according to a predetermined correspondence relationship between the target object and a dynamic obstacle and a static obstacle. In this embodiment, gutters (depressions) are not identified in the obstacle identification process in S103.

[0061] In S104, the control unit 130 generates an occupancy grid map using the grid map generation unit 303. For example, the control unit 130 detects obstacles, e.g., three-dimensional objects of 5 cm or more, from the 3D point cloud image, and generates an occupancy grid map of a predetermined area centered on the mobile object 100 based on the detected obstacles and the position information of the mobile object 100. The grid map generation unit 303 may map the targets identified in S103 on the occupancy grid map as moving obstacles or static obstacles. The grid map generation unit 303 also generates the occupancy grid map based on the determination result of the gutter determination unit 306, which is executed using the method described with reference to FIGS. 6A to 6E. Details of the process will be described using FIG. 9.

[0062] Next, in S105, the control unit 130 generates a driving path for the mobile object 100 using the path generation unit 304. As described above, the path generation unit 304 generates a global path using the occupancy grid map, and generates local paths according to the generated global path. Subsequently, in S106, the control unit 130 determines the speed and angular velocity of the mobile object 100 according to the generated local path, and controls the driving. Thereafter, in S107, the control unit 130 determines whether the mobile object 100 has reached the target position based on position information from the GNSS sensor 134. If the target position has not been reached, the process returns to S102, and the process of generating a path and controlling the driving is repeated while updating the occupancy grid map. On the other hand, if the target position has been reached, the process of this flowchart ends.

[0063] 9 is a flowchart showing the detailed processing steps of the occupancy grid map generation process (S104) according to this embodiment. The processing described below is realized by the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it.

[0064] In S201, the control unit 130 first forgets the accumulated information about obstacles. The forgetting process can be any process, but a new accumulated value is calculated by multiplying the accumulated value of obstacles accumulated for each grid by a forgetting rate (e.g., a value between 0 and 1) previously set for each grid (e.g., in S208). For example, the following can be set: Obstacle Accumulation Value (t) = Accumulation Value (t-1) * Forgetting Rate + New Accumulation Value. For example, if a new accumulated value is set to a predetermined value when an obstacle is detected in a grid and set to 0 when no obstacle is detected, the accumulated value remains while the obstacle is detected, and decreases according to the forgetting rate when the obstacle is no longer detected. A maximum accumulated value may be set when obstacles continue to be detected. Performing the forgetting process generates a local map 500, for example, as shown in FIG. 5.

[0065] Next, in S202, the control unit 130 sets a grid based on the point cloud information. Then, the gutter determination unit 306 applies the first filter shown in FIGS. 6C and 6D to filter point clouds whose variance in the height direction is greater than a first threshold. In S203, the gutter determination unit 306 evaluates the minimum height and elevation difference of the point cloud within the grid (as shown in FIG. 6D) to determine whether a gutter exists. For example, the gutter determination unit 306 determines that a gutter exists if the minimum height of the point cloud is lower than a predetermined height threshold and the elevation difference is greater than a predetermined elevation difference threshold.

[0066] In S204, if the gutter determination unit 306 determines that a gutter exists, the grid map generation unit 303 proceeds to S207; otherwise, the grid map generation unit 303 proceeds to S205. In S205, the grid map generation unit 303 applies a second filter (for example, to determine obstacles with a positive height direction) to the grid point cloud. In S206, the grid map generation unit 303 uses the information on the point cloud after the second filter has been applied to detect obstacles that are, for example, three-dimensional objects of 5 cm or more, and identifies them as obstacle grids. Meanwhile, in S207, the grid map generation unit 303 identifies the grid in which the gutter determination unit 306 determined that a gutter exists as, for example, an obstacle grid.

[0067] In the above example, the first filter is first applied to the grids in step S202, and then the second filter is applied to the grids that are not gutters. However, this embodiment is not limited to this example. For example, it may be possible to determine whether the minimum height of the point cloud of each grid is equal to or greater than 0, and apply the first filter only to grids whose minimum height of the point cloud is less than 0, and apply the second filter to other grids.

[0068] In S208, the control unit 130 sets a forgetting rate for each grid. For example, the grid map generation unit 303 sets the forgetting rate for forgetting accumulated obstacle information to a different value depending on the type of obstacle. If a moving obstacle is set, the forgetting rate for the moving obstacle may be set to a higher value than the forgetting rate for a static obstacle. Through the processing of S202 to S208, the grid map generation unit 303 generates, for example, the obstacle detection map 510 shown in FIG. 5.

[0069] In S209, the control unit 130 generates an occupancy grid map with updated obstacle information, for example, by adding the map that has been subjected to the forgetting process in S201 and the obstacle detection map generated in S202 to S208. When the control unit 130 completes the process of S209, it ends the process of this flowchart and proceeds to S105. Note that the processes of S201 to S209 are performed periodically, for example, at a cycle of 10 Hz.

[0070] As described above, in the above-described embodiment, point cloud information indicating the object surface in three-dimensional space around the moving body 100 is generated, and the gutter determination unit 306 determines whether a depression exists on the road for each divided area (grid) around the moving body based on the elevation difference and height of the point cloud within the divided area. This makes it possible to accurately detect depressed areas around the moving body. Furthermore, in the above-described embodiment, a first filter is used when evaluating the point cloud within a divided area to determine a depression, and a second filter with different height characteristics is used when evaluating the point cloud within a divided area whose height is greater than 0. This improves the accuracy of detecting depressed areas even when the brightness and color of the image of the portion that constitutes the step of the depression are similar.

[0071] <Summary of embodiments> (Item 1) A mobile body control system comprising: a point cloud generation means for generating point cloud information indicating the surface of an object in a three-dimensional space around a mobile body based on an image of the area around the mobile body; a depression determination means for determining whether or not a depression exists on a road for each divided area around the mobile body based on the elevation difference and height of the point cloud within the divided area; and a map generation means for generating an occupancy map indicating the occupancy of an obstacle for each divided area based on the determination result by the depression determination means.

[0072] According to this embodiment, it is possible to accurately detect a recessed area around a moving object.

[0073] (Item 2) The mobile object control system according to item 1, wherein the depression determination means determines that the depression exists in the previous divided area when the elevation difference of the point cloud within the divided area is equal to or greater than a first threshold value and the elevation of at least one of the point clouds within the divided area is lower than a second threshold value.

[0074] According to this embodiment, it is possible to reduce detection errors due to the tilt of the moving body and to detect recessed areas with high accuracy.

[0075] (Item 3) The mobile object control system according to item 1 or 2, further comprising a filtering means for removing point clouds that have outliers in the height direction from the generated point cloud information when evaluating point clouds within the divided region for determination by the dent determination means.

[0076] According to this embodiment, exceptional point clouds (or points) generated in the depth image generation and point cloud generation processes can be excluded from the processing target.

[0077] (Item 4) The mobile object control system according to Item 3, wherein the filtering means has a first filter for evaluating a point cloud within the divided region for determination by the depression determination means, and a second filter having different characteristics in the height direction from the first filter for evaluating a point cloud within the divided region having a height greater than 0.

[0078] According to this embodiment, even when the brightness and color of the portions on the image that become the step of the depression are similar, it is possible to improve the accuracy of detecting the depression region.

[0079] (Item 5) The mobile object control system according to Item 4, wherein the first filter is configured to increase the vertical dispersion of the point cloud within the divided region after application of the first filter compared to when the second filter is applied.

[0080] According to this embodiment, it is possible to improve the detection accuracy when the brightness or color of the portion on the image that becomes the step of the recess is similar.

[0081] (Item 6) The mobile body control system according to any one of items 1 to 5, wherein, when the depression determination means determines that the depression exists in the divided area, the map generation means reflects the divided area as an area having the obstacle in the occupation map.

[0082] According to this embodiment, an occupancy map for pothole avoidance can be used.

[0083] (Item 7) The mobile object control system according to any one of items 1 to 6, wherein the recess is a side ditch.

[0084] According to this embodiment, an occupancy map can be generated that corresponds to the case where a roadway has a gutter.

[0085] (Item 8) A control method for a mobile object control system, comprising: a point cloud generation process for generating point cloud information indicating object surfaces in a three-dimensional space around a mobile object based on an image of the area around the mobile object; a depression determination process for determining whether or not a depression exists on a track for each divided area around the mobile object based on the elevation difference and height of the point cloud within the divided area; and a map generation process for generating an occupancy map indicating the occupancy of an obstacle for each divided area based on the determination result in the depression determination process.

[0086] According to this embodiment, it is possible to accurately detect a recessed area around a moving object.

[0087] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0088] 100...mobile body, 111...seat, 112...traveling unit, 113...battery, 114...detection unit, 120...front wheel, 121...tail wheel, 122a, 122b...motor, 123...steering mechanism, 130...control unit, 131...operation panel, 132...speaker, 133...voice input device, 134...GNSS sensor, 135...storage device, 136...communication unit, 140...communication device

Claims

1. A mobile object control system comprising: a point cloud generation means for generating point cloud information indicating the surface of an object in three-dimensional space around a mobile object based on an image of the area around the mobile object; a depression determination means for determining whether or not a depression exists on a road for each divided area around the mobile object based on the elevation difference and height of the point cloud within the divided area; and a map generation means for generating an occupancy map indicating the occupancy of an obstacle for each divided area based on the determination result by the depression determination means.

2. The mobile body control system described in claim 1, characterized in that the depression determination means determines that the depression exists in the previous divided area when the elevation difference of the point cloud within the divided area is equal to or greater than a first threshold and the elevation of at least one of the point clouds within the divided area is lower than a second threshold.

3. The mobile control system described in claim 1, further comprising a filtering means for removing point clouds that have outliers in the height direction from the generated point cloud information when evaluating the point clouds within the divided area for judgment by the depression judgment means.

4. The mobile body control system described in claim 3, characterized in that the filtering means has a first filter for evaluating the point cloud within the divided area for judgment by the dent judgment means, and a second filter having different characteristics in the height direction from the first filter for evaluating the point cloud within the divided area whose height is greater than 0.

5. The mobile control system described in claim 4, characterized in that the first filter is configured so that the vertical variance of the point cloud within the divided area after application is larger than when the second filter is applied.

6. The mobile control system according to claim 1, characterized in that, when the depression determination means determines that a depression exists in the divided area, the map generation means reflects the divided area in the occupancy map as an area having the obstacle.

7. The mobile object control system according to claim 1, wherein the recess is a gutter.

8. A control method for a mobile object control system, comprising: a point cloud generation step of generating point cloud information indicating the surface of an object in three-dimensional space around a mobile object based on an image of the area around the mobile object; a depression determination step of determining whether or not a depression exists on a road for each divided area around the mobile object based on the elevation difference and height of the point cloud within the divided area; and a map generation step of generating an occupancy map indicating the occupancy of an obstacle for each divided area based on the determination result in the depression determination step.

9. A program for causing a computer to function as each means of the mobile object control system according to any one of claims 1 to 7.

10. A storage medium storing a program for causing a computer to function as each means of the mobile object control system according to any one of claims 1 to 7.

Images