Mobile body control system, control method therefor, program, and storage medium

Abstract

This mobile body control system comprises: a point cloud generation unit that generates information on a point cloud indicating an object surface in a three-dimensional space around a mobile body, on the basis of a captured image of the periphery of the mobile body; an obstacle determination unit that determines, for each divided region of the periphery of the mobile body, whether an obstacle is present in the divided region on the basis of whether a first condition related to the height of the point cloud in the divided region with the moving body as a reference is satisfied; and a map generation unit that generates an occupancy map indicating the occupancy of the obstacle for each divided region, on the basis of the determination result by the obstacle determination unit.

Description

Mobile object control system, control method thereof, program, and storage medium

[0001] The present invention relates to a mobile object control system, a control method therefor, a program, and a storage medium.

[0002] In recent years, there have been known autonomously moving bodies, such as vehicles, that recognize the state of the surrounding roadway using captured images of the surroundings and generate a route for themselves to travel. Patent Document 1 describes an autonomously moving robot that uses a sensor such as LiDAR to recognize surrounding objects and generate a map for travel in order to detect concave areas.

[0003] Japanese Patent Application Laid-Open No. 2023-121291

[0004] The technique proposed in Patent Document 1 leaves room for improvement in the accuracy of obstacle detection. Some aspects of the present invention provide a technique that can accurately detect an area including an obstacle.

[0005] According to some embodiments of the present invention, there is provided a mobile body control system comprising: a point cloud generation means for generating point cloud information indicating the surface of an object in three-dimensional space around a mobile body based on an image of the area around the mobile body; an obstacle determination means for determining, for each divided area around the mobile body, whether an obstacle is present in the divided area based on whether a first condition regarding the height of the point cloud within the divided area relative to the mobile body is satisfied; and a map generation means for generating an occupancy map indicating the occupancy of obstacles for each divided area based on the determination result by the obstacle determination means.

[0006] According to some embodiments of the present invention, areas containing obstacles can be detected with high accuracy.

[0007] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0008] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments of the present invention, and are used, together with the description, to explain the principles of the present invention. A block diagram showing an example of the hardware configuration of a moving body according to the present embodiment. A block diagram showing an example of the hardware configuration of a moving body according to the present embodiment. A block diagram showing the control configuration of a moving body according to the present embodiment. A block diagram showing the functional configuration of a control unit according to the present embodiment. A diagram showing an occupancy grid map according to the present embodiment. A diagram showing a method for generating an occupancy grid map according to the present embodiment. A diagram explaining obstacle conditions according to the present embodiment. A diagram explaining obstacle conditions according to the present embodiment. A diagram explaining height conditions according to the present embodiment. A diagram showing global routes and local routes according to the present embodiment. A flowchart showing a processing procedure for controlling the travel of a moving body according to the present embodiment. A flowchart showing a procedure for generating an occupancy grid map according to the present embodiment.

[0009] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be combined in any desired manner. Furthermore, the same reference numerals are used to designate identical or similar components, and redundant descriptions will be omitted.

[0010] <Configuration of Mobile Body> The configuration of a mobile body 100 according to this embodiment will be described with reference to Figures 1A and 1B. Figure 1A shows a side view of the mobile body 100 according to this embodiment, and Figure 1B shows the internal configuration of the mobile body 100. In the figures, arrow X indicates the front-to-rear direction of the mobile body 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the mobile body 100.

[0011] The mobile body 100 is equipped with a battery 113 and is, for example, an ultra-compact mobility vehicle that moves primarily by motor power. An ultra-compact mobility vehicle is more compact than a typical automobile and has a passenger capacity of approximately one or two people. In this embodiment, a three-wheeled ultra-compact mobility vehicle is described as an example of the mobile body 100. However, this is not intended to limit the present invention, and the mobile body 100 may also be, for example, a four-wheeled vehicle or a saddle-type vehicle. Furthermore, the mobile body of the present invention is not limited to a vehicle, but may also be a vehicle that carries luggage and runs alongside a person walking, or a vehicle that leads a person. Furthermore, the mobile body control system of this embodiment may be a mobile body, a control device such as an ECU included in the mobile body, or a cloud-based information processing server for controlling the mobile body. In other words, some or all of the processing described below according to this embodiment may be executed in the mobile body or in a cloud-based information processing server. Furthermore, the present invention is not limited to four-wheeled or two-wheeled vehicles, but may also be applied to autonomously moving robots and the like.

[0012] The mobile body 100 is an electric autonomous vehicle equipped with a propulsion unit 112 and using a battery 113 as its main power source. The battery 113 is, for example, a secondary battery such as a lithium-ion battery, and the mobile body 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 is equipped with, for example, a pair of left and right front wheels 120 and a tail wheel (driven wheel) 121. The propulsion unit 112 may be in another form, such as a four-wheeled vehicle. The mobile body 100 is equipped with a seat 111 for one or two people.

[0013] The traveling unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that uses motors 122a, 122b as drive sources to change the steering angle of the pair of front wheels 120. Changing the steering angle of the pair of front wheels 120 can change the traveling direction of the mobile body 100. The tail wheel 121 does not have its own drive source, but is a driven wheel that operates in response to the drive of the pair of front wheels 120. The tail wheel 121 is also connected to the body of the mobile body 100 via a swivel. The swivel rotates so that the direction of the tail wheel 121 changes separately from the rotation of the tail wheel 121. In this way, the mobile body 100 according to this embodiment employs a differential two-wheel mobility with a tail wheel, but is not limited to this.

[0014] The moving body 100 includes a detection unit 114 that recognizes a plane in front of the moving body 100. The detection unit 114 is an external sensor that monitors the area in front of the moving body 100. In this embodiment, the detection unit 114 is an imaging device that captures an image in front of the moving body 100. In this embodiment, the detection unit 114 includes, for example, a stereo camera having optical systems such as two lenses and respective image sensors, and a monocular camera. However, it is also possible to employ radar or lidar (light detection and ranging) instead of or in addition to the imaging device. Furthermore, in this embodiment, an example in which the detection unit 114 is provided only in front of the moving body 100 is described, but this is not intended to limit the present invention, and the detection unit 114 may be provided behind, on the left or right sides of the moving body 100. Furthermore, instead of using a monocular camera, an image captured by one of the stereo cameras may be used.

[0015] The mobile body 100 according to this embodiment captures an image of the area ahead of the mobile body 100 using the detection unit 114 and detects obstacles from the captured image. Furthermore, the mobile body 100 divides the area around the mobile body 100 into a grid and controls its travel while generating an occupancy grid map in which obstacle information is accumulated for each divided area (hereinafter also referred to as a grid). The occupancy grid map will be described in detail later.

[0016] <Control Configuration of the Mobile Body> FIG. 2 is a block diagram of a control system of the mobile body 100 according to this embodiment. Here, the configuration necessary for implementing the present invention will be mainly described. Therefore, other configurations may be included in addition to the configurations described below. Furthermore, in this embodiment, the mobile body 100 is described as including each of the components described below, but this is not intended to limit the present invention. A mobile body control system including multiple devices may be realized. For example, some functions of the control unit 130 may be implemented by an information processing server connected to the mobile body 100 in a communicable manner, or the detection unit 114 and the GNSS sensor 134 may be provided as external devices. The mobile body 100 includes a control unit (ECU) 130. The control unit 130 includes a processor, such as a CPU, a storage device such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used by the processor for processing, and the like. Multiple sets of processors, storage devices, and interfaces may be provided for different functions of the mobile body 100 and configured to communicate with each other.

[0017] The control unit 130 acquires the detection results of the detection unit 114, input information from the operation panel 131, audio information input from the audio input device 133, position information from the GNSS sensor 134, and information received via the communication unit 136, and executes corresponding processing. The control unit 130 controls the motors 122a and 122b (travel control of the traveling unit 112), controls the display on the operation panel 131, and notifies and outputs information to the occupants of the mobile object 100 by audio from the speaker 132.

[0018] The voice input device 133 collects the voices of the occupants of the mobile body 100. The control unit 130 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals to detect the current position of the mobile body 100. The storage device 135 is a storage device that stores images captured by the detection unit 114, obstacle information, previously generated routes, and occupancy grid maps. The storage device 135 may also store programs executed by the processor, data used for processing by the processor, and the like. The storage device 135 may also store various parameters (e.g., learned parameters and hyperparameters of a deep neural network) of machine learning models for voice recognition and image recognition executed by the control unit 130.

[0019] The communication unit 136 communicates with a communication device 140, which is an external device, via wireless communication such as Wi-Fi or fifth-generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, or the like. The communication device 140 connects to a network via wireless communication such as Wi-Fi or fifth-generation mobile communication.

[0020] A user who owns the communication device 140 can issue instructions to the mobile body 100 via the communication device 140. The instructions include, for example, an instruction to call the mobile body 100 to a location desired by the user and join the mobile body 100. Upon receiving the instruction, the mobile body 100 sets a target position based on the position information included in the instruction. In addition to such instructions, the mobile body 100 can also set a target position from an image captured by the detection unit 114 or based on an instruction from a user riding in the mobile body 100 via the operation panel 131. When setting a target position from a captured image, for example, a person raising their hand toward the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target position.

[0021] <Functional Configuration of Mobile Body> Next, the functional configuration of the mobile body 100 according to this embodiment will be described with reference to Fig. 3. The functional configuration described here is realized by a processor, such as a CPU, in the control unit 130 reading a program stored in a memory, such as a ROM, into a RAM and executing it. Note that the functional configuration described below describes only the functions necessary for explaining the present invention, and does not describe all of the functional configuration actually included in the mobile body 100. In other words, the functional configuration of the mobile body 100 according to the present invention is not limited to the functional configuration described below.

[0022] The user instruction acquisition unit 301 has a function of accepting instructions from the user, and can accept user instructions via the operation panel 131, user instructions from an external device such as the communication device 140 via the communication unit 136, and user spoken instructions via the voice input device 133. As described above, user instructions include an instruction to set a target position (also referred to as a destination) of the mobile body 100 and instructions related to driving control of the mobile body 100.

[0023] The image information processing unit 302 processes the captured images acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image from the stereo images acquired by the detection unit 114 and converts it into a 3D point cloud. The 3D point cloud image data (also referred to as depth information) is used to detect targets present around the mobile body 100. The targets present around the mobile body 100 include the ground. In this manner, the image information processing unit 302 includes a point cloud generation means for generating point cloud information indicating the surface of an object in a 3D space based on the captured image information. The 3D point cloud image data includes position information of multiple points. The position of each point is represented in a 3D Cartesian coordinate system with the mobile body 100 as the origin. In the following description, the height of a point included in the point cloud refers to the height in the 3D Cartesian coordinate system.

[0024] The image information processing unit 302 may also include a machine learning model that processes image information and execute a learning stage process or an inference stage process of the machine learning model. For example, the image information processing unit 302 may identify targets included in images captured by a monocular camera. For example, the image information processing unit 302 may use the machine learning model to identify the type of target and to identify whether each target type is a predetermined dynamic target or a static target. Dynamic targets are targets that can move autonomously and include, for example, vehicles or other traffic participants such as pedestrians and bicycles. Static targets are targets that do not move autonomously and include, for example, objects such as signs and guardrails.

[0025] The obstacle determination unit 306 determines whether an obstacle exists for each grid around the mobile body 100 based on the position of the point cloud within the grid. An obstacle is a target that obstructs the movement (e.g., driving) of the mobile body 100. Details of the determination process by the obstacle determination unit 306 will be described later. A dynamic target determined to be an obstacle will be referred to as a dynamic obstacle. A static target determined to be an obstacle will be referred to as a static obstacle. As will be described later, some areas of the ground can also be obstacles (specifically, static obstacles).

[0026] The grid map generator 303 creates a grid map of a predetermined size (for example, each cell is 10 cm x 10 cm in a 20 m x 20 m area) based on the 3D point cloud data. This is done to reduce the amount of data required for real-time processing, as the amount of 3D point cloud data is large.

[0027] The path generation unit 304 generates a travel path for the mobile object 100 to the target position set by the user instruction acquisition unit 301. Specifically, the path generation unit 304 generates a path using an occupancy grid map generated by the grid map generation unit 303 from images captured by the detection unit 114, without requiring obstacle information from a high-precision map. Note that because the detection unit 114 is a stereo camera that captures images of the area ahead of the mobile object 100, it cannot recognize obstacles in other directions. Therefore, it is desirable for the mobile object 100 to store detected obstacle information for a predetermined period of time in order to avoid colliding with obstacles outside the field of view or getting stuck in a dead end. This allows the mobile object 100 to generate a path taking into account both obstacles detected in the past and obstacles detected in real time.

[0028] The path generation unit 304 periodically generates a global path using the occupancy grid map, and also periodically generates local paths that follow the global path. In other words, the target position of the local path is determined by the global path. In this embodiment, the generation period for each path is 100 ms for the global path and 50 ms for the local path, but this is not a limitation of the present invention. Various algorithms are known for generating global paths, such as Rapid-Exploring Random Tree (RRT), Probabilistic Road Map (PRM), and A*. The path generation unit 304 can determine the global path using information on grids in the occupancy grid map where static obstacles exist. On the other hand, when generating local paths, the path generation unit 304 can use information on grids in the occupancy grid map where static obstacles exist and grids in the occupancy grid map where dynamic obstacles exist. In this way, a global route can be generated using information on stable obstacles, and a local route can be generated that is drivable and avoids approaching dynamic obstacles.

[0029] The travel control unit 305 controls the travel of the mobile body 100 according to the local route. Specifically, the travel control unit 305 controls the travel unit 112 according to the local route to control the speed and angular velocity of the mobile body 100. Furthermore, the travel control unit 305 controls the travel in accordance with various operations by the driver. When a deviation occurs in the driving plan of the local route due to an operation by the driver, the travel control unit 305 may again acquire a new local route generated by the route generation unit 304 and control the travel, or may control the speed and angular velocity of the mobile body 100 so as to eliminate the deviation from the local route currently in use.

[0030] <Occupancy Grid Map> Figure 4 shows an occupancy grid map 400 including obstacle information according to this embodiment. Since the mobile body 100 according to this embodiment travels without relying on obstacle information from a high-precision map, all obstacle information is obtained from the recognition results of the detection unit 114. At this time, it is necessary to store obstacle information to avoid collisions with obstacles outside the field of view or getting stuck in dead ends. In this embodiment, an occupancy grid map is used as a method of storing obstacle information from the perspectives of reducing the amount of information in the 3D point cloud of stereo images and making it easier to handle in route planning.

[0031] As described above, the grid map generator 303 generates an occupancy grid map. Here, an example is described in which a predetermined area centered on the mobile object 100 is divided into a grid. However, the area may be divided into other shapes instead of a grid, and an occupancy map indicating the presence or absence of obstacles in each divided area may be generated. The occupancy grid map 400 defines a peripheral area, for example, an area 40 m x 40 m or 20 m x 20 m around the mobile object 100, as a 20 cm x 20 cm or 10 cm x 10 cm grid. The grid information is dynamically set in accordance with the movement of the mobile object 100. In other words, the occupancy grid map 400 is a real-time changing area that is shifted so that the mobile object 100 is always at the center as the mobile object 100 moves. The size of the area can be set arbitrarily depending on the hardware resources of the mobile object 100.

[0032] Furthermore, in the occupancy grid map 400, information on the presence or absence of obstacles detected from the image captured by the detection unit 114 is defined for each divided area (grid 401). For example, the presence or absence information is defined as "0" for a drivable area and "1" for an undrivable area (i.e., the presence of an obstacle). Note that in the occupancy grid map 400, obstacle type information (e.g., a moving obstacle or a static obstacle) may be set for each grid 401. In FIG. 4, grids 401 in which obstacles exist are hatched. The mobile body 100 generates a route so as to avoid grids 401 in which obstacles exist.

[0033] <Storage of Obstacle Information> With reference to FIG. 5 , the storage of obstacle information in an occupancy grid map according to this embodiment will be described. 500 indicates a local map that moves in accordance with the movement of the moving object 100. The local map 500 is shifted in accordance with the movement of the moving object 100 in the x-axis direction and the y-axis direction on the grid map. The local map 500 shows how a dotted line area 501 is deleted and a solid line area 502 is added in accordance with, for example, the amount of movement Δx of the moving object 100 in the x-axis direction. The deleted area is an area opposite to the traveling direction of the moving object 100, and the added area is an area in the traveling direction. Similarly, areas are deleted and added in the y-axis direction in accordance with the movement of the moving object 100.

[0034] The local map 500 also stores information about obstacles that have been detected in the past. If an obstacle is present in a grid included in the deletion area, the obstacle information is deleted from the local map 500, but it is desirable to store the information separately from the local map 500 for a certain period of time. This information is useful, for example, when the moving body 100 changes its course and the deletion area is again included in the local map 500, and can improve the accuracy of the moving body 100 in avoiding obstacles. Furthermore, by using the stored information, there is no need to detect obstacles again, and the processing load can be reduced.

[0035] Furthermore, before the local map 500 is added to the obstacle detection map 510 (described later), a forgetting process is performed on the local map 500 according to a forgetting rate set for each grid. When a dynamic obstacle is detected, if previously detected obstacle information stored in grids is retained, a false detection that the obstacle exists in all grids along the obstacle's movement trajectory may occur. Therefore, to avoid erroneously determining that an obstacle exists in a grid that the obstacle has already passed, it is necessary to forget the accumulated obstacle information after a certain period of time has elapsed. In this embodiment, a forgetting rate is set individually for each grid. Here, the forgetting rate indicates the extent to which the accumulated obstacle information is retained. For example, according to this embodiment, the occupancy grid map is generated periodically, and the forgetting rate indicates the number of cycles for which obstacle information is stored.

[0036] The forgetting rate set for each grid can be set in various ways. For example, in this embodiment, the forgetting rate for forgetting accumulated obstacle information can be set to different values ​​depending on the type of obstacle. As an example, the forgetting rate is set to different values ​​for each grid whose obstacle type is a dynamic obstacle and each grid whose obstacle type is a static obstacle. Since dynamic obstacles are moving or may move, the forgetting rate may be set higher than that for static obstacles (i.e., they are more likely to be forgotten than static obstacles) in order to follow their movement.

[0037] Furthermore, the forgetting rate for forgetting accumulated obstacle information may be set to a different value for each grid included in the field of view range 511 and for grids not included in the field of view range 511. For example, a forgetting rate higher than the forgetting rate for grids overlapping the field of view range 511 may be set. Here, a grid overlapping the field of view range 511 is a grid where at least a predetermined area of ​​the grid overlaps the field of view range 511. The size of the predetermined area is arbitrary and can be set, for example, between 1 and 100%.

[0038] Reference numeral 510 denotes an obstacle detection map showing detection information of obstacles present in the forward vicinity of the moving body 100 from captured images captured by the detection unit 114 of the moving body 100. The obstacle detection map 510 shows real-time information and is periodically generated according to the captured images acquired by the detection unit 114. In the forward region of the moving body 100, within the field of view 511 of the detection unit 114, previously detected obstacles are preferably not stored in a fixed form but are updated using the periodically generated obstacle detection map 510. This allows moving obstacles to be recognized and prevents the generation of unnecessary detours. The obstacle detection map 510 may include obstacle type information (e.g., moving obstacle or stationary obstacle) set for each grid. Meanwhile, in the rear region of the moving body 100 (strictly speaking, outside the field of view of the detection unit 114), information on previously detected obstacles is stored as shown in the local map 500. This allows, for example, when an obstacle is detected in the forward region and a detour route is generated, to easily generate a route that avoids collision with the obstacle that has passed through.

[0039] Reference numeral 520 denotes an occupancy grid map generated by adding together the local map 500 and the obstacle detection map 510. In this way, the occupancy grid map 520 is generated as a grid map that combines the local map and obstacle detection information that change in real time with obstacle information that has been detected and accumulated in the past.

[0040] <Determining the Presence or Absence of an Obstacle> The determination of the presence or absence of an obstacle (obstacle determination process) will be described with reference to Figures 6A to 6C. The obstacle determination process is a process for determining whether an obstacle is present and for setting the obstacle occupancy for each grid 401 based on the determination result. The obstacle determiner 306 determines, for each grid 401, whether a condition for determining that an obstacle is present in the grid 401 is satisfied. In the following description, the condition for determining that an obstacle is present in the grid 401 is referred to as an obstacle condition. If the obstacle condition is satisfied for one grid 401 to be determined, the obstacle determiner 306 determines that an obstacle is present in this grid 401, and if the obstacle condition is not satisfied, the obstacle determiner 306 determines that an obstacle is not present in this grid 401. When the obstacle condition includes multiple conditions, the obstacle determination unit 306 may determine that the obstacle condition is satisfied (i.e., an obstacle exists) when at least one of the multiple conditions is satisfied, or may determine that the obstacle condition is not satisfied (i.e., an obstacle does not exist) when none of the multiple conditions is satisfied.

[0041] Fig. 6A shows a cross-sectional view of the environment around the moving body 100 in a plane perpendicular to the Y direction (i.e., the width direction of the moving body 100) in Fig. 1B. The moving body 100 is traveling on a ground surface 600. In the example of Fig. 6A, the surface of the ground surface 600 is horizontal around the moving body 100. In the example of Fig. 6A, a target object 602 is placed on the ground surface 600. The target object 602 may be, for example, a static target object such as a curb.

[0042] The reference plane 601 is a horizontal plane based on the current position of the moving body 100. As will be described later, the reference plane 601 is used as a reference for determining whether an obstacle is present. In the following description, a horizontal plane passing through the point where the tail wheel 121 of the moving body 100 contacts the ground 600 is used as the reference plane 601. Alternatively, the reference plane 601 may be a horizontal plane passing through another part of the moving body 100 (for example, the detection unit 114 or the center of the moving body 100). In the example of FIG. 6A , the ground 600 is horizontal, so the reference plane 601 coincides with the surface of the ground 600.

[0043] 6A and 6B represent points 603 included in a point cloud generated by the image information processing unit 302 performing 3D point cloud generation based on images acquired by the detection unit 114. In each figure, only one point 603 is assigned a reference symbol. The points 603 included in the point cloud are generated on the surface of a target object included in an image acquired by the detection unit 114. In the example of FIG. 6A , points 603 are generated on the surface of the ground 600 and on the side (the side facing the moving body 100) and top surface of the target object 602.

[0044] As described above with reference to FIG. 4 , the periphery of the moving object 100 is divided into a plurality of grids 401. In the following description, when referring to a specific grid included in the plurality of grids 401, a subscript is added to the end of the reference number. Similarly, a specific point 603 is also referred to by adding a subscript to the end of the reference number. In the example of FIG. 6A , the plurality of grids 401 includes grids 401 a and 401 b. Each grid 401 includes a plurality of points 603.

[0045] The obstacle condition may include a condition regarding the height of the point cloud within the grid 401 based on the moving object 100. In the following description, a condition regarding the height of the point cloud within the grid 401 based on the moving object 100 is referred to as a height condition. When the height condition is satisfied for one grid 401 to be determined, the obstacle determination unit 306 determines that an obstacle exists in this grid 401.

[0046] It is assumed that the moving body 100 can overcome targets that are less than a predetermined thickness, but cannot overcome targets that are equal to or thicker than the predetermined thickness. The thickness of a target refers to the width of the target in a direction perpendicular to the surface of the ground 600. In the following description, the thickness of a target that the moving body 100 can overcome is referred to as the allowable thickness. The allowable thickness may vary depending on the performance and structure of the moving body 100. The allowable thickness may be set by the manufacturer or user of the moving body 100. In the following description, it is assumed that the allowable thickness is 5 cm. If the thickness of the target 602 is less than the allowable thickness, the target 602 does not become an obstacle, and if the thickness of the target 602 is equal to or greater than the allowable thickness, the target 602 becomes an obstacle.

[0047] The height condition may be that the grid 401 includes a point 603 whose height from the reference plane 601 is equal to or greater than a threshold value. This threshold value may be determined based on the allowable thickness of the moving object 100, and may be equal to the allowable thickness, for example. In the following, the threshold value is assumed to be 5 cm, which is equal to the allowable thickness. For example, the grid 401a does not include a point 603 whose height from the reference plane 601 is equal to or greater than the threshold value. Therefore, the height condition is not satisfied for the grid 401a. As a result, the obstacle determination unit 306 does not determine that an obstacle exists in the grid 401a. Whether an obstacle exists in the grid 401a depends on other conditions that may be included in the obstacle condition.

[0048] Assume that the thickness of the target object 602 is equal to or greater than the allowable thickness (e.g., 30 cm). In this case, the height 604 from the reference plane 601 of the point 603a generated on the upper surface of the target object 602 is equal to or greater than the threshold value (e.g., 30 cm). That is, the grid 401a includes a point 603 (e.g., point 603a) whose height from the reference plane 601 is equal to or greater than the threshold value. Therefore, the height condition is satisfied for the grid 401b. As a result, the obstacle determination unit 306 determines that an obstacle (specifically, the target object 602) is present in the grid 401b.

[0049] The height condition is not limited to the above example. The height condition may be that the grid 401 includes a predetermined number or more of points 603 whose height from the reference plane 601 is equal to or greater than a threshold. The point cloud data may include outliers in the height direction due to influences such as measurement noise. Therefore, there is a risk that the height condition will be satisfied even when no obstacle is present in the grid 401 due to points 603 having height outliers. By setting the condition that the grid 401 includes a predetermined number or more of points 603 whose height from the reference plane 601 is equal to or greater than a threshold, the influence of such outliers can be reduced. Alternatively or additionally, the obstacle determination unit 306 may determine the height condition using data of points remaining after filtering the point cloud data to remove outliers regarding height.

[0050] In the above example, the threshold value for the height condition is a constant value (e.g., an allowable thickness). When the ground 600 around the moving body 100 includes a slope, if the threshold value for the height condition is a constant value, an obstacle may be overdetected. Such an example will be described with reference to FIG. 6B . FIG. 6B shows a cross-sectional view of the environment around the moving body 100 in a plane perpendicular to the Y direction (i.e., the width direction of the moving body 100) in FIG. 1B . The moving body 100 is traveling on the ground 600. In the example of FIG. 6B , the ground 600 includes a slope 610 in front of the moving body 100.

[0051] Assume that in grid 401c, the height 611 of the ground surface 600 from the reference plane 601 is equal to or greater than the allowable thickness (e.g., 30 cm). In this case, grid 401c includes point 603 (e.g., point 603b) whose height from the reference plane 601 is equal to or greater than the threshold. Therefore, if the allowable thickness is used as the threshold for the height condition, the height condition is satisfied for grid 401c. As a result, the obstacle determiner 306 determines that an obstacle exists in grid 401c. However, the moving object 100 can reach grid 401c by climbing the slope 610. Therefore, the obstacle determiner 306 may determine the height condition (e.g., its threshold) based on the distance between the moving object 100 and the grid 401. In the following description, the distance between the moving object 100 and the grid 401 is referred to as the remaining distance. The height condition (e.g., its threshold) may vary depending on the remaining distance. The residual distance may be the distance between the moving body 100 and any one point in the grid 401, or may be the distance between the moving body 100 and a point determined from the positions of one or more points 603 in the grid 401. The point determined from the positions of one or more points 603 may be the center of gravity of one or more points 603. Furthermore, the residual distance may be a distance in three-dimensional space, or may be a distance projected onto the reference plane 601 or another plane.

[0052] A method for determining the threshold value for the height condition will be described with reference to FIG. 6C . The obstacle determiner 306 may determine the threshold value from the remaining distance using a graph 620. The graph 620 may be determined in advance by the manufacturer of the mobile body 100 and stored in the memory of the mobile body 100. The horizontal axis of the graph 620 represents the remaining distance, and the vertical axis of the graph 620 represents the threshold value for the height condition. In the example of FIG. 6C , the graph 620 is a straight line passing through the origin. Alternatively, the graph 620 may have another shape. The slope of the graph 620 is set to be equal to or less than the maximum slope that the mobile body 100 can climb.

[0053] The graph 620 has a positive slope. Therefore, the greater the remaining distance, the greater the threshold value for the height condition. The greater the remaining distance, the greater the possibility that the mobile object 100 can reach a higher position from the reference plane 601. Therefore, the threshold value for the height condition is large. On the other hand, if the remaining distance is small, even if a slope exists between the mobile object 100 and the grid 401, the mobile object 100 cannot climb the slope, and therefore the mobile object 100 cannot reach the grid 401.

[0054] The obstacle determining unit 306 may determine the threshold value from the remaining distance using a graph 621. The graph 621 may be determined in advance by the manufacturer of the moving body 100 and stored in the memory of the moving body 100. The horizontal axis of the graph 621 represents the remaining distance, and the vertical axis of the graph 621 represents the threshold value for the height condition. In the example of FIG. 6C , the graph 621 is a straight line whose intersection with the vertical axis is positive. The slope of the graph 621 is set so that it is equal to or less than the maximum slope that the moving body 100 can climb. The value of the intersection with the vertical axis of the graph 621 is set so that it is equal to or less than the allowable thickness of the moving body 100.

[0055] As described above, when the height condition is determined based on the remaining distance, the target object 602 may not be determined to be an obstacle, for example, in a situation such as that shown in FIG. 6A . For example, in a situation such as that shown in FIG. 6A , the threshold value of the height condition determined based on the remaining distance is greater than the thickness of the target object 602. In this case, the height condition is not satisfied for the grid 401b. However, as the moving object 100 approaches the target object 602, the remaining distance also becomes shorter, and as a result, the threshold value of the height condition also becomes smaller. Meanwhile, the height of the point 603 in the grid 401b relative to the reference plane 601 does not change. Therefore, the height condition is satisfied for the grid 401b, and it is determined that the grid 401b contains an obstacle.

[0056] In order to determine at an earlier stage whether the grid 401b contains an obstacle, the obstacle condition may include a condition related to the elevation difference of the point cloud within the grid 401. In the following description, the condition related to the elevation difference of the point cloud within the grid 401 is referred to as the elevation difference condition. When the elevation difference condition is satisfied for one grid 401 to be determined, the obstacle determination unit 306 determines that an obstacle exists in this grid 401.

[0057] The elevation difference of the point cloud within the grid 401 may be the difference between the maximum and minimum heights of one or more points 603 included in the grid 401. The elevation difference condition may be that the elevation difference of the point cloud within the grid 401 is equal to or greater than a threshold. This threshold may be determined based on the allowable thickness of the moving body 100, and may be equal to the allowable thickness, for example. The elevation difference of the point cloud within the grid 401 may represent the thickness of a target included in the grid 401. Therefore, when the elevation difference condition is satisfied for the grid 401, an obstacle is included in the grid 401. In the example of FIG. 6A , the difference in height between point 603a and point 603b is the elevation difference of the point cloud within the grid 401. When this elevation difference is equal to or greater than the allowable thickness, it is determined that an obstacle is included in the grid 401b based on the elevation difference condition.

[0058] The elevation difference condition is not limited to the above example. The point cloud data may contain outliers in the height direction due to the influence of measurement noise, etc. Therefore, there is a risk that the elevation difference condition will be satisfied due to a point 603 having an outlier in height even when no obstacle is present in the grid 401. Therefore, the obstacle determination unit 306 may determine the elevation difference condition using the data of points remaining after filtering to remove outliers regarding the height of the point cloud data.

[0059] <Route Generation> A travel route generated in the mobile body 100 according to this embodiment will be described with reference to Fig. 7. The route generation unit 304 according to this embodiment periodically generates a global path 702 using an occupancy grid map in accordance with a set target position 701, and further periodically generates a local path 703 so as to follow the global path.

[0060] The target position 701 is set based on various instructions. For example, these include instructions from a passenger aboard the vehicle 100 or instructions from a user outside the vehicle 100. Instructions from the passenger are given via the operation panel 131 or the voice input device 133. Instructions given via the operation panel 131 may be given by specifying a specific grid on a grid map displayed on the operation panel 131. In this case, the size of each grid may be set large so that a wider area of ​​the map can be selected. Instructions given via the voice input device 133 may be given using nearby landmarks as landmarks. The landmarks may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like, which are included in the spoken information. Upon receiving an instruction via the voice input device 133, the path generation unit 304 detects the specified landmark from the captured image acquired by the detection unit 114 and sets it as the target position.

[0061] Machine learning models are used for these speech recognition and image recognition. The machine learning model, for example, performs calculations using a deep learning algorithm using a deep neural network (DNN) to recognize place names, landmark names such as buildings, store names, and target names contained in speech information and image information. The speech recognition DNN becomes trained by performing a learning stage process, and by inputting new speech information into the trained DNN, recognition processing (inference stage processing) for the new speech information can be performed. In addition, the image recognition DNN can recognize passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, and motorcycles contained in images.

[0062] Furthermore, instructions from a user outside the vehicle 100 can be sent via the user's own communication device 140 to the vehicle 100 via the communication unit 136, or the user can call the vehicle 100 by raising their hand toward the vehicle 100 as shown in Fig. 7. Instructions from the communication device 140 can be given by operation input or voice input, similar to the instructions from the occupant.

[0063] Once the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupancy grid map. As described above, various algorithms such as RRT, PRM, and A* are known as methods for generating a global path, but any method may be used. Next, the path generation unit 304 generates a local path 703 so as to follow the generated global path 702. There are various methods for planning a local path, such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0064] 8 is a flowchart showing the basic control of the mobile body 100 according to this embodiment. The processing described below is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing the program.

[0065] In S101, the control unit 130 sets a target position of the moving object 100 based on a user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be received in various ways. Next, in S102, the control unit 130 acquires a captured image and depth information. Specifically, the control unit 130 captures an image of the area in front of the moving object 100 using the detection unit 114 and acquires the captured image. The acquired captured image is processed by the image information processing unit 302, and a depth image is created and converted into a three-dimensional point cloud (depth information is generated).

[0066] In S103, the control unit 130 identifies a specific target using the image information processing unit 302. The target can be identified by any method, but for example, the target is recognized by a DNN from an image captured by a monocular camera, and whether the target is a static target or a dynamic target is identified according to a predetermined correspondence relationship between the target and a dynamic target and a static target.

[0067] In S104, the control unit 130 generates an occupancy grid map using the grid map generation unit 303. For example, obstacles are detected from the 3D point cloud image using the method described in FIGS. 6A to 6C, and an occupancy grid map of a predetermined area centered on the mobile object 100 is generated based on the detected obstacles and the position information of the mobile object 100. The grid map generation unit 303 may map the targets identified in S103 onto the occupancy grid map as dynamic obstacles or static obstacles. Details of the processing of S104 will be described using FIG. 9.

[0068] Next, in S105, the control unit 130 generates a driving path for the mobile object 100 using the path generation unit 304. As described above, the path generation unit 304 generates a global path using the occupancy grid map, and generates local paths according to the generated global path. Subsequently, in S106, the control unit 130 determines the speed and angular velocity of the mobile object 100 according to the generated local path, and controls the driving. Thereafter, in S107, the control unit 130 determines whether the mobile object 100 has reached the target position based on position information from the GNSS sensor 134. If the target position has not been reached, the process returns to S102, and the process of generating a path and controlling the driving is repeated while updating the occupancy grid map. On the other hand, if the target position has been reached, the process of this flowchart ends.

[0069] 9 is a flowchart showing the detailed processing steps of the occupancy grid map generation process (S104) according to this embodiment. The processing described below is realized by the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it.

[0070] In S201, the control unit 130 first forgets the accumulated information about obstacles. The forgetting process can be any process, but a new accumulated value is calculated by multiplying the accumulated value of obstacles accumulated for each grid by a forgetting rate (e.g., a value between 0 and 1) previously set for each grid (e.g., in S204). For example, the following can be set: Obstacle Accumulation Value (t) = Accumulation Value (t-1) * Forgetting Rate + New Accumulation Value. For example, if the new accumulated value is set to a predetermined value when an obstacle is newly detected in a grid and set to 0 when no obstacle is detected, the accumulated value remains while the obstacle is detected, and decreases according to the forgetting rate when the obstacle is no longer detected. A maximum accumulated value may be set when obstacles continue to be detected. Performing the forgetting process generates a local map 500, for example, as shown in FIG. 5.

[0071] Next, in S202, the control unit 130 sets grids based on the point cloud information. The obstacle determination unit 306 may apply a filter to remove point clouds whose variance in the height direction is greater than a threshold. In S203, the obstacle determination unit 306 identifies grids that satisfy the above-mentioned obstacle conditions as obstacle grids.

[0072] In S204, the control unit 130 sets a forgetting rate for each grid. For example, the grid map generation unit 303 sets the forgetting rate for forgetting accumulated obstacle information to a different value depending on the type of obstacle. If a moving obstacle is set, the forgetting rate for the moving obstacle may be set to a higher value than the forgetting rate for a static obstacle. Through the processes of S202 to S204, the grid map generation unit 303 generates, for example, the obstacle detection map 510 shown in FIG. 5.

[0073] In S205, the control unit 130 generates an occupancy grid map with updated obstacle information, for example, by adding the map that has been subjected to the forgetting process in S201 and the obstacle detection map generated in S202 to S204. When the control unit 130 completes the process of S205, it ends the process of this flowchart and proceeds to S105. Note that the processes of S201 to S205 are performed periodically, for example, at a cycle of 10 Hz.

[0074] As described above, in the above-described embodiment, point cloud information indicating the object surface in the three-dimensional space around the moving body 100 is generated, and obstacles are detected based on the height of the point cloud from the reference plane 601. This allows for accurate detection of areas containing obstacles. Furthermore, by determining the height condition based on the remaining distance, overdetection of obstacles can be suppressed. Furthermore, by using an elevation difference condition in addition to the height condition, areas containing obstacles can be detected with even greater accuracy.

[0075] Summary of the Embodiments (Item 1) A mobile object control system (130) comprising: a point cloud generation means (302) that generates information on a point cloud (603) indicating an object surface in a three-dimensional space around a mobile object (100) based on an image of the area around the mobile object; an obstacle determination means (306) that determines, for each divided area (401) around the mobile object, whether an obstacle (602) is present in the divided area based on whether a first condition related to the height (604) of the point cloud within the divided area relative to the mobile object is satisfied; and a map generation means (303) that generates an occupancy map indicating the occupancy of the obstacle for each divided area based on the determination result by the obstacle determination means. This item enables accurate detection of areas including obstacles. (Item 2) The mobile object control system according to Item 1, wherein the obstacle determination means changes the first condition based on the distance between the mobile object and the divided area. This item enables suppression of overdetection of areas including obstacles. (Item 3) The mobile object control system according to Item 2, wherein the first condition includes that the height of the point cloud within the divided area relative to the mobile object is greater than a threshold, and the obstacle determination means increases the threshold as the distance between the mobile object and the divided area increases. This item makes it possible to prevent overdetection of areas containing obstacles. (Item 4) The mobile object control system according to any one of Items 1 to 3, wherein the obstacle determination means determines whether an obstacle exists in each divided area around the mobile object based on whether a second condition regarding the difference in elevation of the point cloud within the divided area is satisfied. This item makes it possible to more accurately detect areas containing obstacles. (Item 5) The mobile object control system according to any one of Items 1 to 4, further comprising: movement control means (305) for moving the mobile object so as to avoid the obstacle indicated by the occupancy map. This item makes it possible to move the mobile object while avoiding obstacles.(Item 6) A method for controlling a mobile object control system (130), comprising: a point cloud generation step (S102) for generating information on a point cloud (603) indicating an object surface in a three-dimensional space around a mobile object (100) based on an image of the area around the mobile object; an obstacle determination step (S203) for determining, for each divided area (401) around the mobile object, whether an obstacle (602) is present in the divided area based on whether a first condition is satisfied regarding a height (604) of the point cloud within the divided area relative to the mobile object; and a map generation step (S205) for generating an occupancy map indicating the occupancy of the obstacle for each divided area based on the determination result in the obstacle determination step. This item enables accurate detection of areas including obstacles. (Item 7) A program for causing a computer to function as each means of the mobile object control system described in any one of Items 1 to 5. This item enables the above-mentioned effects to be achieved in the form of a program. (Item 8) A storage medium storing a program for causing a computer to function as each means of the mobile object control system according to any one of items 1 to 5. According to this item, the above-mentioned effects can be realized in the form of a program.

[0076] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

Claims

1. A mobile object control system comprising: a point cloud generation means for generating point cloud information indicating the surface of an object in three-dimensional space around a mobile object based on an image of the area around the mobile object; an obstacle determination means for determining whether an obstacle exists in each divided area around the mobile object based on whether a first condition related to the height of the point cloud within the divided area relative to the mobile object is satisfied; and a map generation means for generating an occupancy map indicating the occupancy of obstacles for each divided area based on the determination result by the obstacle determination means.

2. A mobile object control system according to claim 1, wherein said obstacle determining means changes said first condition based on the distance between said mobile object and said divided area.

3. The mobile body control system described in claim 2, characterized in that the first condition includes that the height of the point cloud within the divided area based on the moving body is greater than a threshold value, and the obstacle determination means increases the threshold value as the distance between the moving body and the divided area increases.

4. A mobile body control system as described in any one of claims 1 to 3, characterized in that the obstacle determination means determines whether an obstacle exists in each divided area around the mobile body based further on whether a second condition regarding the difference in elevation of the point cloud within the divided area is satisfied.

5. A mobile object control system according to any one of claims 1 to 4, further comprising a movement control means for moving the mobile object so as to avoid the obstacles indicated by the occupancy map.

6. A method for controlling a mobile object control system, comprising: a point cloud generation step for generating point cloud information indicating the surface of an object in three-dimensional space around a mobile object based on an image of the area around the mobile object; an obstacle determination step for determining whether an obstacle exists in each divided area around the mobile object based on whether a first condition regarding the height of the point cloud within the divided area relative to the mobile object is satisfied; and a map generation step for generating an occupancy map indicating the occupancy of obstacles for each divided area based on the determination result in the obstacle determination step.

7. A program for causing a computer to function as each means of the mobile object control system according to any one of claims 1 to 5.

8. A storage medium storing a program for causing a computer to function as each means of the mobile object control system according to any one of claims 1 to 5.

Images