Mobile body control system, control method thereof, and program

Abstract

To provide a technique for reducing influence of point group noise due to a dynamic obstacle in the periphery of a mobile body.SOLUTION: A mobile body control system includes: acquisition means for acquiring a photographing image photographed in a mobile body and depth information of an environment photographed on the photographing image; identification means for identifying an autonomously movable dynamic obstacle included in the photographing image with the use of the photographing image; and map generation means for generating an occupation map indicating an occupation of an obstacle for each division area obtained by dividing the peripheral area of the mobile body based on the depth information. The occupation map includes the first division area indicating the occupation of the dynamic obstacle to be forgotten in response to a forgetting rate which is higher than that of a static obstacle not autonomously moving. The map generation means sets the division area in a depth direction separated from the mobile body from a position of the identified dynamic obstacle as the first division area indicating the occupation of the dynamic obstacle.SELECTED DRAWING: Figure 6A

Description

[Technical Field]

[0001] The present invention relates to a mobile object control system, a control method therefor, and a program. [Background technology]

[0002] In recent years, there has been an increasing demand for ultra-small mobile vehicles (micromobility) to support people's mobility within small areas. Micromobility can travel in both vehicular and pedestrian areas, so it requires autonomous driving technology for traveling on roadways as well as autonomous mobility technology for traveling in free spaces such as sidewalks. It is expected that in the direction of travel of micromobility, there will be static obstacles that do not move, as well as dynamic obstacles that can move autonomously, such as bicycles.

[0003] Patent Document 1 discloses a technology for an autonomously mobile vehicle to recognize objects in a parking lot using image recognition technology and reflect the positions of the recognized objects on an environmental map. The technology disclosed in Patent Document 1 distinguishes between static objects such as walls and guardrails and dynamic objects such as cars and people according to the type of object recognized by image recognition, and removes the dynamic objects to generate an environmental map that records the static objects as obstacles. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-152234 Summary of the Invention [Problem to be solved by the invention]

[0005] When distance information about obstacles around a moving vehicle is detected using a detection unit such as a stereo camera, point cloud noise of distance information like a shadow may occur around the contour of a moving obstacle (e.g., a person) in the depth direction away from the moving vehicle. Such point cloud noise may be accumulated on the map like an afterimage as an obstacle that exists separately from obstacles such as people. In such cases, the moving vehicle may generate a driving route that unnecessarily avoids an area that is actually free of obstacles and can be traveled through.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide a technique capable of reducing the influence of point cloud noise caused by dynamic obstacles around a moving object. [Means for solving the problem]

[0007] According to the present invention, A mobile object control system, an acquisition means for acquiring a captured image captured by a moving body and depth information of an environment captured in the captured image; an identification means for identifying an autonomously movable moving obstacle included in the captured image using the captured image; a map generating means for generating an occupancy map indicating the occupancy of obstacles for each divided area obtained by dividing the area surrounding the moving object based on the depth information, the occupancy map includes a first segmented region indicating the occupancy of dynamic obstacles that are forgotten according to a higher forgetting rate than static obstacles that do not move autonomously; The mobile body control system is characterized in that the map generation means sets a divided area in the depth direction away from the mobile body from the position of the identified moving obstacle as the first divided area indicating the occupation of the moving obstacle. [Effects of the Invention]

[0008] According to the present invention, it is possible to reduce the influence of point cloud noise caused by obstacles around a moving object. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of a mobile body according to an embodiment of the present invention; [Figure 2] FIG. 1 is a block diagram showing a control configuration of a moving body according to an embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a control unit according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an occupancy grid map according to the present embodiment; [Figure 5] FIG. 10 is a diagram showing a method for generating an occupancy grid map according to the present embodiment. [Figure 6A] FIG. 10 is a diagram illustrating the setting of grids for dynamic and static obstacles according to the present embodiment. [Figure 6B] FIG. 10 is a diagram illustrating another example of setting grids for dynamic obstacles and static obstacles according to the present embodiment. [Figure 7] FIG. 1 shows global routes and local routes according to the present embodiment. [Figure 8] A flowchart showing a processing procedure for controlling the travel of a moving body according to the present embodiment. [Figure 9] A flowchart showing a processing procedure for forgetting accumulated obstacle information according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0011] <Configuration of moving body> The configuration of a moving body 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1(A) shows a side view of the moving body 100 according to this embodiment, and Fig. 1(B) shows the internal configuration of the moving body 100. In the figure, arrow X indicates the front-to-rear direction of the moving body 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-to-right direction) and up-down direction of the moving body 100.

[0012] The mobile body 100 is equipped with a battery 113 and is, for example, an ultra-compact mobility vehicle that moves mainly by motor power. An ultra-compact mobility vehicle is a very small vehicle that is more compact than a typical automobile and has a passenger capacity of approximately one or two people. In this embodiment, a three-wheeled ultra-compact mobility vehicle is described as an example of the mobile body 100. However, this is not intended to limit the present invention, and the mobile body may also be, for example, a four-wheeled vehicle or a saddle-type vehicle. Furthermore, the mobile body of the present invention is not limited to a vehicle, but may also be a vehicle that carries luggage and runs alongside a person walking, or a vehicle that leads a person. Furthermore, the mobile body control system of this embodiment may be a mobile body, a control device such as an ECU included in the mobile body, or an information processing server on a cloud for controlling the mobile body. In other words, some or all of the processing described below according to this embodiment may be executed in the mobile body or in an information processing server on a cloud. Furthermore, the present invention is not limited to four-wheeled or two-wheeled vehicles, but may also be applied to autonomously moving robots and the like.

[0013] The mobile object 100 is an electric autonomous vehicle equipped with a propulsion unit 112 and using a battery 113 as its main power source. The battery 113 is a secondary battery such as a lithium-ion battery, and the mobile object 100 is propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 is a tricycle equipped with a pair of front wheels 120 on the left and right and a tail wheel (driven wheel) 121. The propulsion unit 112 may be in another form, such as a four-wheeled vehicle. The mobile object 100 is equipped with a seat 111 for one or two people.

[0014] The traveling unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that uses motors 122a and 122b as drive sources to change the steering angle of the pair of front wheels 120. By changing the steering angle of the pair of front wheels 120, the traveling direction of the mobile body 100 can be changed. The tail wheel 121 does not have its own drive source, but is a driven wheel that operates in response to the drive of the pair of front wheels 120. The tail wheel 121 is also connected to the body of the mobile body 100 via a swivel unit. The swivel unit rotates so that the direction of the tail wheel 121 changes separately from the rotation of the tail wheel 121. In this way, the mobile body 100 according to this embodiment employs a differential two-wheel mobility with a tail wheel, but is not limited to this.

[0015] The moving body 100 is equipped with a detection unit 114 that recognizes a plane in front of the moving body 100. The detection unit 114 is an external sensor that monitors the front of the moving body 100, and in this embodiment, is an imaging device that captures an image in front of the moving body 100. In this embodiment, the detection unit 114 is described as including, for example, a stereo camera having an optical system such as two lenses and respective image sensors, and a monocular camera. However, it is also possible to adopt a radar or a lidar (Light Detection and Ranging) instead of or in addition to the imaging device. Furthermore, in this embodiment, an example in which the detection unit 114 is provided only in front of the moving body 100 is described, but this is not intended to limit the present invention, and the detection unit 114 may be provided behind, or on the left or right sides of the moving body 100. Furthermore, instead of using a monocular camera, an image captured by one of the stereo cameras may be used.

[0016] The moving body 100 according to this embodiment captures an image of the area ahead of the moving body 100 using the detection unit 114, and detects obstacles from the captured image. Furthermore, the moving body 100 divides the area around the moving body 100 into a grid, and controls its travel while generating an occupancy grid map in which obstacle information is accumulated in each grid (hereinafter also referred to as a grid). The occupancy grid map will be described in detail later.

[0017] <Control structure of moving object> FIG. 2 is a block diagram of a control system of the mobile object 100 according to this embodiment. The following description focuses on the components necessary for implementing the present invention. Therefore, other components may be included in addition to the components described below. Furthermore, in this embodiment, the mobile object 100 is described as including each of the components described below. However, this is not intended to limit the present invention, and the mobile object 100 may be realized as a mobile object control system including multiple devices. For example, some functions of the control unit 130 may be implemented by an information processing server connected to the mobile object 100 in a communicable manner, or the detection unit 114 and the GNSS sensor 134 may be provided as external devices. The mobile object 100 includes a control unit (ECU) 130. The control unit 130 includes a processor, such as a CPU, a storage device, such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used for processing by the processor, and the like. Multiple sets of processors, storage devices, and interfaces may be provided for different functions of the mobile object 100 and configured to communicate with each other.

[0018] The control unit 130 acquires the detection results of the detection unit 114, input information from the operation panel 131, audio information input from the audio input device 133, position information from the GNSS sensor 134, and information received via the communication unit 136, and executes corresponding processing. The control unit 130 controls the motors 122a and 122b (travel control of the traveling unit 112), controls the display on the operation panel 131, and notifies and outputs information to the occupants of the moving object 100 by audio from the speaker 132.

[0019] The voice input device 133 collects the voices of the occupants of the moving body 100. The control unit 130 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals to detect the current position of the moving body 100. The storage device 135 is a storage device that stores images captured by the detection unit 114, obstacle information, previously generated routes, occupancy grid maps, etc. The storage device 135 may also store programs executed by the processor, data used by the processor for processing, etc. The storage device 135 may store various parameters of machine learning models for voice recognition and image recognition executed by the control unit 130 (for example, trained parameters and hyperparameters of a deep neural network, etc.).

[0020] The communication unit 136 communicates with a communication device 140, which is an external device, via wireless communication such as Wi-Fi or fifth generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, or the like. The communication device 140 connects to a network via wireless communication such as Wi-Fi or fifth generation mobile communication.

[0021] A user who owns the communication device 140 can give instructions to the mobile body 100 via the communication device 140. The instructions include, for example, an instruction to call the mobile body 100 to a location desired by the user and join the mobile body 100. Upon receiving the instruction, the mobile body 100 sets a target position based on the position information included in the instruction. In addition to such instructions, the mobile body 100 can also set a target position from an image captured by the detection unit 114 or based on an instruction from a user riding in the mobile body 100 via the operation panel 131. When setting a target position from a captured image, for example, a person raising their hand toward the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target position.

[0022] <Functional configuration of mobile units> Next, the functional configuration of the moving body 100 according to this embodiment will be described with reference to Fig. 3. The functional configuration described here is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it. Note that the functional configuration described below describes only the functions necessary for explaining the present invention, and does not describe all of the functional configuration actually included in the moving body 100. In other words, the functional configuration of the moving body 100 according to the present invention is not limited to the functional configuration described below.

[0023] The user instruction acquisition unit 301 has a function of receiving instructions from the user, and can receive user instructions via the operation panel 131, user instructions from an external device such as the communication device 140 via the communication unit 136, and user spoken instructions via the voice input device 133. As described above, user instructions include instructions to set a target position (also referred to as a destination) of the moving body 100 and instructions related to driving control of the moving body 100.

[0024] The image information processing unit 302 processes the captured images acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image from the stereo images acquired by the detection unit 114 and converts it into a 3D point cloud. The 3D point cloud image data (also referred to as depth information) is used to detect obstacles that impede the travel of the mobile object 100. The image information processing unit 302 may also include a machine learning model that processes image information and execute the learning stage processing and inference stage processing of the machine learning model. For example, the image information processing unit 302 may identify obstacles included in the captured images captured by a monocular camera. For example, the image information processing unit 302 may identify the type of obstacle and, for each obstacle type, may identify whether the obstacle is a dynamic obstacle or a static obstacle, which are predetermined for each obstacle type. Dynamic obstacles are obstacles that can move autonomously, and include, for example, vehicles or other traffic participants such as pedestrians and bicycles. Static obstacles are obstacles that do not move autonomously, and include, for example, objects such as signs and guardrails. The machine learning model of the image information processing unit 302 can perform processing to recognize three-dimensional objects and the like included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN).

[0025] The grid map generation unit 303 creates a grid map of a predetermined size (e.g., each cell is 10 cm x 10 cm in a 20 m x 20 m area) based on the image data (depth information) of the 3D point cloud. This is intended to reduce the size of the grid map, since the amount of data for the 3D point cloud is large and real-time processing is difficult. The grid map includes, for example, a grid map showing the difference between the maximum and minimum heights of the point cloud within the grid (indicating whether the cell is a step) and a grid map showing the maximum height of the point cloud within the grid from a reference point (indicating the topographical shape of the cell). Furthermore, the grid map generation unit 303 removes spike noise and white noise from the generated grid map, detects obstacles with a height above a predetermined level, and generates an occupancy grid map indicating whether or not a three-dimensional object that acts as an obstacle exists for each grid. As will be described later, the grid map generation unit 303 generates an occupancy grid map that distinguishes between grids where dynamic obstacles exist and grids where static obstacles exist.

[0026] The route generation unit 304 generates a travel route for the mobile object 100 to the target position set by the user instruction acquisition unit 301. Specifically, the route generation unit 304 generates a route using an occupancy grid map generated by the grid map generation unit 303 from images captured by the detection unit 114, without requiring obstacle information from a high-precision map. Note that the detection unit 114 is a stereo camera that captures images of the area ahead of the mobile object 100, and therefore cannot recognize obstacles in other directions. Therefore, it is desirable for the mobile object 100 to store detected obstacle information for a predetermined period of time in order to avoid colliding with obstacles outside the field of view or getting stuck in a dead end. This allows the mobile object 100 to generate a route taking into account both obstacles detected in the past and obstacles detected in real time.

[0027] The path generation unit 304 periodically generates a global path using the occupancy grid map, and also periodically generates local paths that follow the global path. In other words, the target position of the local path is determined by the global path. In this embodiment, the generation period for each path is 100 ms for the global path and 50 ms for the local path, but this is not a limitation of the present invention. Various algorithms are known for generating global paths, such as Rapid-Exploring Random Tree (RRT), Probabilistic Road Map (PRM), and A*. The path generation unit 304 can determine the global path using information on grids in the occupancy grid map where static obstacles exist. Meanwhile, when generating local paths, the path generation unit 304 can use information on grids in the occupancy grid map where static obstacles exist and grids in the occupancy grid map where dynamic obstacles exist. In this manner, the global path can be generated using information on stable obstacles, and the local path can be generated as a drivable path that avoids approaching dynamic obstacles.

[0028] The travel control unit 305 controls the travel of the mobile object 100 according to the local route. Specifically, the travel control unit 305 controls the travel unit 112 according to the local route to control the speed and angular velocity of the mobile object 100. Furthermore, the travel control unit 305 controls the travel in accordance with various operations by the driver. When a deviation occurs in the driving plan of the local route due to an operation by the driver, the travel control unit 305 may again acquire a new local route generated by the route generation unit 304 and control the travel, or may control the speed and angular velocity of the mobile object 100 so as to eliminate the deviation from the local route currently in use.

[0029] <Occupancy grid map> FIG. 4 shows an occupancy grid map 400 including obstacle information according to this embodiment. Since the mobile body 100 according to this embodiment travels without relying on obstacle information from a high-precision map, all obstacle information is obtained from the recognition results of the detection unit 114. At this time, it is necessary to store the obstacle information to avoid collisions with obstacles outside the field of view or getting stuck in a dead end. In this embodiment, an occupancy grid map is used as a method of storing obstacle information from the perspectives of reducing the amount of information in the 3D point cloud of stereo images and making it easier to handle in route planning.

[0030] The grid map generator 303 according to this embodiment divides the area surrounding the moving object 100 into a grid and generates an occupancy grid map that includes information indicating the presence or absence of obstacles in each grid-shaped divided area (grid). While an example of dividing a predetermined area into a grid is described here, it is also possible to divide the area into other shapes rather than a grid, and create an occupancy map indicating the presence or absence of obstacles for each divided area. The occupancy grid map 400 defines a surrounding area around the moving object 100 as a 40m x 40m or 20m x 20m area, for example, and divides this area into 20cm x 20cm or 10cm x 10cm grids. Grid information is dynamically set in response to the movement of the moving object 100. In other words, the occupancy grid map 400 is a region that shifts in real time as the moving object 100 moves, always centering the moving object 100. The size of the regions can be set arbitrarily depending on the hardware resources of the moving object 100.

[0031] In addition, the occupancy grid map 400 defines, for each grid, information on the presence or absence of obstacles detected from the image captured by the detection unit 114. For example, the presence or absence information is defined as "0" for a drivable area and "1" for an undrivable area (i.e., the presence of an obstacle). Note that, in the occupancy grid map 400, obstacle type information (e.g., a moving obstacle or a static obstacle) may be set for each grid. Alternatively, the occupancy grid map may include an occupancy grid map for dynamic obstacles in which the presence or absence information of dynamic obstacles is defined for each grid, and an occupancy grid map for static obstacles in which the presence or absence information of static obstacles is defined for each grid. In FIG. 4, 401 indicates a grid in which an obstacle exists. An area in which an obstacle exists indicates an area in which the mobile object 100 cannot pass, and is composed of, for example, a solid object of 5 cm or more. Therefore, the mobile object 100 generates a route to avoid these obstacles 401.

[0032] <Storage of obstacle information> With reference to FIG. 5, the accumulation of obstacle information in an occupancy grid map according to this embodiment will be described. 500 denotes a local map that moves in accordance with the movement of the moving object 100. The local map 500 is shifted in accordance with the movement of the moving object 100 in the x-axis direction and the y-axis direction on the grid map. The local map 500 shows how a dotted line area 501 is deleted and a solid line area 502 is added in accordance with, for example, the amount of movement Δx of the moving object 100 in the x-axis direction. The deleted area is an area opposite to the traveling direction of the moving object 100, and the added area is an area in the traveling direction. Similarly, areas are deleted and added in the y-axis direction as the moving object 100 moves.

[0033] The local map 500 also stores information about obstacles that have been detected in the past. If an obstacle is present in a grid included in the deletion area, the obstacle information is deleted from the local map 500, but it is desirable to store the information separately from the local map 500 for a certain period of time. This information is useful, for example, when the moving object 100 changes its course and the deletion area is again included in the local map 500, and can improve the accuracy of the moving object 100 in avoiding obstacles. Furthermore, by using the stored information, there is no need to detect obstacles again, and the processing load can be reduced.

[0034] Furthermore, before the local map 500 is added to the obstacle detection map 510 (described later), a forgetting process is performed on the local map 500 according to a forgetting rate set for each grid. When a dynamic obstacle is detected, if previously detected obstacle information stored in grids is retained, a false detection that the obstacle exists in all grids along the obstacle's movement trajectory may occur. Therefore, to avoid erroneously determining that an obstacle exists in a grid that the obstacle has already passed, it is necessary to forget the accumulated obstacle information after a certain period of time has elapsed. In this embodiment, a forgetting rate is set individually for each grid. Here, the forgetting rate indicates the extent to which the accumulated obstacle information is retained. For example, in this embodiment, the occupancy grid map is generated periodically, and the forgetting rate indicates the number of cycles for which obstacle information is stored.

[0035] The forgetting rate set for each grid can be set in various ways. For example, in this embodiment, the forgetting rate for forgetting accumulated obstacle information can be set to different values ​​depending on the type of obstacle. For example, the forgetting rate is set to different values ​​for each grid for which the obstacle type is a dynamic obstacle and for each grid for which the obstacle type is a static obstacle. Since dynamic obstacles may move or may move, a high forgetting rate is required to track their movement. In other words, the forgetting rate for dynamic obstacles is set to a higher value than the forgetting rate for static obstacles that do not move. By setting the forgetting rate in this way, obstacle information can be quickly forgotten, which can be used as a countermeasure for dynamic obstacles that move.

[0036] Furthermore, the forgetting rate for forgetting accumulated obstacle information may be set to a different value for each grid included in the viewing angle range 511 and for grids not included in the viewing angle range 511. For example, a forgetting rate higher than the forgetting rate for grids overlapping the viewing angle range 511 may be set. Here, a grid overlapping the viewing angle range 511 is a grid where at least a predetermined area of ​​the grid overlaps the viewing angle range 511. The size of the predetermined area is arbitrary and can be set, for example, between 1 and 100%.

[0037] An obstacle detection map 510 indicates detection information of obstacles present in the vicinity ahead of the moving object 100 from captured images captured by the detection unit 114 of the moving object 100. The obstacle detection map 510 indicates real-time information and is periodically generated according to captured images acquired from the detection unit 114. It is desirable to update the obstacles detected in the field of view 511 of the detection unit 114, which is the area ahead of the moving object 100, using the periodically generated obstacle detection map 510, rather than storing them in a fixed form. This makes it possible to recognize moving obstacles and prevent the creation of unnecessary detours. The obstacle detection map 510 may include obstacle type information (e.g., dynamic obstacle or static obstacle) set for each grid. Alternatively, the obstacle detection map 510 may include an obstacle detection map for dynamic obstacles in which presence / absence information for dynamic obstacles is defined for each grid, and an obstacle detection map for static obstacles in which presence / absence information for static obstacles is defined for each grid. On the other hand, for the rear area of ​​the moving body 100 (strictly speaking, outside the viewing angle of the detection unit 114), information on obstacles detected in the past is accumulated as shown in the local map 500. As a result, for example, when an obstacle is detected in the front area and a detour route is generated, it is possible to easily generate a route that avoids collision with the obstacle that has been passed.

[0038] Reference numeral 520 denotes an occupancy grid map generated by adding together the local map 500 and the obstacle detection map 510. In this way, the occupancy grid map 520 is generated as a grid map that combines the local map and obstacle detection information that change in real time with obstacle information that has been detected and accumulated in the past.

[0039] <Setting grids for dynamic and static obstacles> Referring to FIG. 6A, the setting of grids for dynamic and static obstacles will be described. 600 shows a top view of a plane on which the mobile object 100 travels. The mobile object 100 is located at the origin of a coordinate system, and object 601 is, for example, a person, and object 602 is, for example, a traffic cone placed on a road. Objects 601 and 602 are included in a captured image captured by a monocular camera of the detection unit 114. The image information processing unit 302 performs object recognition processing using the captured image to estimate the type of object. Since the image information processing unit 302 estimates that object 601 is a person and object 602 is a traffic cone, the objects 601 and 602 are identified as a dynamic obstacle and a static obstacle, respectively. The image information processing unit 302 identifies the distance d from the mobile object 100 to object 601 and the boundary area (boundary) of object 601. By identifying the boundary region of the object 601, the image information processing unit 302 can determine, for example, the rotation angle θ1 from the optical axis direction of a line passing through a first end (one end of the boundary region) of the object 601. Similarly, the image information processing unit 302 can determine, for example, the rotation angle θ2 from the optical axis direction of a line passing through a second end (the other end of the boundary region) of the object 601. Note that the rotation angles θ1 and θ2 are rotation angles in a polar coordinate system.

[0040] Next, the grid map generation unit 303 searches for moving obstacles based on a grid map 610 obtained based on depth information from the stereo images and information obtained by the image information processing unit 302 (from images captured by the monocular camera). In the grid map 610, presence / absence information of "1" is set for grids where obstacles exist (i.e., grids indicating the occupation of obstacles) in the depth information obtained from the stereo images. The grids indicating the presence of obstacles are grid 611 (five grids), grid 612 (one grid), and grid 613 (one grid). The grid map generation unit 303 superimposes an area 615 defined by rotation angles θ1 and θ2 (boundaries of the moving obstacle) on the grid map 610 and identifies obstacle grids that overlap with the area 615 by a predetermined area or more. In other words, the grid map generation unit 303 associates the area surrounded by the moving obstacles obtained in the captured images with the obstacle grids in the grid map 610. Through this process, the grid map generation unit 303 can set the five grids indicated by the grid 611 as grids where a moving obstacle exists. The grid map generation unit 303 further integrates, among the grids where an obstacle exists, grids adjacent to the grid 611 as grids where a moving obstacle exists. In this way, the grid 611 and the grid 612 can be set as grids where a single moving obstacle exists. Note that the grid map generation unit 303 may also integrate, among the grids where an obstacle exists, grids near the grid 611 as grids where a moving obstacle exists. In this way, a hidden area caused by a moving obstacle identified in the captured image can be treated as a region of the moving obstacle. If a high forgetting rate is set for the grid of the moving obstacle, the hidden area caused by the moving obstacle will be forgotten more quickly than a static obstacle, and the influence of the point cloud noise caused by the obstacle will also be forgotten more quickly.

[0041] However, the grid map generating unit 303 does not set, among the grids in which obstacles exist, a grid (e.g., 613) that is farther than the distance threshold from the grid in which the moving obstacle exists (i.e., 611 and 612) as a grid for the same moving obstacle. By doing so, it is possible to integrate moving obstacles within an appropriate range and to prevent unnecessary integration of grids for obstacles.

[0042] Furthermore, the grid map generator 303 may set a grid of an obstacle that was not set as a dynamic obstacle as a grid where a static obstacle exists. For example, grid 613 is a grid where an obstacle exists, but this area does not overlap with area 615 and is not near a grid where a dynamic obstacle exists, so it is set as a grid where a static obstacle exists.

[0043] Through this processing, it is possible to generate an obstacle detection map 620 in which dynamic obstacles and static obstacles are distinguished. As described above, grid 613 is set as a grid where static obstacles exist, and grid 621 is set as a grid where dynamic obstacles exist. In the example described above with reference to FIG. 6A, an example was described in which one obstacle detection map is generated in which grids where static obstacles exist and grids where dynamic obstacles exist are set. However, the grid map generation unit 303 may also generate a first obstacle detection map in which grids where static obstacles exist are set, and a second obstacle detection map in which grids where dynamic obstacles exist are set.

[0044] Another example of setting grids for dynamic and static obstacles according to this embodiment will be described with reference to Fig. 6B. 600 is the same as Fig. 6A, and the image information processing unit 302 calculates the distance to the object 601 and the rotation angles θ1 and θ2 based on the captured image by the above-described processing.

[0045] Next, the grid map generation unit 303 sets grids for moving obstacles based on information obtained by the image information processing unit 302 (from images captured by the monocular camera). In the grid map 610, presence / absence information of "1" is set for grids where obstacles exist (i.e., grids indicating the occupation of obstacles) in the depth information obtained from the stereo image. The grids indicating the presence of obstacles are, as in FIG. 6A, five grids 611, one grid 612, and one grid 613; however, in FIG. 6B, the five grids 611 and one grid 612 are not shown to avoid complicating the illustration. The grid map generation unit 303 superimposes a region 615 defined by rotation angles θ1 and θ2 (boundaries of moving obstacles) on the grid map 610. Then, a region 631 where the distance to the object 601 is greater than distance d and is defined by rotation angles θ1 and θ2 (boundaries of moving obstacles) is set as a region where moving obstacles exist. By this process, the grid map generation unit 303 can set the grid on which the area 631 overlaps as the grid on which a moving obstacle exists. For example, in the example shown in FIG. 6B, the grid on which the area 631 overlaps even slightly is set as the grid on which a moving obstacle exists. In this way, the hidden area caused by the moving obstacle identified in the captured image can be treated as a large area of ​​the moving obstacle. If a high forgetting rate is set for the grid of the moving obstacle, the hidden area caused by the moving obstacle will be forgotten more quickly than the static obstacle, and the influence of the point cloud noise caused by the obstacle will also be forgotten more quickly.

[0046] By this processing, it is possible to generate an obstacle detection map 640 in which dynamic obstacles and static obstacles are distinguished. As described above, grid 613 is set as a grid where static obstacles exist, and grid 641 is set as a grid where dynamic obstacles exist. Note that in the example shown in FIG. 6B, an example of generating one obstacle detection map in which grids where static obstacles exist and grids where dynamic obstacles exist are set has been described. However, the grid map generation unit 303 may also generate a first obstacle detection map in which grids where static obstacles exist are set, and a second obstacle detection map in which grids where dynamic obstacles exist are set.

[0047] <Route generation> A travel route generated by the moving body 100 according to this embodiment will be described with reference to Fig. 7. The route generation unit 304 according to this embodiment periodically generates a global path 702 using an occupancy grid map in accordance with a set target position 701, and further periodically generates a local path 703 so as to follow the global path.

[0048] The target position 701 is set based on various instructions. For example, these include instructions from a passenger aboard the vehicle 100 or instructions from a user outside the vehicle 100. Instructions from the passenger are given via the operation panel 131 or the voice input device 133. Instructions given via the operation panel 131 may be given by specifying a specific grid on a grid map displayed on the operation panel 131. In this case, the size of each grid may be set large so that a wider area of ​​the map can be selected. Instructions given via the voice input device 133 may be given using surrounding landmarks as landmarks. The landmarks may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like, included in the spoken information. Upon receiving an instruction via the voice input device 133, the path generation unit 304 detects the specified landmark from the captured image acquired by the detection unit 114 and sets it as the target position.

[0049] Machine learning models are used for these speech and image recognition tasks. Machine learning models, for example, use deep learning algorithms using deep neural networks (DNNs) to recognize place names, landmark names such as buildings, store names, and landmark names contained in speech and image information. Speech recognition DNNs become trained through training phase processing, and by inputting new speech information into the trained DNN, recognition processing for the new speech information (inference phase processing) can be performed. Image recognition DNNs can also recognize passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, and motorcycles contained in images.

[0050] Furthermore, instructions from a user outside the vehicle 100 can be sent via the user's own communication device 140 to the vehicle 100 via the communication unit 136, or the user can call the vehicle 100 by raising their hand toward the vehicle 100 as shown in Fig. 7. Instructions from the communication device 140 are given by operation input or voice input, similar to the instructions from the passengers.

[0051] Once the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupancy grid map. As described above, various algorithms such as RRT, PRM, and A* are known as methods for generating a global path, but any method may be used. Next, the path generation unit 304 generates a local path 703 so as to follow the generated global path 702. There are various methods for planning a local path, such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0052] <Basic control of moving objects> 8 is a flowchart showing the basic control of the moving body 100 according to this embodiment. The processing described below is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it.

[0053] In S101, the control unit 130 sets a target position of the moving object 100 based on a user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be received in various ways. Subsequently, in S102, the control unit 130 acquires a captured image and depth information. Specifically, the control unit 130 captures an image of the area in front of the moving object 100 using the detection unit 114 and acquires the captured image. The acquired captured image is processed by the image information processing unit 302, and a depth image is created and converted into a three-dimensional point cloud (depth information is generated).

[0054] In S103, the control unit 130 distinguishes between moving obstacles and static obstacles from the image captured by the monocular camera using the image information processing unit 302. For a moving obstacle, the image information processing unit 302 can distinguish the distance d to the moving obstacle and the rotation angles θ1 and θ2 described above.

[0055] In S104, the control unit 130 generates an occupancy grid map using the grid map generation unit 303. For example, it detects obstacles that are solid objects, for example, 5 cm or larger, from the 3D point cloud image, and generates a grid map of a predetermined area centered on the moving object 100 based on the detected obstacles and the position information of the moving object 100. Furthermore, the grid map generation unit 303 generates an obstacle detection map that distinguishes between dynamic obstacles and static obstacles using the method described with reference to FIG. 6A or 6B, and adds this to the local map 500 to generate an occupancy grid map. A detailed method will be described using FIG. 9.

[0056] Next, in S105, the control unit 130 generates a driving route for the mobile object 100 using the route generation unit 304. As described above, the route generation unit 304 generates a global route using the occupancy grid map, and generates local routes according to the generated global route. Subsequently, in S106, the control unit 130 determines the speed and angular velocity of the mobile object 100 according to the generated local route, and controls the driving. Thereafter, in S107, the control unit 130 determines whether the mobile object 100 has reached the target position based on position information from the GNSS sensor 134. If the target position has not been reached, the process returns to S102, and the process of generating a route while updating the occupancy grid map and controlling the driving is repeated. On the other hand, if the target position has been reached, the process of this flowchart ends.

[0057] <Method for generating occupancy grid maps> 9 is a flowchart showing the detailed processing steps of the occupancy grid map generation process (S104) according to this embodiment. The processing described below is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it.

[0058] In S201, the control unit 130 first forgets the accumulated information of obstacles. The forgetting process can be any process, but a new accumulated value is calculated by multiplying the accumulated value of obstacles accumulated for each grid by a forgetting rate (for example, a numerical value between 0 and 1) previously set for each grid (for example, in S206). When the forgetting process is executed, for example, a local map 500 shown in FIG. 5 is generated.

[0059] Next, in S202, the control unit 130 sets a grid based on the depth information. For example, the grid map generation unit 303 detects obstacles, e.g., three-dimensional objects of 5 cm or more, from the 3D point cloud image (depth information), and generates a grid map of a predetermined area centered on the moving object 100 according to the detected obstacles and the position information of the moving object 100 (for example, grid map 610 shown in FIG. 6A).

[0060] In S203, the control unit 130 identifies a grid of the moving obstacle using the grid map generation unit 303. For example, the grid map generation unit 303 identifies a grid of the moving obstacle (e.g., grid 611) using information about the moving obstacle identified in S103 and the grid map generated in S202, for example, by the method described above in FIG. 6A. In S204, the control unit 130 merges grids of adjacent or nearby obstacles into the grid of the moving obstacle, for example, by the method described above in FIG. 6A. Of course, the method of identifying the grid of the moving obstacle may be the method described above in FIG. 6B.

[0061] In S205, the control unit 130 causes the grid map generation unit 303 to identify grids of static obstacles. For example, the grid map generation unit 303 identifies, among the obstacle grids, grids to which no dynamic obstacles are set as grids of static obstacles. In S206, the control unit 130 sets a forgetting rate for each grid. For example, the grid map generation unit 303 sets a forgetting rate for forgetting accumulated obstacle information to a different value depending on the type of obstacle. At this time, the forgetting rate for dynamic obstacles may be set to a higher value than the forgetting rate for static obstacles. Through the processes of S202 to S206, the grid map generation unit 303 generates, for example, the obstacle detection map 620 shown in FIG. 6A.

[0062] In S207, the control unit 130 generates an occupancy grid map with updated obstacle information, for example, by adding the map that has been subjected to the forgetting process in S201 and the obstacle detection map generated in S202 to S206. When the control unit 130 completes the process of S207, it ends the process of this flowchart and proceeds to S105. Note that the processes of S201 to S207 are performed periodically, for example, at a cycle of 10 Hz.

[0063] As described above, in the above-described embodiment, the image information processing unit 302 uses the captured image to identify dynamic obstacles included in the captured image. The grid map generating unit 303 generates an occupancy map indicating the occupancy of obstacles for each divided area (grid) obtained by dividing the area surrounding the moving object based on the depth information. The occupancy map includes grids in which static obstacles are set and grids in which dynamic obstacles are set. To identify the grid in which the dynamic obstacle is set, the grid map generating unit 303 sets a grid in the depth direction away from the moving object from the position of the dynamic obstacle identified in the captured image as the dynamic obstacle grid. The dynamic obstacle grid is then forgotten at a higher forgetting rate than the static obstacle grid. In this way, the hidden area due to the dynamic obstacle identified in the captured image can be treated as the dynamic obstacle area, and the influence of point cloud noise caused by the obstacle occurring in this area can also be quickly forgotten. Therefore, the influence of point cloud noise caused by the dynamic obstacle around the moving object can be reduced.

[0064] <Summary of the embodiment> (Item 1) A mobile object control system, an acquisition means for acquiring a captured image captured by a moving body and depth information of an environment captured in the captured image; an identification means for identifying an autonomously movable moving obstacle included in the captured image using the captured image; a map generating means for generating an occupancy map indicating the occupancy of obstacles for each divided area obtained by dividing the area surrounding the moving object based on the depth information, the occupancy map includes a first segmented region indicating the occupancy of dynamic obstacles that are forgotten according to a higher forgetting rate than static obstacles that do not move autonomously; The mobile object control system is characterized in that the map generation means sets a divided area in a depth direction away from the mobile object from the position of the identified moving obstacle as the first divided area indicating the occupation of the moving obstacle.

[0065] According to this embodiment, the hidden area caused by the moving obstacle identified in the captured image can be treated as the area of ​​the moving obstacle, and the influence of the point cloud noise caused by the obstacle occurring in this area can be quickly forgotten, thereby reducing the influence of the point cloud noise caused by the moving obstacle around the moving object.

[0066] (Item 2) The mobile object control system described in item 1, characterized in that the map generation means generates a first occupancy map including the first divided area indicating the occupancy of a dynamic obstacle, and a second occupancy map including a second divided area indicating the occupancy of a static obstacle.

[0067] According to this embodiment, separate occupancy maps can be provided that represent occupancy by dynamic obstacles and occupancy maps that represent occupancy by static obstacles.

[0068] (Item 3) 2. The mobile object control system according to claim 1, wherein the map generation means generates a third occupancy map including the first divided area indicating the occupancy of a dynamic obstacle and a second divided area indicating the occupancy of a static obstacle.

[0069] According to this embodiment, the occupancy due to dynamic obstacles and the occupancy due to static obstacles can be combined into a single occupancy map.

[0070] (Item 4) The mobile object control system described in item 1, characterized in that the map generation means identifies which of the divided areas indicating the occupancy of the obstacle, identified based on the depth information, is the first divided area indicating the occupancy of the moving obstacle.

[0071] According to this embodiment, the area of ​​the moving obstacle obtained in the captured image can be associated with the area of ​​the obstacle in the grid map specified by the depth information.

[0072] (Item 5) The mobile body control system described in item 4, characterized in that the map generation means sets, among the divided areas indicating the occupancy of an obstacle identified based on the depth information, a divided area in the depth direction away from the mobile body from the position of the identified moving obstacle as the first divided area indicating the occupancy of a moving obstacle.

[0073] According to this embodiment, the hidden area caused by the moving obstacle identified in the captured image can be treated as the area of ​​the moving obstacle.

[0074] (Item 6) 6. The mobile object control system according to item 5, wherein the map generating means integrates, among the divided areas indicating the occupancy of obstacles identified based on the depth information, divided areas adjacent to the first divided area indicating the occupancy of a moving obstacle as the first divided area for the same moving obstacle.

[0075] According to this embodiment, the areas of adjacent obstacles can be integrated into one area of ​​a moving obstacle.

[0076] (Item 7) 7. The mobile object control system according to item 6, wherein the map generating means does not set, among the divided areas indicating the occupancy of an obstacle identified based on the depth information, any divided area that is located farther than a threshold from the first divided area indicating the occupancy of a moving obstacle as the first divided area for the same moving obstacle.

[0077] According to this embodiment, dynamic obstacles can be integrated within an appropriate range, and unnecessary integration of obstacle grids can be prevented.

[0078] (Item 8) The mobile body control system described in item 1, characterized in that the divided area in the depth direction away from the moving body from the position of the identified moving obstacle includes an area defined by the boundary of the identified moving obstacle that is deeper than the depth from the moving body to the identified moving obstacle.

[0079] According to this embodiment, the hidden area caused by the moving obstacle identified in the captured image can be treated as a large moving obstacle area.

[0080] (Item 9) a storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the moving object; 2. The mobile object control system according to item 1, further comprising: a forgetting unit that causes the accumulated information about the obstacle to be forgotten in accordance with a forgetting rate assigned to each divided area.

[0081] According to this embodiment, it becomes possible to respond to changes in obstacles.

[0082] (Item 10) 2. The mobile object control system according to item 1, wherein the dynamic obstacle is a vehicle or another traffic participant.

[0083] This embodiment makes it possible to avoid contact or proximity with vehicles and other traffic participants.

[0084] (Item 11) The mobile object control system described in item 1, characterized in that the acquisition means acquires the captured image captured by a monocular imaging device and the depth information obtained from stereo images captured by multiple imaging devices.

[0085] According to this embodiment, a grid map technology that takes dynamic obstacles into consideration can be realized with an inexpensive configuration using an imaging device and image processing.

[0086] (Item 12) 2. The mobile object control system according to claim 1, wherein the identification means further identifies the distance from the mobile object to the moving obstacle and the boundary area of ​​the moving obstacle based on the captured image.

[0087] According to this embodiment, it is possible to obtain three-dimensional information for identifying the area occupied by a moving obstacle from the captured image.

[0088] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0089] 100...mobile body, 111...seat, 112...traveling unit, 113...battery, 114...detection unit, 120...front wheel, 121...tail wheel, 122a, 122b...motor, 123...steering mechanism, 130...control unit, 131...operation panel, 132...speaker, 133...voice input device, 134...GNSS sensor, 135...storage device, 136...communication unit, 140...communication device

Claims

1. A mobile object control system, an acquisition means for acquiring a captured image captured by a moving body and depth information of an environment captured in the captured image; an identification means for identifying an autonomously movable moving obstacle included in the captured image using the captured image; a map generating means for generating an occupancy map indicating the occupancy of obstacles for each divided area obtained by dividing the area surrounding the moving object based on the depth information, the occupancy map includes a first segmented region indicating the occupancy of dynamic obstacles that are forgotten according to a higher forgetting rate than static obstacles that do not move autonomously; The mobile object control system is characterized in that the map generation means sets a divided area in a depth direction away from the mobile object from the position of the identified moving obstacle as the first divided area indicating the occupation of the moving obstacle.

2. 2. The mobile object control system according to claim 1, wherein the map generating means generates a first occupancy map including the first divided area indicating the occupancy of a dynamic obstacle, and a second occupancy map including a second divided area indicating the occupancy of a static obstacle.

3. 2. The mobile object control system according to claim 1, wherein the map generating means generates a third occupancy map including the first divided area indicating the occupancy of a dynamic obstacle and a second divided area indicating the occupancy of a static obstacle.

4. 2. The mobile object control system according to claim 1, wherein the map generating means identifies which of the divided areas indicating the occupancy of an obstacle, identified based on the depth information, is the first divided area indicating the occupancy of the moving obstacle.

5. The mobile body control system according to claim 4, characterized in that the map generation means sets, among the divided areas indicating the occupancy of an obstacle identified based on the depth information, a divided area in a depth direction away from the mobile body from the position of the identified moving obstacle as the first divided area indicating the occupancy of the moving obstacle.

6. 6. The mobile body control system according to claim 5, wherein the map generating means integrates, among the divided areas indicating the occupancy of obstacles identified based on the depth information, divided areas adjacent to the first divided area indicating the occupancy of a moving obstacle, as the first divided area for the same moving obstacle.

7. The mobile body control system according to claim 6, characterized in that the map generating means does not set, among the divided areas indicating the occupancy of an obstacle identified based on the depth information, any divided area that is located farther than a threshold from the first divided area indicating the occupancy of a moving obstacle as the first divided area for the same moving obstacle.

8. The mobile body control system of claim 1, characterized in that the divided area in the depth direction away from the moving body from the position of the identified moving obstacle includes an area defined by the boundary of the identified moving obstacle that is deeper than the depth from the moving body to the identified moving obstacle.

9. a storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the moving object; 2. The mobile object control system according to claim 1, further comprising: a forgetting unit that causes the accumulated information about the obstacle to be forgotten in accordance with a forgetting rate assigned to each divided area.

10. 2. The mobile object control system according to claim 1, wherein the dynamic obstacle is a vehicle or other traffic participant.

11. 2. The mobile object control system according to claim 1, wherein the acquisition means acquires the captured image captured by a monocular imaging device and the depth information obtained from stereo images captured by a plurality of imaging devices.

12. 2. The mobile object control system according to claim 1, wherein the identification means further identifies a distance from the mobile object to the moving obstacle and a boundary area of ​​the moving obstacle based on the captured image.

13. A control method for a mobile object control system, comprising: an acquisition step of acquiring a captured image captured by a moving body and depth information of an environment captured in the captured image; an identification step of identifying an autonomously movable moving obstacle included in the captured image using the captured image; a map generation step of generating an occupancy map indicating the occupancy of obstacles for each divided area obtained by dividing the area surrounding the moving object based on the depth information, the occupancy map includes a first segmented region indicating the occupancy of dynamic obstacles that are forgotten according to a higher forgetting rate than static obstacles that do not move autonomously; A control method characterized in that, in the map generation process, a divided area in a depth direction away from the moving body from the position of the identified moving obstacle is set as the first divided area indicating the occupation of the moving obstacle.

14. A program for causing a computer to function as each of the means of the mobile object control system according to any one of claims 1 to 12.

Images