Mobile body control system, control method therefor, program, and mobile body

Abstract

This mobile body control system accumulates information on obstacles detected in the past in each divided region made by dividing the peripheral region of the mobile body, and has the accumulated obstacle information forgotten according to a prescribed forgetting rate allocated to each divided region. This mobile body acquires captured images, detects obstacles included in the captured images, and generates an occupancy map showing the occupancy of obstacles for each divided region according to the accumulated obstacle information and the detected obstacles for the current peripheral region of the mobile body.

Description

Mobile object control system, control method thereof, program, and mobile object

[0001] The present invention relates to a mobile object control system, a control method therefor, a program, and a mobile object.

[0002] In recent years, small mobile objects, such as electric vehicles with a seating capacity of one to two people called ultra-compact mobility (also called micromobility) and mobile robots that provide various services to people, have become known. Some of these mobile objects can travel autonomously while periodically generating a route to a destination.

[0003] Patent Literature 1 proposes a technology for autonomous driving in which road surface shape is measured and evaluated in detail to avoid unevenness that could become obstacles, an obstacle map is generated from the evaluation results, and route planning is performed using the map. More specifically, it proposes a method for removing ghosts from moving obstacles while the obstacle map is updated while reflecting external measurement results. Patent Literature 2 proposes using an ultrasonic sensor to generate a dynamically changing surrounding map based on sensor information at two different points in time. Patent Literature 3 proposes a technology for eliminating non-existent moving objects based on reflected pulses received by a radar sensor, leaving only highly reliable real images in road map data.

[0004] JP 2010-102485 A JP 2017-532234 A JP 2019-057197 A

[0005] Micromobility is compact, and it is necessary to minimize the use of hardware resources. Therefore, it is desirable to use a single sensor that detects the surroundings in front of the vehicle as the sensor for detecting the surrounding conditions. In such a configuration, when detecting obstacles using a detection unit such as a camera, whether to retain or discard the accumulated information of the detected obstacles (e.g., the forgetting rate) varies depending on the area around the vehicle. This is because, while changes in obstacles can be detected within the range of the camera's captured image, such changes cannot be detected outside the range.

[0006] On the other hand, when generating a driving route for autonomous driving using information about obstacles detected around the moving body, it is conceivable that, for example, an obstacle is detected ahead of the moving body, and a route is generated that passes near the obstacle that has shifted out of the field of view in order to make a detour. In such a case, by accumulating information about obstacles detected in the past for a certain period of time, it is possible to generate a route that avoids the obstacle that has shifted out of the field of view.

[0007] The present invention has been made in view of the above-mentioned problems, and aims to suitably set the forgetting rate of information about obstacles detected around a moving object for each area.

[0008] According to the present invention, for example, a mobile body is characterized by comprising: a storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the mobile body; a forgetting means for forgetting the obstacle information stored by the storage means in accordance with a predetermined forgetting rate assigned to each divided area; an imaging means for acquiring an image; a detection means for detecting obstacles included in the image; and a map generation means for generating an occupancy map showing the occupancy of obstacles for each divided area in accordance with the obstacle information stored by the storage means and the obstacles detected by the detection means for the current area surrounding the mobile body.

[0009] According to the present invention, the forgetting rate of information about obstacles detected around a moving object can be suitably set for each area.

[0010] FIG. 1 is a block diagram showing an example of the hardware configuration of a mobile body according to the present embodiment; FIG. 2 is a block diagram showing the control configuration of a mobile body according to the present embodiment; FIG. 3 is a block diagram showing the functional configuration of a mobile body according to the present embodiment; FIG. 4 is an occupancy grid map according to the present embodiment; FIG. 5 is a diagram showing a method for generating an occupancy grid map according to the present embodiment; FIG. 6 is a diagram explaining forgetting of obstacle information according to the present embodiment; FIG. 7 is a diagram showing a global route and a local route according to the present embodiment;

[0011] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention as claimed, and not all combinations of features described in the embodiments are necessarily essential to the invention. Two or more of the features described in the embodiments may be arbitrarily combined. Furthermore, the same reference numerals are used for the same or similar components, and redundant explanations will be omitted.

[0012] <Configuration of Mobile Body> The configuration of a mobile body 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1A shows a side view of the mobile body 100 according to this embodiment, and Fig. 1B shows the internal configuration of the mobile body 100. In the figure, arrow X indicates the front-to-rear direction of the mobile body 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and up-down direction of the mobile body 100.

[0013] The moving body 100 is equipped with a battery 113 and is, for example, an ultra-compact mobility vehicle that moves mainly by motor power. An ultra-compact mobility vehicle is a vehicle that is more compact than a typical automobile and has a passenger capacity of approximately one or two people. In this embodiment, a three-wheeled ultra-compact mobility vehicle is described as an example of the moving body 100, but this is not intended to limit the present invention, and the moving body 100 may also be, for example, a four-wheeled vehicle or a saddle-ride vehicle. Furthermore, the vehicle of the present invention is not limited to a vehicle, but may also be a vehicle that carries luggage and runs alongside a person walking, or a vehicle that leads a person. Furthermore, the present invention is not limited to four-wheeled or two-wheeled vehicles, and may also be applicable to walking robots that are capable of autonomous movement.

[0014] The mobile body 100 is an electric autonomous vehicle equipped with a propulsion unit 112 and using a battery 113 as its main power source. The battery 113 is a secondary battery such as a lithium-ion battery, and the mobile body 100 is self-propelled by the propulsion unit 112 using power supplied from the battery 113. The propulsion unit 112 is a tricycle equipped with a pair of front wheels 120 on the left and right and a tail wheel (driven wheel) 121. The propulsion unit 112 may be in another form, such as a four-wheeled vehicle. The mobile body 100 is equipped with a seat 111 for one or two people.

[0015] The traveling unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that uses motors 122a, 122b as drive sources to change the steering angle of the pair of front wheels 120. Changing the steering angle of the pair of front wheels 120 can change the traveling direction of the mobile body 100. The tail wheel 121 does not have its own drive source, but is a driven wheel that operates in response to the drive of the pair of front wheels 120. The tail wheel 121 is also connected to the body of the mobile body 100 via a swivel. The swivel rotates so that the direction of the tail wheel 121 changes separately from the rotation of the tail wheel 121. In this way, the mobile body 100 according to this embodiment employs a differential two-wheel mobility with a tail wheel, but is not limited to this.

[0016] The moving body 100 includes a detection unit 114 that recognizes a plane in front of the moving body 100. The detection unit 114 is an external sensor that monitors the area in front of the moving body 100, and in this embodiment, is an imaging device that captures an image in front of the moving body 100. In this embodiment, the detection unit 114 is described as an example of a stereo camera having an optical system such as two lenses and respective image sensors. However, it is also possible to employ radar or lidar (light detection and ranging) instead of or in addition to the imaging device. Furthermore, in this embodiment, an example in which the detection unit 114 is provided only in front of the moving body 100 is described, but this is not intended to limit the present invention, and the detection unit 114 may be provided behind, or on the left or right sides of the moving body 100.

[0017] The moving body 100 according to this embodiment captures an image of the area ahead of the moving body 100 using the detection unit 114 and detects obstacles from the captured image. Furthermore, the moving body 100 divides the area around the moving body 100 into a grid and controls its travel while generating an occupancy grid map in which obstacle information is stored in each grid (hereinafter also referred to as a grid). The occupancy grid map will be described in detail later.

[0018] <Control Configuration of the Mobile Body> FIG. 2 is a block diagram of a control system of the mobile body 100 according to this embodiment. Here, the configuration necessary for implementing the present invention will be mainly described. Therefore, other configurations may be included in addition to the configurations described below. Furthermore, in this embodiment, the mobile body 100 will be described as including each of the components described below. However, this is not intended to limit the present invention, and the mobile body 100 may be realized as a mobile body control system including multiple devices. For example, some functions of the control unit 130 may be implemented by a communicatively connected server device, or the detection unit 114 and the GNSS sensor 134 may be provided as external devices. The mobile body 100 includes a control unit (ECU) 130. The control unit 130 includes a processor, such as a CPU, a storage device, such as a semiconductor memory, an interface with an external device, and the like. The storage device stores programs executed by the processor, data used by the processor for processing, and the like. Multiple sets of processors, storage devices, and interfaces may be provided for different functions of the mobile body 100 and configured to communicate with each other.

[0019] The control unit 130 acquires the detection results of the detection unit 114, input information from the operation panel 131, audio information input from the audio input device 133, position information from the GNSS sensor 134, and information received via the communication unit 136, and executes corresponding processing. The control unit 130 controls the motors 122a and 122b (travel control of the traveling unit 112), controls the display on the operation panel 131, and notifies and outputs information to the occupants of the mobile object 100 by audio from the speaker 132.

[0020] The voice input device 133 collects the voices of the occupants of the mobile body 100. The control unit 130 can recognize the input voices and execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals to detect the current position of the mobile body 100. The storage device 135 is a storage device that stores images captured by the detection unit 114, obstacle information, previously generated routes, and occupancy grid maps. The storage device 135 may also store programs executed by the processor, data used for processing by the processor, and the like. The storage device 135 may also store various parameters (e.g., learned parameters and hyperparameters of a deep neural network) of machine learning models for voice recognition and image recognition executed by the control unit 130.

[0021] The communication unit 136 communicates with a communication device 140, which is an external device, via wireless communication such as Wi-Fi or fifth-generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited to this and may also be an earphone-type communication terminal, a personal computer, a tablet terminal, a game console, or the like. The communication device 140 connects to a network via wireless communication such as Wi-Fi or fifth-generation mobile communication.

[0022] A user who owns the communication device 140 can issue instructions to the mobile body 100 via the communication device 140. The instructions include, for example, an instruction to call the mobile body 100 to a location desired by the user and join the mobile body 100. Upon receiving the instruction, the mobile body 100 sets a target position based on the position information included in the instruction. In addition to such instructions, the mobile body 100 can also set a target position from an image captured by the detection unit 114 or based on an instruction from a user riding in the mobile body 100 via the operation panel 131. When setting a target position from a captured image, for example, a person raising their hand toward the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target position.

[0023] <Functional Configuration of Mobile Body> Next, the functional configuration of the mobile body 100 according to this embodiment will be described with reference to Fig. 3. The functional configuration described here is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it. Note that the functional configuration described below describes only the functions necessary for explaining the present invention, and does not describe all of the functional configuration actually included in the mobile body 100. In other words, the functional configuration of the mobile body 100 according to the present invention is not limited to the functional configuration described below.

[0024] The user instruction acquisition unit 301 has a function of accepting instructions from the user, and can accept user instructions via the operation panel 131, user instructions from an external device such as the communication device 140 via the communication unit 136, and user spoken instructions via the voice input device 133. As described above, user instructions include an instruction to set a target position (also referred to as a destination) of the mobile body 100 and instructions related to driving control of the mobile body 100.

[0025] The image information processing unit 302 processes the captured images acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image from the stereo images acquired by the detection unit 114 and converts it into a 3D point cloud. The 3D point cloud image data is used to detect obstacles that impede the travel of the mobile object 100. The image information processing unit 302 may also include a machine learning model that processes image information and execute learning stage processing and inference stage processing of the machine learning model. The machine learning model of the image information processing unit 302 can perform processing to recognize three-dimensional objects, etc. included in the image information, for example, by performing calculations of a deep learning algorithm using a deep neural network (DNN).

[0026] The grid map generator 303 creates a grid map of a predetermined size (e.g., each cell is 10 cm x 10 cm in a 20 m x 20 m area) based on the image data of the 3D point cloud. This is done to reduce the size of the grid map, since the amount of data in the 3D point cloud is large and real-time processing is difficult. The grid map includes, for example, a grid map showing the difference between the maximum and minimum heights of the point cloud within the grid (representing whether the cell is a step) and a grid map showing the maximum height of the point cloud within the grid from a reference point (representing the topographical shape of the cell). Furthermore, the grid map generator 303 removes spike noise and white noise from the generated grid map, detects obstacles above a predetermined height, and generates an occupancy grid map indicating whether or not there are any three-dimensional obstacles in each grid.

[0027] The path generation unit 304 generates a travel path for the mobile object 100 to the target position set by the user instruction acquisition unit 301. Specifically, the path generation unit 304 generates a path using an occupancy grid map generated by the grid map generation unit 303 from images captured by the detection unit 114, without requiring obstacle information from a high-precision map. Note that because the detection unit 114 is a stereo camera that captures images of the area ahead of the mobile object 100, it cannot recognize obstacles in other directions. Therefore, it is desirable for the mobile object 100 to store detected obstacle information for a predetermined period of time in order to avoid colliding with obstacles outside the field of view or getting stuck in a dead end. This allows the mobile object 100 to generate a path taking into account both obstacles detected in the past and obstacles detected in real time.

[0028] The path generation unit 304 periodically generates a global path using the occupancy grid map, and also periodically generates local paths that follow the global path. In other words, the target position of the local path is determined by the global path. In this embodiment, the generation period for each path is set to 100 ms, and the generation period for the local path is set to 50 ms, but this is not a limitation of the present invention. Various algorithms are known for generating global paths, such as Rapid-Exploring Random Tree (RRT), Probabilistic Road Map (PRM), and A*. Furthermore, because the vehicle 100 employs a differential two-wheel mobility system with a tail wheel, the path generation unit 304 generates local paths that take into account the tail wheel 121, which is a driven wheel.

[0029] The travel control unit 305 controls the travel of the mobile body 100 according to the local route. Specifically, the travel control unit 305 controls the travel unit 112 according to the local route to control the speed and angular velocity of the mobile body 100. Furthermore, the travel control unit 305 controls the travel in accordance with various operations by the driver. When a deviation occurs in the driving plan of the local route due to an operation by the driver, the travel control unit 305 may again acquire a new local route generated by the route generation unit 304 and control the travel, or may control the speed and angular velocity of the mobile body 100 so as to eliminate the deviation from the local route currently in use.

[0030] <Occupancy Grid Map> Figure 4 shows an occupancy grid map 400 including obstacle information according to this embodiment. Since the mobile body 100 according to this embodiment travels without relying on obstacle information from a high-precision map, all obstacle information is obtained from the recognition results of the detection unit 114. At this time, it is necessary to store obstacle information to avoid collisions with obstacles outside the field of view or getting stuck in dead ends. Therefore, in this embodiment, an occupancy grid map is used as a method of storing obstacle information from the perspectives of reducing the amount of information in the 3D point cloud of stereo images and making it easier to handle in route planning.

[0031] The grid map generator 303 according to this embodiment divides the area surrounding the mobile object 100 into a grid and generates an occupancy grid map that includes information indicating the presence or absence of obstacles for each grid (divided area). While an example of dividing a predetermined area into a grid is described here, it is also possible to divide the area into other shapes rather than a grid, and create an occupancy map indicating the presence or absence of obstacles for each divided area. The occupancy grid map 400 defines a surrounding area around the mobile object 100 as a 40m x 40m or 20m x 20m area, for example, and divides this area into 20cm x 20cm or 10cm x 10cm grids, which are dynamically set in response to the movement of the mobile object 100. In other words, the occupancy grid map 400 is a real-time changing area that shifts as the mobile object 100 moves, always centering the mobile object 100. The size of the areas can be set arbitrarily depending on the hardware resources of the mobile object 100.

[0032] Furthermore, in the occupancy grid map 400, information on the presence or absence of obstacles detected from the image captured by the detection unit 114 is defined for each grid. For example, the presence or absence information is defined as "0" for a drivable area and "1" for an undrivable area (i.e., the presence of an obstacle). In FIG. 4, 401 indicates a grid in which an obstacle exists. An area in which an obstacle exists indicates an area that the mobile object 100 cannot pass through, and is composed of, for example, a solid object of 5 cm or more in size. Therefore, the mobile object 100 generates a route in such a way as to avoid these obstacles 401.

[0033] <Storage of Obstacle Information> Fig. 5 illustrates the storage of obstacle information in an occupancy grid map according to this embodiment. 500 indicates a local map that moves in accordance with the movement of the mobile object 100. The local map 500 is shifted in accordance with the movement of the mobile object 100 in the x-axis direction and the y-axis direction on the grid map. The local map 500 shows how a dotted line area 501 is deleted and a solid line area 502 is added in accordance with, for example, the amount of movement Δx of the mobile object 100 in the x-axis direction. The deleted area is an area opposite to the traveling direction of the mobile object 100, and the added area is an area in the traveling direction. Similarly, areas are deleted and added in the y-axis direction in accordance with the movement of the mobile object 100.

[0034] The local map 500 also stores information about obstacles that have been detected in the past. If an obstacle is present in a grid included in the deletion area, the obstacle information is deleted from the local map 500, but it is desirable to store the information separately from the local map 500 for a certain period of time. This information is useful, for example, when the moving body 100 changes its course and the deletion area is again included in the local map 500, and can improve the accuracy of the moving body 100 in avoiding obstacles. Furthermore, by using the stored information, there is no need to detect obstacles again, and the processing load can be reduced.

[0035] Furthermore, before the local map 500 is added to the obstacle detection map 510 (described later), a forgetting process is performed on the local map 500 according to a forgetting rate set for each grid. When a dynamic obstacle that moves is detected, if obstacle information previously detected and accumulated in grids continues to be retained, a false detection may occur in which the obstacle exists in all grids along the obstacle's movement trajectory. Therefore, to avoid erroneously determining that an obstacle exists in a grid that the obstacle has already passed through, it is necessary to forget the accumulated obstacle information after a certain period of time has elapsed. Forgetting of accumulated obstacle information will be described later using FIG. 6.

[0036] Reference numeral 510 denotes an obstacle detection map showing detection information of obstacles present in the forward vicinity of the mobile body 100 from captured images captured by the detection unit 114 of the mobile body 100. The obstacle detection map 510 shows real-time information and is periodically generated according to the captured images acquired by the detection unit 114. Note that, since moving obstacles such as people and vehicles are also anticipated, it is desirable to update the obstacle detection map 510 periodically generated rather than storing previously detected obstacles in the forward region of the mobile body 100, within the field of view 511 of the detection unit 114. This allows for recognition of moving obstacles and prevents the generation of unnecessary detours. On the other hand, information on previously detected obstacles is stored in the rear region of the mobile body 100 (strictly speaking, outside the field of view of the detection unit 114), as shown in the local map 500. This makes it possible, for example, when an obstacle is detected in the forward region and a detour route is generated, to easily generate a route that avoids collision with the obstacle that has passed through.

[0037] Reference numeral 520 denotes an occupancy grid map generated by adding together the local map 500 and the obstacle detection map 510. In this way, the occupancy grid map 520 is generated as a grid map that combines the local map and obstacle detection information that change in real time with obstacle information that has been detected and accumulated in the past.

[0038] <Obstacle Information Forgetting> FIG. 6 illustrates the forgetting of obstacle information in an occupancy grid map according to this embodiment. Reference numeral 600 denotes an occupancy grid map including generated obstacle information 603, 604 around the mobile object 100. Reference numeral 601 denotes the actual field of view (performance) of the detection unit 114. That is, the field of view 601 denotes the imaging range determined by the performance of the detection unit 114, such as a stereo camera. Reference numeral 602 denotes a field of view range defined as a narrower range than the actual field of view 601 when setting the obstacle information forgetting rate. This makes it possible to reduce erroneous detection of obstacles even when the detection accuracy of obstacles in edge regions of the captured image decreases.

[0039] In this embodiment, a forgetting rate is set for each grid individually. Here, the forgetting rate indicates the extent to which the accumulated obstacle information is retained. For example, in this embodiment, the occupancy grid map is generated periodically, and the forgetting rate indicates the number of cycles for which the obstacle information is stored.

[0040] Specifically, in this embodiment, the forgetting rate for forgetting accumulated obstacle information is set to a different value for each grid included in the field of view range 602 and for grids not included in the field of view range 602. For example, a first forgetting rate is set for grids that overlap the field of view range 602 shown in FIG. 6, and a second forgetting rate that is lower than the first forgetting rate is set for other grids. Here, a grid that overlaps the field of view range 602 is a grid where at least a predetermined area within the grid overlaps the field of view range 602. The size of the predetermined area is arbitrary and can be set, for example, between 1% and 100%.

[0041] As described above, according to this embodiment, a high forgetting rate is set for grids within the field of view of the detection unit 114, and a low forgetting rate is set for other areas. This setting allows obstacle information within the field of view to be quickly forgotten, which can be used to deal with moving obstacles. On the other hand, outside the field of view, obstacle information is stored for a certain period of time, which can prevent collisions with obstacles or getting stuck in dead ends. However, in order to reduce the computational load of path generation, it is desirable to retain obstacle information in a specified peripheral area of ​​the mobile object 100 and forget information in other areas. Therefore, according to this embodiment, a forgetting rate is set even outside the field of view.

[0042] While an example in which different forgetting rates are set for areas within the field of view and areas outside the field of view is described here, the forgetting rates may be set separately for each area. For example, the forgetting rate may be set so that it increases with the distance from the moving body 100 outside the field of view. Furthermore, the type of obstacle may be determined, and the forgetting rate may be changed according to the determined type. Obstacle types include at least dynamic obstacles that move and static obstacles that do not move. Dynamic obstacles include, for example, vehicles and other traffic participants. Static obstacles are obstacles that do not move, and include, for example, fixed objects such as traffic lights and posts, as well as movable tables and desks that are not fixed. Furthermore, the forgetting rate may be set higher for areas containing shadows in the captured image used to detect obstacles than for other areas. This reduces the impact of erroneously detected obstacle information even when the detection accuracy of obstacles in areas containing shadows is low. Furthermore, the forgetting rate may be set higher when there are a large number of obstacles in the grid map or when the proportion of obstacles occupied by obstacles is large. This makes it possible to prevent information about obstacles detected in the past from significantly interfering with route generation. A specific example of the forgetting process will be described later with reference to FIG.

[0043] 7 shows a travel route generated by the mobile body 100 according to this embodiment. The route generation unit 304 according to this embodiment periodically generates a global path 702 using an occupancy grid map in accordance with a set target position 701, and further periodically generates a local path 703 so as to follow the global path.

[0044] The target position 701 is set based on various instructions. For example, these include instructions from a passenger aboard the vehicle 100 or instructions from a user outside the vehicle 100. Instructions from the passenger are given via the operation panel 131 or the voice input device 133. Instructions given via the operation panel 131 may be given by specifying a specific grid on a grid map displayed on the operation panel 131. In this case, the size of each grid may be set large so that a wider area of ​​the map can be selected. Instructions given via the voice input device 133 may be given using nearby landmarks as landmarks. The landmarks may include passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, and the like, which are included in the spoken information. Upon receiving an instruction via the voice input device 133, the path generation unit 304 detects the specified landmark from the captured image acquired by the detection unit 114 and sets it as the target position.

[0045] Machine learning models are used for these speech recognition and image recognition. The machine learning model, for example, performs calculations using a deep learning algorithm using a deep neural network (DNN) to recognize place names, landmark names such as buildings, store names, and target names contained in speech information and image information. The speech recognition DNN becomes trained by performing a learning stage process, and by inputting new speech information into the trained DNN, recognition processing (inference stage processing) for the new speech information can be performed. In addition, the image recognition DNN can recognize passersby, signs, road signs, outdoor facilities such as vending machines, building components such as windows and entrances, roads, vehicles, and motorcycles contained in images.

[0046] Furthermore, instructions from a user outside the vehicle 100 can be sent via the user's own communication device 140 to the vehicle 100 via the communication unit 136, or the user can call the vehicle 100 by raising their hand toward the vehicle 100 as shown in Fig. 7. Instructions from the communication device 140 are given by operation input or voice input, similar to the instructions from the occupant.

[0047] Once the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupancy grid map. As described above, various algorithms such as RRT, PRM, and A* are known as methods for generating a global path, but any method may be used. Next, the path generation unit 304 generates a local path 703 so as to follow the generated global path 702. There are various methods for planning a local path, such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0048] 8 is a flowchart showing the basic control of the mobile body 100 according to this embodiment. The processing described below is realized in the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing the program.

[0049] In S101, the control unit 130 sets a target position for the moving object 100 based on a user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be received in various ways. Next, in S102, the control unit 130 captures an image of the area in front of the moving object 100 using the detection unit 114 and acquires the captured image. The acquired captured image is processed by the image information processing unit 302, creating a depth image and converting it into a three-dimensional point cloud. In S103, the control unit 130 detects obstacles, e.g., three-dimensional objects of 5 cm or more, from the three-dimensional point cloud image. In S104, the control unit 130 generates an occupancy grid map of a predetermined area centered on the moving object 100 based on the detected obstacles and the position information of the moving object 100. The detailed method will be described with reference to FIG. 9 .

[0050] Next, in S105, the control unit 130 generates a driving path for the mobile object 100 using the path generation unit 304. As described above, the path generation unit 304 generates a global path using the occupancy grid map, and generates local paths according to the generated global path. Subsequently, in S106, the control unit 130 determines the speed and angular velocity of the mobile object 100 according to the generated local path, and controls the driving. Thereafter, in S107, the control unit 130 determines whether the mobile object 100 has reached the target position based on position information from the GNSS sensor 134. If the target position has not been reached, the process returns to S102, and the process of generating a path and controlling the driving is repeated while updating the occupancy grid map. On the other hand, if the target position has been reached, the process of this flowchart ends.

[0051] 9 is a flowchart showing a detailed process of the occupancy grid map generation control (S104) according to this embodiment. The process described below is realized by the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into the RAM and executing it.

[0052] First, in S201, the control unit 130 acquires the moving speed of the moving body 100 from the traveling unit 112. Next, in S202, the control unit 130 determines whether the acquired moving speed of the moving body 100 is 0, that is, whether the moving body 100 is in a stopped state. If the moving speed is 0, the process proceeds to S203; if not, the process proceeds to S204.

[0053] In S203, if the moving body 100 is in a stopped state, the control unit 130 sets the forgetting rate k_out outside the field of view to 0 so as not to forget the accumulated obstacle information, and proceeds to S205. On the other hand, in S204, if the moving body 100 is in a moving state (traveling state), the control unit 130 sets the forgetting rate outside the field of view to a default value, and proceeds to S205. Note that the forgetting rate k_in within the field of view is set to a predetermined value regardless of the moving speed of the moving body 100.

[0054] An example of a method for determining the forgetting rate will be described. The method for determining the forgetting rate is based on the following formulas (1) and (2): Accumulation value max = k_in / dt * dynamic_object_forget_time Formula (1) k_out = accumulation max * dt / out_of_fov_forget_time Formula (2) Here, "k_in" indicates the forgetting rate within the field of view. "k_out" indicates the forgetting rate outside the field of view. "dynamic_object_forget_time" indicates the forgetting time within the field of view. "out_of_fov_forget_time" indicates the forgetting time outside the field of view. "dt" indicates the period. By determining the forgetting rate using the above formulas (1) and (2), it is possible to quickly update obstacle information within the field of view and to prioritize past obstacle information outside the field of view.

[0055] In S205, the control unit 130 forgets the accumulated information about obstacles. This forgetting process is performed using the following formula (3): map = map - k_in*fovmap - k_out (1 - fovmap) ... formula (3) Here, "fovmap" indicates a grid map within the field of view. For example, "fovmap" indicates grids within the field of view as "1" and grids outside the field of view as "0". According to formula (3), the forgetting process is performed by changing the forgetting rate inside and outside the field of view as described above. When the forgetting process is performed, for example, a local map 500 shown in FIG. 5 is generated.

[0056] Next, in S206, the control unit 130 acquires the obstacle information detected in S103 (for example, the obstacle detection map 510 shown in FIG. 5). Subsequently, in S207, the control unit 130 generates an occupancy grid map with updated obstacle information by adding the map that has been subjected to the forgetting process in S205 and the obstacle detection map acquired in S206 using the following equation (4): Map = map + k_acc * new_map ... equation (4) Here, "Map" indicates the occupancy grid map with updated obstacle information, "map" indicates the obstacle detection map that has been subjected to the forgetting process in S205, and "new_map" indicates the newly detected obstacle detection map acquired in S206. "k_acc" indicates an accumulation coefficient. This coefficient is set so as not to be greater than the accumulation value max. When the processing of S207 ends, the processing of this flowchart ends, and the processing proceeds to S105. The processes of S201 to S207 are performed periodically, for example, at a cycle of 10 Hz.

[0057] Summary of the embodiment 1. The mobile object control system (for example, 100) of the above embodiment is characterized by comprising: a storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the mobile object; (303) a forgetting means for forgetting the obstacle information stored by the storage means according to a predetermined forgetting rate; (303) an imaging means for acquiring a captured image; (114) a detection means for detecting an obstacle included in the captured image; and (130, 302, 303) a map generation means (303) for generating an occupancy map showing the occupancy of obstacles for each divided area in accordance with the obstacle information stored by the storage means and the obstacles detected by the detection means for the current area surrounding the mobile object.

[0058] According to this embodiment, the forgetting rate of information about obstacles detected around the moving object is suitably set for each area, which allows the present invention to take measures against moving obstacles and to effectively utilize information about obstacles detected in the past to avoid collisions with those obstacles or getting stuck in a dead end.

[0059] 2. In the mobile object control system of the above embodiment, the predetermined forgetting rate differs between within the viewing angle of the imaging means and outside the viewing angle (FIG. 6).

[0060] According to this embodiment, by switching the forgetting rate between inside and outside the viewing angle, it is possible to update the obstacle information while effectively utilizing obstacle information detected in the past.

[0061] 3. In the mobile object control system of the above embodiment, the forgetting rate within the viewing angle is higher than the forgetting rate outside the viewing angle (FIG. 6).

[0062] According to this embodiment, obstacle information within the field of view is updated more quickly to deal with dynamic obstacles that move, and past detection information is retained for a certain period of time outside the field of view, making it possible to avoid collisions with obstacles or getting stuck in dead ends.

[0063] 4. In the mobile object control system of the above embodiment, the area within the viewing angle is set to be narrower than the imaging range of the imaging means, which is determined by the performance of the imaging means (601, FIG. 6).

[0064] According to this embodiment, it is possible to reduce erroneous detection of an obstacle due to the detection accuracy of the edge region in the captured image.

[0065] 5. The mobile object control system of the above embodiment further comprises a discrimination means for discriminating the type of the detected obstacle, and the predetermined forgetting rate is set for each type of the obstacle.

[0066] According to this embodiment, it is possible to effectively use obstacle information detected in the past according to the type of obstacle.

[0067] 6. In the mobile object control system of the above embodiment, the types of obstacles include at least dynamic obstacles that involve movement and static obstacles that do not involve movement.

[0068] According to this embodiment, the forgetting rate can be suitably set depending on whether the obstacle is a moving obstacle or a static obstacle.

[0069] 7. In the mobile object control system of the above embodiment, the forgetting rate set for the dynamic obstacle is higher than the forgetting rate set for the static obstacle.

[0070] According to this embodiment, dynamic obstacles are quickly forgotten because they involve movement, and static obstacles do not move on their own, so by retaining past detection information for a certain period of time, it is possible to avoid collisions with obstacles or getting stuck in dead ends.

[0071] 8. In the mobile object control system of the above embodiment, the dynamic obstacle is a vehicle or other traffic participant.

[0072] According to this embodiment, by identifying traffic participants as dynamic obstacles, it is possible to generate a route for a moving object in an appropriate manner.

[0073] 9. In the mobile object control system of the above embodiment, the static obstacle is an obstacle that does not move by itself.

[0074] According to this embodiment, fixed obstacles are identified as static obstacles, thereby making it possible to generate a route for a moving object in an appropriate manner.

[0075] 10. In the mobile object control system of the above embodiment, the forgetting rate of a region in the captured image that includes a shadow is higher than the forgetting rate of a region in the captured image that does not include a shadow.

[0076] According to this embodiment, areas including shadows that reduce the accuracy of obstacle detection can be quickly forgotten, thereby reducing the impact of false detection.

[0077] 11. In the mobile object control system of the above embodiment, the forgetting rate is set higher as the number of obstacles increases.

[0078] According to this embodiment, it is possible to avoid the route generation being severely hindered by obstacles that have been detected in the past.

[0079] 12. In the mobile object control system of the above embodiment, the forgetting rate is set higher as the number of divided areas occupied by the obstacles increases.

[0080] According to this embodiment, it is possible to avoid the route generation being severely hindered by obstacles that have been detected in the past.

[0081] 13. In the mobile object control system of the above embodiment, the map generation means divides the area around the mobile object into a grid, and generates, as the occupancy map, an occupancy grid map indicating the occupancy of obstacles detected by the detection means for each grid.

[0082] According to this embodiment, a predetermined planar area can be easily divided in the x and y directions, and the predetermined range can be covered without omission.

[0083] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention.

[0084] This application claims priority based on Japanese Patent Application No. 2022-158638, filed on September 30, 2022, the entire contents of which are incorporated herein by reference.

Claims

1. A mobile object control system, A storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the moving object; a forgetting means for forgetting the obstacle information stored by the storage means in accordance with a predetermined forgetting rate assigned to each divided area; An imaging means for acquiring a captured image; A detection means for detecting an obstacle included in the captured image; a map generating means for generating an occupancy map showing the occupancy of obstacles for each divided area in accordance with the obstacle information stored by the storage means and the obstacles detected by the detection means for the current surrounding area of ​​the moving object; A mobile object control system comprising:

2. 2. The mobile object control system according to claim 1, wherein the predetermined forgetting rate is different between within the viewing angle of the imaging means and outside the viewing angle.

3. The mobile object control system according to claim 2 , wherein the forgetting rate within the viewing angle is higher than the forgetting rate outside the viewing angle.

4. 3. The mobile object control system according to claim 2, wherein the area within the viewing angle is set to be narrower than an imaging range of the imaging means, which is determined by the performance of the imaging means.

5. A discrimination means for discriminating a type of the detected obstacle is further provided, The mobile object control system according to claim 1 , wherein the predetermined forgetting rate is set for each type of the obstacle.

6. 6. The mobile object control system according to claim 5, wherein the types of the obstacles include at least a dynamic obstacle that involves movement and a static obstacle that does not involve movement.

7. 7. The mobile object control system according to claim 6, wherein a forgetting rate set for the dynamic obstacle is higher than a forgetting rate set for the static obstacle.

8. 7. The mobile control system according to claim 6, wherein the dynamic obstacle is a vehicle or other traffic participant.

9. 7. The mobile object control system according to claim 6, wherein the static obstacle is an obstacle that does not move by itself.

10. The mobile object control system according to claim 1 , wherein a forgetting rate of an area including a shadow in the captured image is higher than a forgetting rate of an area not including a shadow in the captured image.

11. The mobile object control system according to claim 1 , wherein the forgetting rate is set higher as the number of the obstacles increases.

12. The mobile object control system according to claim 1 , wherein the forgetting rate is set to be higher as the number of divided areas occupied by the obstacles increases.

13. 2. The mobile body control system according to claim 1, wherein the map generation means divides the area around the mobile body into a grid and generates, as the occupancy map, an occupancy grid map indicating the occupancy of obstacles detected by the detection means for each grid.

14. A control method for a mobile object control system, comprising: a storage step of storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the moving object; a forgetting step of forgetting the obstacle information stored in the storing step in accordance with a predetermined forgetting rate assigned to each divided area; An imaging step of acquiring a captured image; a detection step of detecting an obstacle included in the captured image; a map generating step of generating an occupancy map showing the occupancy of obstacles for each divided area in accordance with the obstacle information stored in the storing step and the obstacles detected in the detecting step for the current surrounding area of ​​the moving object; A control method for a mobile object control system comprising:

15. A program for causing a computer to function as each of the means of the mobile object control system according to any one of claims 1 to 13.

16. A mobile object, A storage means for storing information on obstacles detected in the past for each divided area obtained by dividing the area surrounding the moving object; a forgetting means for forgetting the obstacle information stored by the storage means in accordance with a predetermined forgetting rate assigned to each divided area; An imaging means for acquiring a captured image; A detection means for detecting an obstacle included in the captured image; a map generating means for generating an occupancy map showing the occupancy of obstacles for each divided area in accordance with the obstacle information stored by the storage means and the obstacles detected by the detection means for the current surrounding area of ​​the moving object; A moving object comprising: