Moving body control system, control method therefor, program, and moving body

Abstract

This moving body control system stores information on obstacles detected in the past for each divided region obtained by dividing the peripheral region of a moving body, and forgets the stored information on obstacles in accordance with a predetermined forgetting rate allocated for each divided region. Furthermore, the mobile body acquires a captured image, detects an obstacle included in the captured image, and generates an occupancy map indicating the occupancy of the obstacle for each divided region in accordance with the stored obstacle information and the detected obstacle for the current peripheral region of the mobile body.

Description

Technical Field

[0001] The present invention relates to a mobile body control system and a control method thereof, a program, and a mobile body. Background Art

[0002] In recent years, small mobile bodies such as electric vehicles with a rated passenger capacity of 1 to 2 people, mobile robots that provide various services to people, etc., which are called ultra-small mobile bodies (also called micro mobile vehicles), are known. Among such mobile bodies, there are mobile bodies that travel autonomously while periodically generating a travel route to a destination.

[0003] Patent document 1 proposes the following technology: in order to avoid bumps and depressions that become obstacles, the shape of the road surface is measured and evaluated in detail, an obstacle map is generated based on the evaluation results, and the obstacle map is used for path planning for autonomous driving. More specifically, a method is proposed for removing ghost images in moving obstacles while the obstacle map is updated while successively reflecting the external measurement results. Patent document 2 proposes the use of ultrasonic sensors to generate a dynamically changing surrounding map based on sensor information at two different time points. Patent document 3 proposes a technology that eliminates non-existent moving objects based on the reflected pulses received by the radar sensor, leaving only real images with high reliability in the road map data.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2010-102485

[0007] Patent Document 2: Japanese Patent Application No. 2017-532234

[0008] Patent Document 3: Japanese Patent Application Publication No. 2019-057197 Summary of the invention

[0009] Problems to be solved by the invention

[0010] The micro mobile vehicle is small, and the use of hardware resources needs to be suppressed as much as possible. Therefore, as a sensor for detecting the surrounding conditions, it is preferable to use a sensor that detects the front surrounding of the mobile body. In such a structure, when a detection unit such as a camera is used to detect obstacles, the storage information (for example, the forgetting rate) of the detected obstacles is maintained or discarded differently according to the area around the mobile body. This is because changes in obstacles can be detected within the range of the camera's captured image, but such changes cannot be detected outside the range.

[0011] On the other hand, it is also conceivable that, in the case of generating a driving path for autonomous driving using information about obstacles detected around a moving body, for example, an obstacle is detected in front of the moving body, and thus a path is generated to bypass the obstacle that is offset outside the field of view. In such a case, by storing information about obstacles detected in the past for a certain period of time, a path that avoids obstacles that are offset outside the field of view can be generated.

[0012] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to appropriately set a forgetting rate of information on obstacles detected around a moving object for each area.

[0013] Means used to solve problems

[0014] According to the present invention, for example, a mobile body is characterized in that the mobile body comprises: a storage mechanism, which stores information of obstacles detected in the past according to divided areas formed by dividing the surrounding area of ​​the mobile body; a forgetting mechanism, which forgets the information of obstacles stored by the storage mechanism according to a predetermined forgetting rate assigned to the divided areas; a shooting mechanism, which acquires a shot image; a detection mechanism, which detects obstacles contained in the shot image; and a map generating mechanism, which generates an occupancy map showing the occupancy of obstacles by divided areas according to the information of obstacles stored by the storage mechanism and the obstacles detected by the detection mechanism for the current surrounding area of ​​the mobile body.

[0015] Effects of the Invention

[0016] According to the present invention, the forgetting rate of information on obstacles detected around a moving object can be appropriately set for each area. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1A This is a block diagram showing a configuration example of hardware of a moving object according to the present embodiment.

[0018] Figure 1B This is a block diagram showing a configuration example of hardware of a moving object according to the present embodiment.

[0019] Figure 2 This is a block diagram showing a control structure of a moving object according to the present embodiment.

[0020] Figure 3 It is a block diagram showing the functional configuration of a moving object according to the present embodiment.

[0021] Figure 4 It is a diagram showing an occupation grid map according to this embodiment.

[0022] Figure 5 It is a diagram showing a method for generating an occupation grid map according to the present embodiment.

[0023] Figure 6 It is a diagram for explaining the forgetting of obstacle information according to the present embodiment.

[0024] Figure 7 It is a diagram showing the global path and local paths involved in this embodiment.

[0025] Figure 8 This is a flowchart showing a processing procedure for controlling the travel of a mobile body according to the present embodiment.

[0026] Fig. 9 This is a flowchart showing a process of forgetting stored obstacle information according to the present embodiment. DETAILED DESCRIPTION

[0027] Hereinafter, the embodiments are described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention involved in the technical solution. In addition, the combination of features described in the embodiments may not all be necessary for the invention. Two or more of the multiple features described in the embodiments may also be combined arbitrarily. In addition, the same or similar configurations are marked with the same reference numerals, and repeated descriptions are omitted.

[0028] <Structure of moving body>

[0029] 1 , the structure of a moving object 100 according to the present embodiment will be described. Figure 1A 1 shows a side view of a moving object 100 according to the present embodiment. Figure 1B 1 shows the internal structure of the moving body 100. In the figure, arrow X indicates the front-rear direction of the moving body 100, F indicates the front, and R indicates the rear. Arrows Y and Z indicate the width direction (left-right direction) and the up-down direction of the moving body 100.

[0030] The mobile body 100 is equipped with a battery 113, and is, for example, an ultra-small mobile body that moves mainly by the power of a motor. An ultra-small mobile vehicle is more compact than a general car, and is an ultra-small vehicle with a rated passenger capacity of about 1 or 2 people. In the present embodiment, a three-wheeled ultra-small mobile vehicle is used as an example of the mobile body 100, but it is not intended to limit the present invention, and for example, it may be a four-wheeled vehicle or a straddle-type vehicle. In addition, the vehicle of the present invention is not limited to a means of transportation, and may also be a vehicle that carries cargo and moves parallel to a person's walking, or a vehicle that leads a person. Furthermore, in the present invention, it is not limited to vehicles such as four-wheeled and two-wheeled vehicles, and can also be applied to walking robots that can move autonomously.

[0031] The mobile body 100 is an electric autonomous vehicle having a travel unit 112 and using a battery 113 as a main power source. The battery 113 is, for example, a rechargeable battery such as a lithium-ion battery, and the travel unit 112 is used to make the mobile body 100 travel autonomously by using the power supplied from the battery 113. The travel unit 112 is a three-wheeled vehicle having a pair of left and right front wheels 120 and a tail wheel (driven wheel) 121. The travel unit 112 may also be in other forms such as a four-wheeled vehicle. The mobile body 100 has a seat 111 for one person or two people.

[0032] The travel unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that changes the steering angle of the pair of front wheels 120 using motors 122a and 122b as a driving source. By changing the steering angle of the pair of front wheels 120, the travel direction of the mobile body 100 can be changed. The tail wheel 121 is a driven wheel that does not have an independent driving source but moves following the drive of the pair of front wheels 120. In addition, the tail wheel 121 is connected to the body of the mobile body 100 in a manner having a rotating portion. The rotating portion rotates in a manner that changes the direction of the tail wheel 121 separately from the rotation of the tail wheel 121. In this way, the mobile body 100 involved in this embodiment adopts a differential two-wheel type mobile body with a tail wheel, but is not limited to this.

[0033] The mobile body 100 is provided with a detection unit 114 for identifying a plane in front of the mobile body 100. The detection unit 114 is an external sensor for monitoring the front of the mobile body 100, and in the case of the present embodiment, is a camera device for capturing an image in front of the mobile body 100. In the present embodiment, the detection unit 114 is described by taking, for example, a stereo camera having an optical system such as two lenses and respective image sensors. However, a radar or an optical radar (Light Detection and Ranging) may be used instead of the camera or in addition. In addition, in the present embodiment, an example is described in which the detection unit 114 is only provided in front of the mobile body 100, but this is not intended to limit the present invention, and the detection unit 114 may also be provided in the rear, left, or right of the mobile body 100.

[0034] The mobile body 100 according to the present embodiment uses the detection unit 114 to capture the area in front of the mobile body 100 and detect obstacles based on the captured images. Furthermore, the mobile body 100 divides the surrounding area of ​​the mobile body 100 into a grid shape, and controls driving while generating an occupation grid map in which obstacle information is stored in each grid (hereinafter also referred to as a grid). The details of the occupation grid map will be described later.

[0035] <Control structure of moving body>

[0036] Figure 2It is a block diagram of the control system of the mobile body 100 involved in this embodiment. Here, the structure required for implementing the present invention is mainly described. Therefore, in addition to the structure described below, other structures may also be included. In addition, in this embodiment, the various parts described below are included in the mobile body 100 for description, but it is not intended to limit the present invention, and it can also be implemented as a mobile body control system including multiple devices. For example, part of the functions of the control unit 130 can also be implemented by a server device connected in a communicative manner, and the detection unit 114 and the GNSS sensor 134 can also be set as external devices. The mobile body 100 is equipped with a control unit (ECU) 130. The control unit 130 includes a processor represented by a CPU, a storage device such as a semiconductor memory, an interface with an external device, etc. The program executed by the processor, the data used by the processor in the processing, etc. are stored in the storage device. The processor, the storage device, and the interface can also be configured to be set in multiple groups according to the functions of the mobile body 100 so that they can communicate with each other.

[0037] The control unit 130 obtains the detection result of the detection unit 114, the input information of the operation panel 131, the sound information input from the sound input device 133, the position information from the GNSS sensor 134, and the received information via the communication unit 136, and performs corresponding processing. The control unit 130 controls the motors 122a and 122b (travel control of the travel unit 112), controls the display of the operation panel 131, reports to the occupants of the mobile body 100 based on the sound of the speaker 132, and outputs information.

[0038] The sound input device 133 collects the sound of the occupants of the mobile body 100. The control unit 130 can recognize the input sound and perform corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals to detect the current position of the mobile body 100. The storage device 135 is a storage device that stores the captured images of the detection unit 114, obstacle information, paths generated in the past, and occupancy grid maps. The storage device 135 can also store programs executed by the processor, data used by the processor in processing, etc. The storage device 135 can also store various parameters of the machine learning model used for sound recognition and image recognition performed by the control unit 130 (such as learning completion parameters and hyperparameters of deep neural networks, etc.).

[0039] The communication unit 136 communicates with the communication device 140 as an external device via wireless communication such as Wi-Fi and fifth-generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited thereto, and may also be a headset-type communication terminal, a personal computer, a tablet terminal, a game console, etc. The communication device 140 is connected to the network via wireless communication such as Wi-Fi and fifth-generation mobile communication.

[0040] A user who has the communication device 140 can give instructions to the mobile body 100 via the communication device 140. The instructions include, for example, instructions for summoning the mobile body 100 to the user's desired position and merging with it. When receiving the instructions, the mobile body 100 sets the target position based on the position information included in the instructions. In addition to such instructions, the mobile body 100 can also set the target position based on the captured image of the detection unit 114, or based on instructions from the user riding the mobile body 100 via the operation panel 131. In the case of setting the target position based on the captured image, for example, a person who raises his hand toward the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target position.

[0041] <Functional structure of mobile body>

[0042] Next, refer to Figure 3 , the functional structure of the mobile body 100 involved in this embodiment is described. The functional structure described here is realized by, in the control unit 130, for example, the CPU reading a program stored in a memory such as a ROM into a RAM and executing it. In addition, the functional structure described below only describes the functions required to describe the present invention, and does not describe all the functional structures actually included in the mobile body 100. That is, the functional structure of the mobile body 100 involved in the present invention is not limited to the functional structure described below.

[0043] The user instruction acquisition unit 301 has a function of receiving instructions from the user, and can receive user instructions via the operation panel 131, receive user instructions from an external device such as the communication device 140 via the communication unit 136, and receive instructions based on user speech via the voice input device 133. As described above, the user instructions include instructions for setting a target position (also called a destination) of the mobile body 100 and instructions related to the driving control of the mobile body 100.

[0044] The image information processing unit 302 processes the captured image acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image based on the stereo image acquired by the detection unit 114 and converts it into a three-dimensional point cloud. The image data after the three-dimensional point cloud is used to detect obstacles that hinder the travel of the mobile body 100. In addition, the image information processing unit 302 may also include a machine learning model for processing image information, and execute the processing of the learning stage and the inference stage of the machine learning model. The machine learning model of the image information processing unit 302 can, for example, perform operations using a deep learning algorithm using a deep neural network (DNN) to perform processing such as recognizing three-dimensional objects contained in the image information.

[0045] The grid map generation unit 303 creates a grid map of a predetermined size (for example, each unit in an area of ​​20m×20m is 10cm×10cm) based on the image data of the three-dimensional point cloud. This is because the amount of data in the three-dimensional point cloud is large and real-time processing is difficult, so the purpose is to reduce the size. The grid map is, for example, composed of a grid map indicating the difference between the maximum height and the minimum height of the point cloud in the grid (indicating whether the unit is a step) and a grid map indicating the maximum height of the point cloud in the grid from the reference point (indicating the terrain shape of the unit). Furthermore, the grid map generation unit 303 removes the spike noise and white noise contained in the generated grid map, detects obstacles with a height above a predetermined height, and generates an occupancy grid map indicating whether there are three-dimensional objects that become obstacles in each grid.

[0046] The path generation unit 304 generates a driving path for the mobile body 100 relative to the target position set by the user instruction acquisition unit 301. Specifically, the path generation unit 304 does not require obstacle information of a high-precision map, but uses an occupied grid map generated by the grid map generation unit 303 based on the captured image of the detection unit 114 to generate a path. In addition, the detection unit 114 is a stereo camera that captures the area in front of the mobile body 100, and therefore cannot identify obstacles in other directions. Therefore, in order to avoid colliding with obstacles outside the field of view and getting stuck in a dead end, the mobile body 100 preferably stores the detected obstacle information for a predetermined period of time. As a result, the mobile body 100 can generate a path by considering both obstacles detected in the past and obstacles detected in real time.

[0047] In addition, the path generation unit 304 uses the occupancy grid map to periodically generate a global path (global path), and also periodically generates a local path (local path) in a manner that follows the global path. That is, the target position of the local path is determined by the global path. In addition, as the generation period of each path, in the present embodiment, the generation period of the global path is set to 100ms, and the generation period of the local path is set to 50ms, but this is not a limitation of the present invention. As algorithms for generating a global path, various algorithms such as RRT (Rapid-exploring Random Tree), PRM (Probabilistic Road Map) and A* are known. In addition, since a differential two-wheel type mobile body with a tail wheel is used as the mobile body 100, the path generation unit 304 generates a local path that takes into account the tail wheel 121 as a driven wheel.

[0048] The driving control unit 305 controls the driving of the mobile body 100 according to the local path. Specifically, the driving control unit 305 controls the driving unit 112 according to the local path, thereby controlling the speed and angular velocity of the mobile body 100. Furthermore, the driving control unit 305 controls the driving according to various operations of the driver. When the driving plan of the local path deviates due to the operation of the driver, the driving control unit 305 can obtain a new local path generated by the path generation unit 304 again to control the driving, or can control the speed and angular velocity of the mobile body 100 in a manner to eliminate the deviation from the local path in use.

[0049] <Occupy Grid Map>

[0050] Figure 4 The occupancy grid map 400 including obstacle information according to the present embodiment is shown. The mobile body 100 according to the present embodiment travels without relying on the obstacle information of the high-precision map, so all the obstacle information is obtained from the recognition result of the detection unit 114. At this time, in order to avoid collision with obstacles outside the field of view and getting stuck in a dead end, it is necessary to store the obstacle information. Therefore, in the present embodiment, as a method of storing obstacle information, an occupancy grid map is used from the perspective of reducing the amount of information of the three-dimensional point cloud of the stereo image and the ease of processing in the path planning.

[0051] The grid map generation unit 303 of the present embodiment divides the surrounding area of ​​the mobile body 100 into a grid shape, and generates an occupancy grid map containing information indicating the presence or absence of obstacles according to the grid (divided area). In addition, here, an example of dividing a predetermined area into a grid shape is used for explanation, but it is also possible to divide it into other shapes instead of a grid shape, and to create an occupancy map indicating the presence or absence of obstacles for each divided area. The occupancy grid map 400 divides the area of ​​the surrounding area of ​​the mobile body 100, such as 40m×40m and 20m×20m, into grids (grids) of 20cm×20cm and 10cm×10cm, and is dynamically set according to the movement of the mobile body 100. That is, the occupancy grid map 400 is an area that shifts in real time in accordance with the movement of the mobile body 100 so that the mobile body 100 always comes to the center. In addition, the size of the area can be arbitrarily set according to the hardware resources of the mobile body 100.

[0052] In addition, in the occupancy grid map 400, the presence or absence information of obstacles detected from the captured image of the detection unit 114 is defined in each grid. As the presence or absence information, for example, a drivable area is defined as "0" and a non-drivable area (i.e., an obstacle) is defined as "1". Figure 4 In FIG. 4 , 401 indicates a grid where obstacles exist. The area where obstacles exist is an area that the moving body 100 cannot pass through, and is composed of, for example, a three-dimensional object larger than 5 cm. Therefore, the moving body 100 generates a path in a manner that avoids these obstacles 401.

[0053] <Storage of obstacle information>

[0054] Figure 5 The storage of obstacle information in the occupied grid map involved in the present embodiment is described. 500 represents a local map that moves in coordination with the movement of the mobile body 100. The local map 500 is shifted according to the movement of the mobile body 100 relative to the x-axis direction and the y-axis direction on the grid map. For example, the local map 500 shows a situation where the dotted area 501 is deleted and the solid area 502 is added according to the movement amount Δx of the mobile body 100 in the x-axis direction. The deleted area becomes the area opposite to the moving direction of the mobile body 100, and the added area becomes the area in the moving direction. Similarly, with respect to the y-axis direction, the deletion and addition of areas are also performed according to the movement of the mobile body 100.

[0055] In addition, the local map 500 stores obstacle information detected in the past. In addition, when an obstacle exists in a grid included in the deleted area, the obstacle information is deleted from the local map 500, but it is preferably kept separately from the local map 500 for a certain period of time. Such information is effective, for example, when the mobile body 100 changes its route and the deleted area is included in the local map 500 again, and can improve the accuracy of the mobile body 100 in avoiding obstacles. In addition, by using the stored information, it is not necessary to detect obstacles again, which can reduce the processing load.

[0056] Furthermore, the local map 500 is forgotten according to the forgetting rate set for each grid before being added to the obstacle detection map 510 described later. When a dynamic obstacle is detected while moving, if the obstacle information previously detected and stored in the grid is kept, it may cause false detection that obstacles exist in all grids along the moving track of the obstacle. Therefore, in order to avoid mistakenly judging that a grid through which an obstacle has passed exists, it is necessary to forget the stored information of the obstacle after a certain period of time. Figure 6 This will be described later.

[0057] 510 represents an obstacle detection map, which represents detection information of obstacles existing in the front and surrounding areas of the mobile body 100 based on the captured images captured by the detection unit 114 of the mobile body 100. The obstacle detection map 510 represents real-time information and is periodically generated based on the captured images acquired from the detection unit 114. In addition, since moving obstacles such as people and vehicles are also assumed, it is preferred that obstacles detected in the past are not fixedly stored in the field of view 511 of the detection unit 114, which is the front area of ​​the mobile body 100, but are updated by the periodically generated obstacle detection map 510. In this way, moving obstacles can also be identified, and the generation of unnecessary detour routes can be prevented. On the other hand, with respect to the rear area of ​​the mobile body 100 (strictly speaking, outside the field of view of the detection unit 114), information on obstacles detected in the past is stored as shown in the local map 500. In this way, for example, when a detour route is generated when an obstacle is detected in the front area, a route that avoids collision with the passing obstacle can be easily generated.

[0058] Reference numeral 520 indicates an occupancy grid map generated by adding the local map 500 and the obstacle detection map 510. In this way, the occupancy grid map 520 is generated as a grid map synthesized from the local map and obstacle detection information that change in real time and the obstacle information that was detected and stored in the past.

[0059] <Obstacle information forgotten>

[0060] Figure 6 The forgetting of obstacle information in the occupancy grid map involved in the present embodiment is described. 600 represents an occupancy grid map including obstacle information 603 and 604 generated around the mobile body 100. 601 represents the actual viewing angle (performance) of the detection unit 114. That is, the viewing angle 601 represents the shooting range determined by the performance of the detection unit 114 such as a stereo camera. 602 represents the range within the viewing angle defined by a range narrower than the actual viewing angle 601 when the forgetting rate of the obstacle information is set. Thus, even when the detection accuracy of obstacles in the end area in the captured image is reduced, it is possible to reduce erroneous detection of obstacles.

[0061] In this embodiment, a forgetting rate is set for each grid. Here, the forgetting rate indicates the degree to which the stored obstacle information is retained. For example, according to this embodiment, the occupancy grid map is generated periodically, and the forgetting rate indicates how many cycles the obstacle information is stored.

[0062] Specifically, in this embodiment, the forgetting rate for forgetting the stored obstacle information is set to different values ​​for each grid included in the viewing angle range 602 and the grid not included in the viewing angle range 602. Figure 6 The first forgetting rate is set for the grids overlapping the viewing angle range 602 shown, and the second forgetting rate lower than the first forgetting rate is set for the other grids. Here, the grids overlapping the viewing angle range 602 refer to the grids overlapping the viewing angle range 602 with a predetermined area or more in the grid. The size of the predetermined area is arbitrary, and can be set to 1 to 100%, for example.

[0063] Thus, according to the present embodiment, the forgetting rate is set high for the grids within the field of view of the detection unit 114, and the forgetting rate is set low for other areas. By setting in this way, obstacle information can be forgotten as early as possible within the field of view as a countermeasure for dynamic obstacles accompanying movement. On the other hand, outside the field of view, by storing obstacle information for a certain period of time, collisions with obstacles and dead ends can be avoided. However, in order to reduce the computational load of path generation, it is preferred to maintain obstacle information in a predetermined peripheral area of ​​the mobile body 100 and forget other areas. Therefore, according to the present embodiment, the forgetting rate is also set outside the field of view.

[0064] In addition, here, an example of setting different forgetting rates within and outside the field of view is described, but it is also possible to further set them separately in each area. For example, it is also possible to set the forgetting rate to be higher the farther the distance from the moving body 100 is outside the field of view. In addition, the category of the obstacle can be determined, and the forgetting rate can be changed according to the determined category. The category of obstacles includes at least dynamic obstacles that are accompanied by movement, and static obstacles that are not accompanied by movement. Dynamic obstacles are, for example, vehicles and other traffic participants. Static obstacles are, for example, obstacles that do not move by themselves, and also include fixed objects such as traffic lights and pillars, and tables and stands that are not fixed but can move. In addition, for areas that contain shadows in the captured image used to detect obstacles, the forgetting rate can be set higher than other areas. Thus, even if the detection accuracy of obstacles in the area containing shadows is low, the influence of erroneously detected obstacle information can be reduced. In addition, when there are many obstacles in the grid map or the proportion of obstacles is large, the forgetting rate can be set high. Thus, it is possible to avoid extreme obstruction of path generation due to information of obstacles detected in the past. Regarding specific forgetting processing, use Fig. 9 This will be described later.

[0065] <Path Generation>

[0066] Figure 7 1 shows the travel route generated in the mobile body 100 according to the present embodiment. The route generation unit 304 according to the present embodiment periodically generates a global path 702 using an occupancy grid map according to a set target position 701, and periodically generates a local path 703 so as to follow the global path.

[0067] The target position 701 is set based on various instructions. For example, it includes instructions from passengers riding on the mobile body 100 and instructions from users outside the mobile body 100. The instructions from the passengers are performed via the operation panel 131 and the voice input device 133. The instructions via the operation panel 131 can also be a method of specifying a predetermined grid of the grid map displayed on the operation panel 131. In this case, the size of each grid can also be set larger, so that it can be selected from a wider range of maps. The instructions via the voice input device 133 can also be instructions that target surrounding targets. The target object can include pedestrians, signs, logos, vending machines and other equipment set in the wild, components of buildings such as windows or entrances, roads, vehicles, two-wheeled vehicles, etc. included in the speech information. If the instruction via the voice input device 133 is received, the path generation unit 304 detects the specified target object from the captured image obtained by the detection unit 114 and sets it as the target position.

[0068] For these voice recognition and image recognition, machine learning models are used. The machine learning model, for example, performs operations using a deep learning algorithm using a deep neural network (DNN) to recognize the place names, landmark names such as buildings, store names, names of targets, etc. contained in speech information and image information. The DNN for voice recognition becomes a learned state by performing the learning phase. By inputting new speech information into the learned DNN, it is possible to perform recognition processing (inference phase processing) for the new speech information. In addition, the DNN for image recognition can recognize pedestrians, signs, vending machines and other equipment located outdoors, building components such as windows or entrances, roads, vehicles, and two-wheeled vehicles contained in the image.

[0069] In addition, regarding instructions from a user outside the mobile body 100, the mobile body 100 can also be notified of instructions via the communication device 140 owned by the mobile body 100 via the communication unit 136, or as shown in FIG. Figure 7 As shown, the moving object 100 is summoned by raising a hand toward the moving object 100. The instruction from the communication device 140 is performed by operation input or voice input similarly to the instruction from the above-mentioned occupant.

[0070] When the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupancy grid map. As described above, various algorithms such as RRT, PRM, and A* are known as methods for generating a global path, but any method may be used. Next, the path generation unit 304 generates a local path 703 in a manner that follows the generated global path 702. As methods for local path planning, there are various methods such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0071] <Basic Control of Moving Objects>

[0072] Figure 8 1 is a flowchart showing basic control of the mobile body 100 according to the present embodiment. The processing described below is implemented by the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into a RAM and executing the program.

[0073] In S101, the control unit 130 sets the target position of the moving body 100 based on the user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be accepted by various methods. Next, in S102, the control unit 130 captures the area in front of the moving body 100 through the detection unit 114 to obtain the captured image. The acquired camera image is processed by the image information processing unit 302 to generate a depth image and perform three-dimensional point clouding. In S103, the control unit 130 detects obstacles, such as solid objects larger than 5 cm, from the three-dimensional point clouded image. In S104, the control unit 130 generates an occupancy grid map of a predetermined area centered on the moving body 100 based on the detected obstacles and the position information of the moving body 100. Use Fig. 9 The detailed method is described.

[0074] Next, in S105, the control unit 130 generates a driving path for the mobile body 100 through the path generation unit 304. As described above, the path generation unit 304 generates a global path using the occupancy grid map, and generates a local path based on the generated global path. Next, in S106, the control unit 130 determines the speed and angular velocity of the mobile body 100 according to the generated local path, and controls the driving. Thereafter, in S107, the control unit 130 determines whether the mobile body 100 has reached the target position based on the position information from the GNSS sensor 134, and returns the processing to S102 if the target position has not been reached, generates a path while updating the occupancy grid map, and repeatedly performs the process of controlling the driving. On the other hand, if the target position has been reached, the processing of this flowchart is terminated.

[0075] <Method for generating an occupation grid map (forgetfulness control)>

[0076] Fig. 9 This is a flowchart showing a detailed process procedure of the generation control (S104) of the occupation grid map according to the present embodiment. The process described below is implemented by the control unit 130, for example, by the CPU reading a program stored in a memory such as a ROM into a RAM and executing the program.

[0077] First, in S201, the control unit 130 obtains the moving speed of the moving body 100 from the travel unit 112. Next, in S202, the control unit 130 determines whether the obtained moving speed of the moving body 100 is 0, that is, whether the moving body 100 is in a stopped state. If the moving speed is 0, the process proceeds to S203, otherwise the process proceeds to S204.

[0078] In S203, when the moving body 100 is in a stopped state, the control unit 130 sets the forgetting rate k_out outside the viewing angle to 0 to avoid forgetting the stored obstacle information, and the process proceeds to S205. On the other hand, in S204, when the moving body 100 is in a moving state (driving state), the control unit 130 sets a default value for the forgetting rate outside the viewing angle, and the process proceeds to S205. In addition, regarding the forgetting rate k_in within the viewing angle, a predetermined value is set regardless of the moving speed of the moving body 100.

[0079] An example of a method for determining the forgetting rate will be described. The forgetting rate is determined based on the following equations (1) and (2).

[0080] Storage value max = k_in / dt*dynamic_object_forget_time... Formula (1)

[0081] k_out=storage max*dt / out_of_fov_forget_time ···Formula (2)

[0082] Here, "k_in" represents the forgetting rate within the field of view. "k_out" represents the forgetting rate outside the field of view. "dynamic_object_forget_time" represents the forgetting time within the field of view. "out_of_fov_forget_time" represents the forgetting time outside the field of view. "dt" represents the period. By using the above formulas (1) and (2) to determine the forgetting rate, it is possible to update the obstacle information within the field of view earlier, and to attach importance to the past obstacle information outside the field of view.

[0083] In S205, the control unit 130 forgets the stored information of the obstacle. This forgetting process is performed using the following equation (3).

[0084] map=map-k_in*fovmap-k_out(1-fovmap)...Equation (3)

[0085] Here, "fovmap" represents a grid map within the field of view. For example, "fovmap" represents a grid within the field of view with "1" and represents a grid outside the field of view with "0". According to equation (3), as described above, the forgetting process is performed by changing the forgetting rate within and outside the field of view. When the forgetting process is performed, for example, Figure 5 Local map 500 is shown.

[0086] Next, in S206, the control unit 130 obtains the obstacle information detected in S103 (for example, Figure 5Next, in S207, the control unit 130 adds the map to which the forgetting process was performed in S205 and the obstacle detection map acquired in S206 using the following equation (4), thereby generating an occupancy grid map with updated obstacle information.

[0087] Map=map+k_acc*new_map…Formula (4)

[0088] Here, "Map" indicates the occupancy grid map with updated obstacle information, "map" is the obstacle detection map that has been forgotten in S205, and "new_map" indicates the newly detected obstacle detection map obtained in S206. "k_acc" indicates the stored coefficient. This coefficient is set to a value that does not exceed the stored value max.

[0089] When the process of S207 is completed, the process of this flowchart is terminated, and the process proceeds to S105. In addition, the processes of S201 to S207 are performed periodically, for example, at a cycle of 10 Hz.

[0090] <Summary of implementation methods>

[0091] 1. The mobile object control system (for example, 100) of the above embodiment is characterized in that the mobile object control system includes:

[0092] A storage mechanism (303) stores information on obstacles detected in the past according to divided areas formed by dividing the surrounding area of ​​the moving body;

[0093] A forgetting mechanism (303) which forgets the information of the obstacle stored by the storage mechanism according to a predetermined forgetting rate;

[0094] A photographing mechanism (114) for acquiring a photographed image;

[0095] A detection mechanism (130, 302, 303) for detecting obstacles contained in the captured image; and

[0096] A map generating means (303) generates an occupancy map showing the occupancy of obstacles by divided area according to the obstacle information stored by the storing means and the obstacles detected by the detecting means for the current surrounding area of ​​the moving body.

[0097] According to this embodiment, the forgetting rate of the information of the obstacles detected around the moving body is appropriately set for each area. Therefore, according to the present invention, it is possible to take measures against dynamic obstacles and effectively utilize the information of obstacles detected in the past to avoid collisions with these obstacles and dead ends.

[0098] 2. In the mobile object control system of the above embodiment, the predetermined forgetting rate is different between the inside and outside of the angle of view of the imaging mechanism ( Figure 6 ).

[0099] According to this embodiment, by switching the forgetting rate inside and outside the viewing angle, it is possible to effectively utilize obstacle information detected in the past and update the obstacle information.

[0100] 3. In the mobile object control system of the above embodiment, the forgetting rate within the viewing angle is higher than the forgetting rate outside the viewing angle ( Figure 6 ).

[0101] According to this embodiment, obstacle information is updated earlier within the field of view as a countermeasure against dynamic obstacles that move, and past detection information is retained for a certain period outside the field of view, thereby avoiding collision with obstacles and getting stuck in a dead end.

[0102] 4. In the mobile object control system of the above embodiment, the area within the angle of view is set to be narrower than the imaging range of the imaging mechanism determined according to the performance of the imaging mechanism (601, Figure 6 ).

[0103] According to this embodiment, it is possible to reduce erroneous detection of obstacles caused by the detection accuracy of the end region in the captured image.

[0104] 5. In the mobile object control system of the above embodiment, the mobile object control system further comprises a determination means for determining the type of the detected obstacle.

[0105] The predetermined forgetting rate is set according to the type of the obstacle.

[0106] According to this embodiment, obstacle information detected in the past can be effectively used according to the type of the obstacle.

[0107] 6. In the moving object control system of the above embodiment, the types of obstacles include at least dynamic obstacles that are accompanied by movement and static obstacles that are not accompanied by movement.

[0108] According to this embodiment, the forgetting rate can be appropriately set according to a dynamic obstacle or a static obstacle.

[0109] 7. In the moving object control system of the above embodiment, the forgetting rate set for the dynamic obstacle is higher than the forgetting rate set for the static obstacle.

[0110] According to this embodiment, since dynamic obstacles are forgotten earlier due to accompanying movement, and since static obstacles do not move autonomously, past detection information is retained for a certain period of time, collision with obstacles and dead ends can be avoided.

[0111] 8. In the mobile object control system of the above embodiment, the dynamic obstacle refers to a vehicle or other traffic participants.

[0112] According to this embodiment, by determining a traffic participant as a dynamic obstacle, a path of a moving object can be appropriately generated.

[0113] 9. In the mobile object control system of the above embodiment, the static obstacle refers to an obstacle that cannot move autonomously.

[0114] According to this embodiment, by determining a fixed obstacle as a static obstacle, it is possible to appropriately generate a path for a moving object.

[0115] 10. In the moving object control system of the above embodiment, the forgetting rate of the region including the shadow in the captured image is higher than the forgetting rate of the region not including the shadow in the captured image.

[0116] According to this embodiment, a region including a shadow where the obstacle detection accuracy is low can be forgotten earlier, thereby reducing the influence of erroneous detection.

[0117] 11. In the mobile object control system of the above embodiment, the forgetting rate is set higher as the number of obstacles increases.

[0118] According to this embodiment, it is possible to avoid a situation where the generation of a route is extremely hampered by an obstacle detected in the past.

[0119] 12. In the moving object control system of the above embodiment, the forgetting rate is set higher as the number of divided areas occupied by the obstacle increases.

[0120] According to this embodiment, it is possible to avoid a situation where the generation of a route is extremely hampered by an obstacle detected in the past.

[0121] 13. In the mobile body control system of the above embodiment, the map generating means divides the area around the mobile body into a grid shape, and generates an occupancy grid map which represents the occupancy of obstacles detected by the detecting means in grids as the occupancy map.

[0122] According to this embodiment, a predetermined plane area can be easily divided in the x and y directions, and a predetermined range can be covered without omission.

[0123] The present invention is not limited to the above-described embodiments, and various modifications and changes can be made within the scope of the gist of the invention.

[0124] This application claims priority based on Japanese patent application No. 2022-158638 filed on September 30, 2022, and all the contents described are incorporated herein by reference.

Claims

1. A mobile control system, characterized in that: The mobile control system comprises: A storage mechanism for storing information of obstacles detected in the past according to divided areas formed by dividing the surrounding area of ​​the moving body; a forgetting mechanism for forgetting the information of the obstacle stored by the storage mechanism according to a predetermined forgetting rate allocated to each divided area; A photographing mechanism for acquiring photographed images; A detection mechanism, which detects obstacles contained in the captured image; as well as A map generating means generates an occupancy map showing occupancy of obstacles for each divided area according to the obstacle information stored by the storing means and the obstacles detected by the detecting means for the current surrounding area of ​​the moving body.

2. The mobile body control system according to claim 1, characterized in that: The predetermined forgetting rate is different within and outside the viewing angle of the photographing mechanism.

3. The mobile body control system according to claim 2, characterized in that: The forgetting rate within the viewing angle is higher than the forgetting rate outside the viewing angle.

4. The mobile body control system according to claim 2 or 3, characterized in that: The area within the angle of view is set to be narrower than the imaging range of the imaging mechanism determined according to the performance of the imaging mechanism.

5. The mobile body control system according to any one of claims 1 to 4, characterized in that: The mobile object control system further comprises a determination unit that determines the type of the detected obstacle. The predetermined forgetting rate is set according to the type of the obstacle.

6. The mobile body control system according to claim 5, characterized in that: The types of obstacles include at least dynamic obstacles that move and static obstacles that do not move.

7. The mobile body control system according to claim 6, characterized in that: The forgetting rate set for the dynamic obstacle is higher than the forgetting rate set for the static obstacle.

8. The mobile body control system according to claim 6 or 7, characterized in that: The dynamic obstacles refer to vehicles or other traffic participants.

9. The mobile body control system according to any one of claims 6 to 8, characterized in that: The static obstacle refers to an obstacle that cannot move autonomously.

10. The mobile body control system according to any one of claims 1 to 9, characterized in that: The forgetting rate of the region including the shadow in the captured image is higher than the forgetting rate of the region not including the shadow in the captured image.

11. The mobile body control system according to any one of claims 1 to 10, characterized in that: The greater the number of obstacles, the higher the forgetting rate is set.

12. The mobile body control system according to any one of claims 1 to 11, characterized in that: The greater the number of divided areas occupied by the obstacle, the higher the forgetting rate is set.

13. The mobile body control system according to any one of claims 1 to 12, characterized in that: The map generating means divides the area around the mobile body into a grid shape, and generates an occupancy grid map which indicates the occupancy of obstacles detected by the detecting means in grids as the occupancy map.

14. A control method for a mobile control system, characterized in that: The control method comprises: a storing step of storing information of obstacles detected in the past for each divided area formed by dividing the surrounding area of ​​the moving body; a forgetting step in which the information of the obstacle stored in the storing step is forgotten according to a predetermined forgetting rate assigned to each divided area; A photographing step, in which a photographed image is acquired; a detection step, in which obstacles contained in the captured image are detected; as well as A map generating step, in which an occupancy map showing occupancy of obstacles by divided area is generated for a current surrounding area of ​​the mobile body according to the obstacle information stored in the storing step and the obstacles detected in the detecting step. 15 . A program for causing a computer to function as each mechanism of the mobile object control system according to claim 1 .

16. A mobile object, characterized in that: The mobile body comprises: a storage mechanism for storing information of obstacles detected in the past in each divided area formed by dividing the surrounding area of ​​the moving body; a forgetting mechanism for forgetting the information of the obstacle stored by the storage mechanism according to a predetermined forgetting rate allocated to each divided area; A photographing mechanism for acquiring photographed images; A detection mechanism, which detects obstacles contained in the captured image; as well as A map generating means generates an occupancy map showing occupancy of obstacles for each divided area according to the obstacle information stored by the storing means and the obstacles detected by the detecting means for the current surrounding area of ​​the moving body.

Images