Mobile vehicle control system, control method thereof, program, and mobile vehicle

Abstract

This mobile body control system accumulates information on obstacles detected in the past in each divided region made by dividing the peripheral region of the mobile body, and has the accumulated obstacle information forgotten according to a prescribed forgetting rate allocated to each divided region. This mobile body acquires captured images, detects obstacles included in the captured images, and generates an occupancy map showing the occupancy of obstacles for each divided region according to the accumulated obstacle information and the detected obstacles for the current peripheral region of the mobile body.

Description

Technical Field

[0005]

[0001] The present invention relates to a movement control system, its control method, program, and a moving body.

Background Art

[0002] In recent years, small moving bodies such as electric vehicles with about 1 to 2 passengers, called ultra-small mobility (also referred to as micromobility), and mobile robots that provide various services to people are known. Some of these moving bodies perform autonomous driving while periodically generating a driving route to a destination.

[0003] Patent Document 1 proposes a technique for performing autonomous driving by measuring and evaluating the shape of a road surface in detail to avoid unevenness that becomes an obstacle, generating an obstacle map from the evaluation result, and using it for route planning. More specifically, it proposes a method for removing ghosts in moving obstacles while updating the obstacle map by sequentially reflecting the external measurement results. Patent Document 2 proposes generating a dynamically changing surrounding map based on sensor information at two different points in time using an ultrasonic sensor. Patent Document 3 proposes a technique for eliminating non-existent moving objects based on reflected pulses received by a radar sensor and leaving only highly reliable real images in road map data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] Micromobility devices are small, and it is necessary to minimize the use of hardware resources. Therefore, it is desirable to use a single sensor that detects the area in front of the moving object as the sensor for detecting the surrounding environment. In such a configuration, when detecting obstacles using a detection unit such as a camera, the method of retaining or discarding the accumulated information of detected obstacles (e.g., forgetting rate) will differ depending on the area around the moving object. This is because changes in obstacles can be detected within the range of the image captured by the camera, but such changes cannot be detected outside that range.

[0006] On the other hand, when generating an autonomous driving path using detected obstacle information around a moving object, for example, if an obstacle is detected in front of the moving object, it is conceivable that a path will be generated that passes near the obstacle, which has shifted out of the field of view in order to detour. In such cases, by accumulating previously detected obstacle information for a certain period of time, it is possible to generate a path that avoids the obstacle that has shifted out of the field of view.

[0007] The present invention has been made in view of the above problems, and aims to suitably set the forgetting rate of information on obstacles detected around a moving object for each region. [Means for solving the problem]

[0008] According to the present invention, for example, a mobile body control system comprises: storage means for storing information on previously detected obstacles for each divided region obtained by dividing the surrounding area of ​​a mobile body; forgetting means for forgetting the information on obstacles stored by the storage means according to a predetermined forgetting rate assigned to each divided region; imaging means for acquiring captured images; detection means for detecting obstacles included in the captured images; and map generation means for generating an occupancy map for each divided region showing the occupation of obstacles in the current surrounding area of ​​the mobile body, according to the information on obstacles stored by the storage means and the obstacles detected by the detection means. The predetermined forgetting rate is set according to whether the divided region is currently observable by the imaging means. It is characterized by the following. [Effects of the Invention]

[0009] According to the present invention, the forgetting rate of information about obstacles detected around a moving object can be suitably set for each region. [Brief explanation of the drawing]

[0010] [Figure 1A] Block diagram showing an example of the hardware configuration of the mobile unit according to this embodiment. [Figure 1B] Block diagram showing an example of the hardware configuration of the mobile unit according to this embodiment. [Figure 2] Block diagram showing the control configuration of the mobile body according to this embodiment. [Figure 3] Block diagram showing the functional configuration of the mobile body according to this embodiment. [Figure 4] Figure showing an occupying grid map according to this embodiment. [Figure 5] This figure shows the method for generating an occupy grid map according to this embodiment. [Figure 6] Diagram illustrating the forgetting of obstacle information according to this embodiment. [Figure 7] Figure showing the global and local routes according to this embodiment. [Figure 8] Flowchart showing the processing procedure for controlling the movement of the mobile body according to this embodiment. [Figure 9] Flowchart showing the procedure for forgetting accumulated obstacle information according to this embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features from the multiple features described in the embodiments may be combined arbitrarily. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0012] <Mobile Unit Configuration> Referring to FIG. 1, the configuration of the mobile body 100 according to this embodiment will be described. FIG. 1A shows a side view of the mobile body 100 according to this embodiment, and FIG. 1B shows the internal configuration of the mobile body 100. In the figures, arrow X indicates the front-rear direction of the mobile body 100, with F indicating the front and R indicating the rear. Arrows Y and Z indicate the width direction (left-right direction) and the up-down direction of the mobile body 100, respectively.

[0013] The mobile body 100 is equipped with a battery 113 and is, for example, a super-compact mobility that mainly moves by the power of a motor. Super-compact mobility is a super-compact vehicle that is more compact than a general automobile and has a seating capacity of about 1 or 2 people. In this embodiment, as an example of the mobile body 100, a three-wheeled super-compact mobility will be described, but there is no intention to limit the present invention, and for example, a four-wheeled vehicle or a straddle-type vehicle may also be used. Further, the vehicle of the present invention is not limited to a passenger vehicle, and may be a vehicle that carries luggage and runs parallel to a person's walking, or a vehicle that leads a person. Furthermore, the present invention is applicable not only to vehicles such as four-wheeled and two-wheeled vehicles, but also to walking-type robots that can move independently.

[0014] The mobile body 100 includes a traveling unit 112 and is an electric autonomous vehicle with the battery 113 as the main power source. The battery 113 is a secondary battery such as a lithium-ion battery, for example, and the mobile body 100 self-propels by the traveling unit 112 using the power supplied from the battery 113. The traveling unit 112 is a three-wheeled vehicle including a pair of left and right front wheels 120 and a rear wheel (driven wheel) 121. The traveling unit 112 may be in other forms such as the form of a four-wheeled vehicle. The mobile body 100 includes a seat 111 for one person or two people.

[0015] The traveling unit 112 includes a steering mechanism 123. The steering mechanism 123 is a mechanism that changes the steering angles of a pair of front wheels 120 using motors 122a and 122b as drive sources. By changing the steering angles of the pair of front wheels 120, the traveling direction of the moving body 100 can be changed. The rear wheel 121 does not have an individual drive source and is a driven wheel that operates following the drive of the pair of front wheels 120. Also, the rear wheel 121 is connected to the vehicle body of the moving body 100 with a swivel part. The swivel part rotates so that the direction of the rear wheel 121 changes separately from the rotation of the rear wheel 121. Thus, the moving body 100 according to the present embodiment adopts differential two-wheel mobility with a rear wheel, but is not limited thereto.

[0016] The moving body 100 includes a detection unit 114 that recognizes the plane in front of the moving body 100. The detection unit 114 is an external sensor that monitors the front of the moving body 100, and in the case of the present embodiment, is an imaging device that captures an image of the front of the moving body 100. In the present embodiment, the detection unit 114 will be described by taking, for example, a stereo camera having an optical system such as two lenses and respective image sensors as an example. However, it is also possible to adopt a radar or a lidar (Light Detection and Ranging) instead of or in addition to the imaging device. Also, in the present embodiment, an example of providing only in front of the moving body 100 will be described, but there is no intention to limit the present invention, and it may be provided behind or on the left and right of the moving body 100.

[0017] The moving body 100 according to the present embodiment captures an image of the front area of the moving body 100 using the detection unit 114 and detects an obstacle from the captured image. Further, the moving body 100 divides the peripheral area of the moving body 100 into a grid pattern and controls its travel while generating an occupancy grid map in which obstacle information is accumulated in each grid (hereinafter also referred to as a grid). Details of the occupancy grid map will be described later.

[0018] <Control Configuration of the Moving Body> Figure 2 is a block diagram of the control system of the mobile body 100 according to this embodiment. Here, the configuration necessary for carrying out the present invention will be mainly described. Therefore, other configurations may be included in addition to the configuration described below. Also, in this embodiment, each part described below is described as being included in the mobile body 100, but there is no intention to limit the present invention, and it may be realized as a mobile body control system including multiple devices. For example, some functions of the control unit 130 may be realized by a server device that is connected to communicate, and the detection unit 114 and GNSS sensor 134 may be provided as external devices. The mobile body 100 includes a control unit (ECU) 130. The control unit 130 includes a processor represented by a CPU, a storage device such as a semiconductor memory, and an interface with an external device. The storage device stores programs executed by the processor and data used by the processor for processing. Multiple sets of processors, storage devices, and interfaces may be provided according to the functions of the mobile body 100 and configured to communicate with each other.

[0019] The control unit 130 acquires the detection results from the detection unit 114, the input information from the operation panel 131, the voice information input from the voice input device 133, the position information from the GNSS sensor 134, and the received information via the communication unit 136, and executes the corresponding processing. The control unit 130 controls the motors 122a and 122b (driving control of the driving unit 112), controls the display on the operation panel 131, provides voice notifications to the occupants of the mobile body 100 via the speaker 132, and outputs information.

[0020] The voice input device 133 captures the voices of the occupants of the mobile unit 100. The control unit 130 recognizes the input voice and can execute corresponding processing. The GNSS (Global Navigation Satellite system) sensor 134 receives GNSS signals and detects the current position of the mobile unit 100. The storage device 135 is a storage device that stores images captured by the detection unit 114, obstacle information, previously generated routes, and occupied grid maps. The storage device 135 may also store programs executed by the processor and data used by the processor for processing. The storage device 135 may also store various parameters of machine learning models for speech recognition and image recognition executed by the control unit 130 (for example, trained parameters and hyperparameters of a deep neural network).

[0021] The communication unit 136 communicates with an external device, the communication device 140, via wireless communication such as Wi-Fi or fifth-generation mobile communication. The communication device 140 is, for example, a smartphone, but is not limited to this; it may also be an earphone-type communication terminal, a personal computer, a tablet device, a game console, etc. The communication device 140 connects to a network via wireless communication such as Wi-Fi or fifth-generation mobile communication.

[0022] A user who owns the communication device 140 can issue instructions to the mobile body 100 via the communication device 140. Such instructions include, for example, instructions to summon the mobile body 100 to a location desired by the user and meet up. Upon receiving such instructions, the mobile body 100 sets a target location based on the location information included in the instructions. In addition to such instructions, the mobile body 100 can also set a target location from images captured by the detection unit 114, or from instructions given by a user riding in the mobile body 100 via the operation panel 131. When setting a target location from captured images, for example, a person raising their hand towards the mobile body 100 is detected in the captured image, and the position of the detected person is estimated and set as the target location.

[0023] <Functional configuration of the mobile unit> Next, with reference to Figure 3, the functional configuration of the mobile body 100 according to this embodiment will be described. The functional configuration described here is realized in the control unit 130 by, for example, the CPU reading a program stored in memory such as ROM into RAM and executing it. Note that the functional configuration described below will only describe the functions necessary to explain the present invention and will not describe all of the functional configurations actually included in the mobile body 100. In other words, the functional configuration of the mobile body 100 according to the present invention is not limited to the functional configuration described below.

[0024] The user instruction acquisition unit 301 has the function of receiving instructions from the user and can receive user instructions via the operation panel 131, user instructions from external devices such as the communication device 140 via the communication unit 136, and instructions uttered by the user via the voice input device 133. As described above, user instructions include instructions to set the target position (also referred to as the destination) of the mobile body 100 and instructions related to the driving control of the mobile body 100.

[0025] The image information processing unit 302 processes the captured image acquired by the detection unit 114. Specifically, the image information processing unit 302 creates a depth image from the stereo image acquired by the detection unit 114 and converts it into a 3D point cloud. The 3D point cloud image data is used to detect obstacles that obstruct the movement of the moving object 100. The image information processing unit 302 may also include a machine learning model for processing image information and may perform processing for the learning stage and the inference stage of the machine learning model. The machine learning model of the image information processing unit 302 can perform processing to recognize three-dimensional objects and the like contained in the image information by performing calculations of a deep learning algorithm using a deep neural network (DNN), for example.

[0026] The grid map generation unit 303 creates a grid map of a predetermined size (for example, 10cm x 10cm for each cell in a 20m x 20m area) based on the image data of the 3D point cloud. This is done to reduce the size of the data, as the amount of data in the 3D point cloud is large and real-time processing is difficult. The grid map is composed of, for example, a grid map showing the difference between the maximum and minimum heights of the point cloud within the grid (representing whether the cell is a step) and a grid map showing the maximum height of the point cloud within the grid from a reference point (representing the terrain shape of the cell). Furthermore, the grid map generation unit 303 removes spike noise and white noise contained in the generated grid map, detects obstacles with a height above a predetermined level, and generates an occupied grid map showing whether or not there are three-dimensional objects that act as obstacles for each grid.

[0027] The route generation unit 304 generates a travel path for the mobile body 100 to the target position set by the user instruction acquisition unit 301. Specifically, the route generation unit 304 generates a route using an occupied grid map generated by the grid map generation unit 303 from the image captured by the detection unit 114, without requiring obstacle information from a high-precision map. Since the detection unit 114 is a stereo camera that captures the area in front of the mobile body 100, it cannot recognize obstacles in other directions. Therefore, it is desirable for the mobile body 100 to store detected obstacle information for a predetermined period in order to avoid collisions with obstacles outside the field of view and getting stuck in dead ends. This allows the mobile body 100 to generate a route considering both previously detected obstacles and obstacles detected in real time.

[0028] Furthermore, the path generation unit 304 periodically generates global paths using an occupied grid map, and then periodically generates local paths that follow the global paths. In other words, the target position of the local path is determined by the global path. In this embodiment, the generation period for each path is set to 100 ms for the global path and 50 ms for the local path, but this does not limit the present invention. Various algorithms are known for generating global paths, such as RRT (Rapid-exploring Random Tree), PRM (Probabilistic Road Map), and A*. Also, since a differential two-wheeled mobility vehicle with a tail wheel is used as the mobile body 100, the path generation unit 304 generates local paths that take into account the tail wheel 121, which is the driven wheel.

[0029] The travel control unit 305 controls the movement of the mobile body 100 according to the local path. Specifically, the travel control unit 305 controls the speed and angular velocity of the mobile body 100 by controlling the travel unit 112 according to the local path. Furthermore, the travel control unit 305 controls the movement in response to various operations of the driver. If a deviation occurs in the driving plan of the local path due to the driver's operation, the travel control unit 305 may acquire a new local path generated again by the path generation unit 304 and control the movement, or it may control the speed and angular velocity of the mobile body 100 to eliminate the deviation from the local path in use.

[0030] <Occupation grid map> Figure 4 shows an occupied grid map 400 containing obstacle information according to this embodiment. In this embodiment, the mobile body 100 travels without relying on obstacle information from a high-precision map, so all obstacle information is obtained from the recognition results of the detection unit 114. At this time, it is necessary to store obstacle information to avoid collisions with obstacles outside the field of view and getting stuck in dead ends. Therefore, in this embodiment, an occupied grid map is used as a method for storing obstacle information from the viewpoint of reducing the amount of information in the 3D point cloud of stereo images and ease of handling in path planning.

[0031] The grid map generation unit 303 according to this embodiment divides the area surrounding the mobile body 100 into a grid and generates an occupied grid map for each grid (divided area) that includes information indicating the presence or absence of obstacles. Here, an example is described in which a predetermined area is divided into a grid, but instead of dividing it into a grid, it may be divided into other shapes, and an occupied map indicating the presence or absence of obstacles may be created for each divided area. The occupied grid map 400 uses an area of ​​size, for example, 40m x 40m or 20m x 20m, around the mobile body 100 as the surrounding area, and divides this area into grids of 20cm x 20cm or 10cm x 10cm, which are dynamically set according to the movement of the mobile body 100. In other words, the occupied grid map 400 is an area that is always shifted so that the mobile body 100 is at the center in accordance with the movement of the mobile body 100, and changes in real time. The size of the area can be arbitrarily set according to the hardware resources of the mobile body 100.

[0032] Furthermore, the occupied grid map 400 defines information about the presence or absence of obstacles detected from the images captured by the detection unit 114 for each grid. For example, the presence or absence information is defined as "0" for a drivable area and "1" for an impassable area (i.e., with an obstacle). In Figure 4, 401 indicates a grid where an obstacle exists. The area where an obstacle exists indicates an area that the mobile body 100 cannot pass through, and is composed of, for example, three-dimensional objects of 5 cm or more. Therefore, the mobile body 100 generates a path that avoids these obstacles 401.

[0033] <Accumulation of obstacle information> Figure 5 illustrates the accumulation of obstacle information in the occupied grid map according to this embodiment. 500 represents a local map that moves in accordance with the movement of the moving body 100. The local map 500 is shifted according to the movement of the moving body 100 in the x-axis and y-axis directions on the grid map. The local map 500 shows, for example, that the dotted line area 501 is deleted and the solid line area 502 is added according to the amount Δx of movement of the moving body 100 in the x-axis direction. The area to be deleted is the area opposite to the direction of movement of the moving body 100, and the area to be added is the area in the direction of movement. Similarly, areas are deleted and added in the y-axis direction according to the movement of the moving body 100.

[0034] Furthermore, the local map 500 stores information on obstacles detected in the past. If an obstacle exists in a grid included in the deletion area, the information on that obstacle is deleted from the local map 500, but it is desirable to retain it separately from the local map 500 for a certain period. Such information is useful, for example, if the moving object 100 changes its course and the deletion area is again included in the local map 500, thereby improving the accuracy of the moving object 100's obstacle avoidance. Additionally, by utilizing the stored information, it is not necessary to detect obstacles again, thus reducing the processing load.

[0035] Furthermore, before the local map 500 is added to the obstacle detection map 510 described later, it undergoes a forgetting process according to a forgetting rate set for each grid. When a dynamic obstacle that moves is detected, if the obstacle information previously detected and accumulated in the grid is retained as is, a false detection may occur where the obstacle is found to exist in all grids along the obstacle's movement trajectory. Therefore, in order to avoid mistakenly determining that an obstacle exists in a grid that the obstacle has already passed through, it is necessary to forget the accumulated obstacle information after a certain period of time. The forgetting of accumulated obstacle information will be described later using Figure 6.

[0036] 510 shows an obstacle detection map that displays detection information of obstacles present in the area in front of the moving body 100, based on the captured image taken by the detection unit 114 of the moving body 100. The obstacle detection map 510 displays real-time information and is periodically generated according to the captured image acquired from the detection unit 114. Since moving obstacles such as people and vehicles are also expected, it is desirable that the obstacle detection map 510, which is within the field of view 511 of the detection unit 114 in the area in front of the moving body 100, be updated with periodically generated obstacle detection maps rather than fixedly accumulating previously detected obstacles. This makes it possible to recognize moving obstacles as well and prevents the generation of routes that take unnecessary detours. On the other hand, in the area behind the moving body 100 (strictly speaking, outside the field of view of the detection unit 114), information on previously detected obstacles is accumulated, as shown in the local map 500. This makes it possible, for example, to easily generate a route that avoids collisions with obstacles that have been passed when an obstacle is detected in the area in front and a detour is generated.

[0037] 520 shows the occupied grid map generated by adding the local map 500 and the obstacle detection map 510. In this way, the occupied grid map 520 is generated as a grid map that combines the local map and obstacle detection information, which change in real time, with obstacle information that has been detected and stored in the past.

[0038] <Forgetting Obstacle Information> Figure 6 illustrates the forgetting of obstacle information in the occupied grid map according to this embodiment. 600 shows the occupied grid map including generated obstacle information 603 and 604 around the moving object 100. 601 shows the actual field of view (performance) of the detection unit 114. In other words, the field of view 601 indicates the imaging range determined by the performance of the detection unit 114, such as a stereo camera. 602 shows the range within the field of view defined as being narrower than the actual field of view 601 when setting the obstacle information forgetting rate. This reduces false detection of obstacles even when the detection accuracy of obstacles in the edge regions of the captured image decreases.

[0039] In this embodiment, a forgetting rate is set individually for each grid. Here, the forgetting rate indicates how long the accumulated obstacle information is retained. For example, according to this embodiment, the occupied grid map is generated periodically, and the forgetting rate indicates how many periods the obstacle information is stored over.

[0040] Specifically, in this embodiment, the forgetting rate for forgetting accumulated obstacle information is set to different values ​​for each grid included in the field of view range 602 and for grids not included in the field of view range 602. For example, a first forgetting rate is set for grids that overlap with the field of view range 602 as shown in Figure 6, and a second forgetting rate lower than the first forgetting rate is set for the other grids. Here, a grid that overlaps with the field of view range 602 is a grid in which a predetermined area or more within the grid overlaps with the field of view range 602. The size of the predetermined area is arbitrary and can be set, for example, from 1 to 100%.

[0041] As described above, according to this embodiment, the forgetting rate is set high for the grid within the field of view of the detection unit 114, and low for other areas. By setting it this way, obstacle information can be quickly forgotten within the field of view, making it possible to counter dynamic obstacles that move. On the other hand, outside the field of view, by storing obstacle information for a certain period of time, collisions with obstacles and getting stuck in dead ends can be avoided. However, in order to reduce the computational load of path generation, it is desirable to retain obstacle information in a predetermined peripheral area of ​​the moving body 100 and forget the information in other areas. Therefore, according to this embodiment, the forgetting rate is also set outside the field of view.

[0042] Here, we will describe an example of setting different forgetting rates for areas within and outside the field of view, but it is also possible to set them individually within each area. For example, the forgetting rate may be set to increase as the distance from the moving object 100 increases outside the field of view. It is also possible to identify the type of obstacle and change the forgetting rate according to the identified type. The types of obstacles include at least dynamic obstacles that move and static obstacles that do not move. Dynamic obstacles are, for example, vehicles and other traffic participants. Static obstacles are, for example, obstacles that do not move on their own, and include fixed objects such as traffic lights and posts, as well as movable objects such as tables and desks that are not fixed. Furthermore, in areas of the captured image used for obstacle detection that contain shadows, the forgetting rate may be set higher compared to other areas. This reduces the impact of falsely detected obstacle information even if the detection accuracy of obstacles in areas containing shadows is low. In addition, if there are many obstacles in the grid map, or if obstacles occupy a large proportion, the forgetting rate may be set higher. This prevents path generation from being severely hindered by information about obstacles detected in the past. The specific forgetting process will be described later using Figure 9.

[0043] <Path generation> Figure 7 shows the travel path generated by the mobile body 100 according to this embodiment. The path generation unit 304 according to this embodiment periodically generates a global path 702 using an occupied grid map according to a set target position 701, and further periodically generates a local path 703 that follows the global path.

[0044] The target location 701 is set based on various instructions. These include instructions from an occupant riding in the mobile unit 100 and instructions from a user outside the mobile unit 100. Instructions from the occupant are given via the operation panel 131 or the voice input device 133. Instructions via the operation panel 131 may also be given by specifying a predetermined grid on a grid map displayed on the operation panel 131. In this case, the size of each grid may be set to be larger, allowing selection from a wider area of ​​the map. Instructions via the voice input device 133 may also be given using nearby landmarks as markers. These landmarks may include pedestrians included in the spoken information, signs, billboards, equipment installed outdoors such as vending machines, building components such as windows and entrances, roads, vehicles, motorcycles, etc. Upon receiving instructions via the voice input device 133, the route generation unit 304 detects the specified landmark from the captured image acquired by the detection unit 114 and sets it as the target location.

[0045] These speech and image recognition functions utilize machine learning models. These models employ deep learning algorithms, such as deep neural networks (DNNs), to recognize place names, landmark names (including buildings), store names, and object names contained in speech and image information. A DNN for speech recognition becomes trained after completing the training phase, and can then perform recognition processing (inference phase) on new speech information by inputting it into the trained DNN. Similarly, a DNN for image recognition can recognize pedestrians, signs, billboards, outdoor equipment such as vending machines, building components like windows and entrances, roads, vehicles, and motorcycles within images.

[0046] Furthermore, instructions from users outside the mobile unit 100 can be transmitted to the mobile unit 100 via the communication unit 136 using the user's own communication device 140, or the mobile unit 100 can be summoned by actions such as raising a hand towards the mobile unit 100, as shown in Figure 7. Instructions via the communication device 140 are given via operation input or voice input, similar to instructions from the crew.

[0047] Once the target position 701 is set, the path generation unit 304 generates a global path 702 using the generated occupied grid map. As mentioned above, various algorithms such as RRT, PRM, and A* are known for generating global paths, but any method may be used. Subsequently, the path generation unit 304 generates local paths 703 that follow the generated global path 702. Various methods exist for local path planning, such as DWA (Dynamic Window Approach), MPC (Model Predictive Control), clothoid tentacles, and PID (Proportional-Integral-Differential) control.

[0048] <Basic control of mobile vehicles> Figure 8 is a flowchart showing the basic control of the mobile body 100 according to this embodiment. The processes described below are realized in the control unit 130 by, for example, the CPU reading a program stored in memory such as ROM into RAM and executing it.

[0049] In S101, the control unit 130 sets the target position of the moving object 100 based on the user instruction received by the user instruction acquisition unit 301. As described above, the user instruction can be received in various ways. Next, in S102, the control unit 130 uses the detection unit 114 to image the area in front of the moving object 100 and acquires the image. The acquired image is processed by the image information processing unit 302 to create a depth image and convert it into a 3D point cloud. In S103, the control unit 130 detects obstacles that are, for example, three-dimensional objects of 5 cm or more from the 3D point cloud image. In S104, the control unit 130 generates an occupied grid map of a predetermined area centered on the moving object 100, according to the detected obstacles and the position information of the moving object 100. A detailed method will be explained using Figure 9.

[0050] Next, in S105, the control unit 130 generates a travel path for the mobile body 100 using the path generation unit 304. As described above, the path generation unit 304 generates a global path using the occupied grid map and then generates a local path according to the generated global path. Subsequently, in S106, the control unit 130 determines the speed and angular velocity of the mobile body 100 according to the generated local path and controls its movement. After that, in S107, the control unit 130 determines whether the mobile body 100 has reached the target position based on the position information from the GNSS sensor 134. If it has not reached the target position, the process returns to S102 and repeatedly generates a path while updating the occupied grid map and controls its movement. On the other hand, if it has reached the target position, the process in this flowchart ends.

[0051] <Method for generating occupied grid maps (with forgetting control)> Figure 9 is a flowchart showing the detailed processing procedure for the generation control of the occupied grid map (S104) according to this embodiment. The processing described below is realized in the control unit 130 by, for example, the CPU reading a program stored in memory such as ROM into RAM and executing it.

[0052] First, in S201, the control unit 130 obtains the movement speed of the mobile body 100 from the travel unit 112. Next, in S202, the control unit 130 determines whether the obtained movement speed of the mobile body 100 is 0, that is, whether the mobile body 100 is in a stopped state. If the movement speed is 0, the process proceeds to S203; otherwise, it proceeds to S204.

[0053] In S203, if the mobile body 100 is stationary, the control unit 130 sets the forgetting rate k_out for outside the field of view to 0 to prevent the accumulated obstacle information from being forgotten, and proceeds to S205. On the other hand, in S204, if the mobile body 100 is moving (driving), the control unit 130 sets the forgetting rate for outside the field of view to a default value, and proceeds to S205. The forgetting rate k_in within the field of view is set to a predetermined value regardless of the mobile body 100's speed.

[0054] An example of a method for determining the forgetting rate is described below. The forgetting rate is determined based on the following formulas (1) and (2). Accumulated value max = k_in / dt * dynamic_object_forget_time ... Formula (1) k_out = accumulation max*dt / out_of_fov_forget_time...Formula (2) Here, "k_in" represents the forgetting rate within the field of view. "k_out" represents the forgetting rate outside the field of view. "dynamic_object_forget_time" represents the forgetting time within the field of view. "out_of_fov_forget_time" represents the forgetting time outside the field of view. "dt" represents the period. By determining the forgetting rate using the above formulas (1) and (2), obstacle information can be updated quickly within the field of view, while past obstacle information outside the field of view can be emphasized.

[0055] In S205, the control unit 130 forgets the accumulated information of obstacles. This forgetting process is performed using the following formula (3). map = map - k_in*fovmap - k_out (1 - fovmap)...Formula (3) Here, "fovmap" represents the grid map within the field of view. For example, "fovmap" shows grids within the field of view as "1" and grids outside the field of view as "0". According to equation (3), the forgetting process is performed by changing the forgetting rate inside and outside the field of view, as described above. When the forgetting process is performed, a local map 500, as shown in Figure 5, is generated.

[0056] Next, in S206, the control unit 130 acquires the obstacle information detected in S103 (for example, the obstacle detection map 510 shown in Figure 5). Subsequently, in S207, the control unit 130 generates an updated occupancy grid map by adding the map that underwent the forgetting process in S205 and the obstacle detection map acquired in S206 using the following formula (4). Map = map+k_acc*new_map...Formula (4) Here, "Map" refers to the occupied grid map with updated obstacle information, "map" is the obstacle detection map that underwent the forgetting process in S205 above, and "new_map" refers to the newly detected obstacle detection map obtained in S206 above. "k_acc" indicates the accumulation coefficient. This coefficient is set so that it does not exceed the accumulation value max. Once processing in S207 is complete, the process in this flowchart ends and the process proceeds to S105. Note that processing in S201 through S207 is performed periodically, for example, at a frequency of 10 Hz.

[0057] <Summary of Embodiments> 1. The mobile control system (e.g., 100) of the above embodiment is: A storage means for storing information on previously detected obstacles in each divided region obtained by dividing the surrounding area of ​​the moving object, and (303) A forgetting means for forgetting information about obstacles stored by the storage means according to a predetermined forgetting rate, and (303) An imaging means for acquiring captured images, and (114) A detection means for detecting obstacles included in the captured image, and (130, 302, 303) (303) A map generation means generates an occupancy map showing the occupancy of obstacles for each divided region, according to the information on obstacles stored by the storage means and the obstacles detected by the detection means, for the current surrounding area of ​​the moving body. It is characterized by having the following features.

[0058] According to this embodiment, the forgetting rate of information on obstacles detected around the moving object is suitably set for each region. As a result, according to the present invention, it is possible to take measures against dynamic obstacles, and it is also possible to effectively utilize information on obstacles detected in the past to avoid collisions with those obstacles and getting stuck in dead ends.

[0059] 2. In the mobile control system of the above embodiment, the predetermined forgetting rate differs between the field of view of the imaging means and the field of view (Figure 6).

[0060] According to this embodiment, by switching the forgetting rate between inside and outside the field of view, it is possible to update obstacle information while effectively utilizing previously detected obstacle information.

[0061] 3. In the mobile control system of the above embodiment, the forgetting rate within the field of view is higher than the forgetting rate outside the field of view (Figure 6).

[0062] According to this embodiment, within the field of view, obstacle information is updated more quickly to counter moving obstacles, and outside the field of view, past detection information is retained for a certain period of time to avoid collisions with obstacles and getting stuck in dead ends.

[0063] 4. In the mobile body control system of the above embodiment, the area within the field of view is set to be narrower than the imaging range of the imaging means, which is determined by the performance of the imaging means (601, Figure 6).

[0064] According to this embodiment, it is possible to reduce false detection of obstacles due to the detection accuracy of the edge regions in the captured image.

[0065] 5. The mobile control system of the above embodiment further includes a determination means for determining the type of the detected obstacle, The predetermined forgetting rate is set for each type of obstacle.

[0066] According to this embodiment, previously detected obstacle information can be effectively utilized according to the type of obstacle.

[0067] 6. In the mobile object control system of the above embodiment, the types of obstacles include at least dynamic obstacles that move and static obstacles that do not move.

[0068] According to this embodiment, the forgetting rate can be suitably set depending on whether there is a dynamic or static obstacle.

[0069] 7. In the mobile control system of the above embodiment, the forgetting rate set for the dynamic obstacle is higher than the forgetting rate set for the static obstacle.

[0070] According to this embodiment, dynamic obstacles are quickly forgotten because they move, and static obstacles do not move on their own, so past detection information is retained for a certain period of time, thereby preventing collisions with obstacles and getting stuck in dead ends.

[0071] 8. In the mobile object control system of the above embodiment, the dynamic obstacle is a vehicle or other traffic participant.

[0072] According to this embodiment, by identifying traffic participants as dynamic obstacles, the path of moving objects can be suitably generated.

[0073] 9. In the above embodiment of the mobile body control system, the static obstacle is an obstacle that does not move on its own.

[0074] According to this embodiment, by identifying obstacles fixed as static obstacles, a suitable path for a moving object can be generated.

[0075] 10. In the mobile body control system of the above embodiment, the forgetting rate of the region containing a shadow in the captured image is higher than the forgetting rate of the region not containing a shadow in the captured image.

[0076] According to this embodiment, areas containing shadows, which reduce the accuracy of obstacle detection, can be forgotten earlier, thereby reducing the impact of false detections.

[0077] 11. In the mobile control system of the above embodiment, the forgetting rate is set to be higher the greater the number of obstacles.

[0078] According to this embodiment, it is possible to avoid the extreme obstruction of path generation caused by obstacles detected in the past.

[0079] 12. In the mobile control system of the above embodiment, the forgetting rate is set to be higher the larger the number of divided regions occupied by the obstacle.

[0080] According to this embodiment, it is possible to avoid the extreme obstruction of path generation caused by obstacles detected in the past.

[0081] 13. In the mobile body control system of the above embodiment, the map generation means divides the area around the mobile body into a grid and generates an occupation grid map as the occupation map, which shows the occupation of obstacles detected by the detection means for each grid.

[0082] According to this embodiment, a predetermined planar region can be easily divided in the x and y directions, and the predetermined range can be covered without any omissions.

[0083] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention.

[0084] This application claims priority based on Japanese Patent Application No. 2022-158638, filed on September 30, 2022, and all of its contents are incorporated herein by reference.

Claims

1. A mobile control system, A storage means for storing information on previously detected obstacles in each divided region obtained by dividing the surrounding area of ​​a moving object, A forgetting means for forgetting information about obstacles stored by the storage means according to a predetermined forgetting rate assigned to each divided region, An imaging means for acquiring captured images, A detection means for detecting obstacles included in the captured image, A map generation means generates an occupancy map showing the occupancy of obstacles for each divided region, according to the information on obstacles stored by the storage means and the obstacles detected by the detection means, for the current surrounding area of ​​the moving body. Equipped with, A mobile body control system characterized in that the predetermined forgetting rate is set according to whether or not the divided region is currently observable by the imaging means.

2. The mobile body control system according to claim 1, characterized in that the predetermined forgetting rate differs between the field of view of the imaging means and the field of view outside of the field of view.

3. The mobile body control system according to claim 2, characterized in that the forgetting rate within the field of view is higher than the forgetting rate outside the field of view.

4. The mobile body control system according to claim 2, characterized in that the area within the field of view is set to be narrower than the imaging range of the imaging means, which is determined by the performance of the imaging means.

5. The system further includes a determination means for determining the type of the detected obstacle, The mobile body control system according to claim 1, characterized in that the predetermined forgetting rate is set for each type of obstacle.

6. The mobile body control system according to claim 5, characterized in that the types of obstacles include at least dynamic obstacles that move and static obstacles that do not move.

7. The mobile body control system according to claim 6, characterized in that the forgetting rate set for the dynamic obstacle is higher than the forgetting rate set for the static obstacle.

8. The mobile body control system according to claim 6, characterized in that the aforementioned dynamic obstacle is a vehicle or another traffic participant.

9. The mobile body control system according to claim 6, characterized in that the static obstacle is an obstacle that does not move on its own.

10. The mobile body control system according to claim 1, characterized in that the forgetting rate of the region containing a shadow in the captured image is higher than the forgetting rate of the region not containing a shadow in the captured image.

11. The mobile body control system according to claim 1, characterized in that the forgetting rate is set to be higher the greater the number of obstacles.

12. The mobile body control system according to claim 1, characterized in that the forgetting rate is set to be higher the larger the number of divided regions occupied by the obstacle.

13. The mobile body control system according to claim 1, characterized in that the map generation means divides the area around the mobile body into a grid and generates an occupation grid map as the occupation map, which shows the occupation of obstacles detected by the detection means for each grid.

14. A control method for a mobile control system, A storage process is performed to accumulate information on previously detected obstacles for each divided region obtained by dividing the surrounding area of ​​the moving object, A forgetting step is performed to forget the information about obstacles accumulated in the accumulation step according to a predetermined forgetting rate assigned to each divided region. An imaging step in which an image is acquired by an imaging means, A detection step for detecting obstacles included in the captured image, A map generation step generates an occupancy map showing the occupancy of obstacles for each divided region, based on the information about obstacles accumulated in the accumulation step and the obstacles detected in the detection step, for the current surrounding area of ​​the moving object. Includes, A control method for a mobile object control system, characterized in that the predetermined forgetting rate is set according to whether or not the divided region is currently observable by the imaging means.

15. A program for causing a computer to function as one of the means of a mobile control system according to any one of claims 1 to 13.

16. It is a mobile object, For each divided region obtained by dividing the surrounding area of ​​the moving body, there is a storage means for accumulating information on obstacles previously detected, A forgetting means for forgetting information about obstacles stored by the storage means according to a predetermined forgetting rate assigned to each divided region, An imaging means for acquiring captured images, A detection means for detecting obstacles included in the captured image, A map generation means generates an occupancy map showing the occupancy of obstacles for each divided region, according to the information on obstacles stored by the storage means and the obstacles detected by the detection means, for the current surrounding area of ​​the moving body. Equipped with, The predetermined forgetting rate is set according to whether or not the divided region is currently observable by the imaging means.

Images