Obstacle detection device, control device, obstacle detection method, control method, and program

Abstract

To maintain continuity of identification numbers of detected obstacles even when a plurality of obstacles are close to each other.SOLUTION: The obstacle detection device includes a depth information acquisition unit that acquires depth information of a traveling path of the mobile robot based on sensor data, an obstacle detection unit that detects one or more obstacles existing on the traveling path based on the depth information and determines whether an attribute of the detected obstacle is a dynamic obstacle or a static obstacle, a combination determination unit that determines whether the obstacle detected as one obstacle includes two or more obstacles, a separation unit that separates at least one obstacle from the obstacle determined to include the two or more obstacles based on the detected obstacle attribute and a past detection result of the obstacle, and an obstacle management unit that manages an obstacle identification number based on a first obstacle list including the detected obstacle information and a second obstacle list including the detected obstacle attribute.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an obstacle detection device, a control device, an obstacle detection method, a control method, and a program. [Background technology]

[0002] As a related technique, Patent Document 1 discloses a control device for a self-propelled mobile body. In Patent Document 1, the self-propelled mobile body moves toward a target position while avoiding obstacles. An obstacle sensor such as a Positioning Sensitive Detector (PSD) camera is installed on the ceiling surface of the area in which the self-propelled mobile body travels. The obstacle sensor sequentially detects the position of obstacles moving on the plane in which the self-propelled mobile body moves. Based on the position information of the obstacles, the control device generates a stochastic potential field that represents the probability that the obstacle may exist. Based on the generated stochastic potential field, the control device searches for a route for the self-propelled mobile body to move toward the target position and moves the mobile body along the searched route.

[0003] As another related technique, Patent Document 2 discloses a collision avoidance system for work vehicles. In Patent Document 2, a vehicle such as a tractor has an obstacle detection unit. The obstacle detection unit includes a Light Detection and Ranging (LiDAR) sensor. The LiDAR sensor emits measurement light from the vehicle body toward a group of ranging points present within a predetermined measurement range and receives light reflected from the measurement light. The obstacle detection unit measures multiple distance values ​​for each ranging point based on the measurement light and the reflected light. The obstacle detection unit determines the presence or absence of an obstacle based on measurement information including the multiple measured distance values.

[0004] In Patent Document 2, the obstacle detection unit uses a grid map to identify the position of an obstacle. The grid map has a number of grids obtained by dividing the measurement range of the LiDAR sensor at a predetermined resolution. The obstacle detection unit identifies the highest height information in each grid from the coordinates of multiple measurement points in the up-down direction of the tractor, which are included in the point cloud information of each grid, and registers each identified height information as height information for that grid.

[0005] The obstacle detection unit extracts feature points for identifying obstacles. The obstacle detection unit also extracts candidate grids in which point cloud information that may indicate an obstacle is registered. Of the candidate grids, the obstacle detection unit identifies candidate grids in which point cloud information having feature points for identifying obstacles exists as obstacle grids. The obstacle detection unit groups consecutive obstacle grids from the identified multiple obstacle grids as a single obstacle. If the difference between the center position of the obstacle grid group in the past three steps and the center position of the current obstacle grid group is less than a predetermined threshold, the obstacle detection unit determines that the obstacle detected in the past and the obstacle detected in the current frame are the same obstacle. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-241836 [Patent Document 2] Patent No. 7402608 Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 2, consecutive obstacle grid groups are grouped as a single obstacle. In this case, if multiple obstacles are close to each other, they may be grouped as a single obstacle. In other words, in the obstacle detection results, multiple obstacles may be combined into a single obstacle. When multiple obstacles are combined into a single obstacle, the center position of the obstacle may change significantly before and after the combination. When the center position of the obstacle changes significantly, the obstacle detected by the combination may be detected as a new obstacle different from obstacles previously detected. When detected as a new obstacle, the continuity of the identification number of the detected obstacle cannot be maintained before and after the combination.

[0008] One of the purposes of the present disclosure is to provide an obstacle detection device, a control device, an obstacle detection method, a control method, and a program that can maintain the continuity of the identification numbers of detected obstacles even when multiple obstacles are in close proximity. [Means for solving the problem]

[0009] An obstacle detection device according to a first aspect of the present disclosure includes: a depth information acquisition unit that acquires depth information of a travel path of a mobile robot based on sensor data from a sensor that senses the travel path; an obstacle detection unit that detects one or more obstacles present on the travel path based on the depth information and determines whether the attributes of the detected obstacles are dynamic or static; a combination determination unit that determines whether an obstacle detected as a single obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; a separation unit that separates at least one obstacle from obstacles determined to include two or more obstacles using the attributes of the detected obstacle and the attributes in the detection results of the obstacles detected in the past; and an obstacle management unit that manages identification numbers of the obstacles based on a first obstacle list including information of the detected obstacles and a second obstacle list including information of obstacles that have already been detected.

[0010] A control device according to a second aspect of the present disclosure includes the obstacle detection device described above and a control unit that controls the mobile robot based on information about obstacles included in the second obstacle list.

[0011] An obstacle detection method according to a third aspect of the present disclosure includes: acquiring depth information of a travel path of a mobile robot based on sensor data from a sensor that senses the travel path; detecting one or more obstacles present on the travel path based on the depth information; determining whether the attributes of the detected obstacles indicate a dynamic obstacle or a static obstacle; determining whether the obstacle detected as a single obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; separating at least one obstacle from the obstacles determined to include two or more obstacles using the attributes of the detected obstacle and the attributes in the detection results of the obstacles detected in the past; and managing identification numbers of the obstacles based on a first obstacle list including information of the detected obstacles and a second obstacle list including information of obstacles that have already been detected.

[0012] A control method according to a fourth aspect of the present disclosure includes: acquiring depth information of a travel path of a mobile robot based on sensor data from a sensor that senses the travel path; detecting one or more obstacles present on the travel path based on the depth information; determining whether the attributes of the detected obstacles are dynamic or static; determining whether the obstacles detected as a single obstacle include two or more obstacles based on the detected obstacles and detection results of obstacles detected in the past; separating at least one obstacle from the obstacles determined to include two or more obstacles using the attributes of the detected obstacles and the attributes in the detection results of the obstacles detected in the past; managing identification numbers of the obstacles based on a first obstacle list including information of the detected obstacles and a second obstacle list including information of obstacles that have already been detected; and controlling the mobile robot based on the information of the obstacles included in the second obstacle list.

[0013] A program according to a fifth aspect of the present disclosure causes a computer to perform processing including acquiring depth information of a travel path of a mobile robot based on sensor data from a sensor that senses the travel path; detecting one or more obstacles present on the travel path based on the depth information; determining whether the attributes of the detected obstacles are dynamic or static; determining whether an obstacle detected as a single obstacle includes two or more obstacles based on the detected obstacles and attributes in the detection results of obstacles detected in the past; separating at least one obstacle from the obstacles determined to include two or more obstacles using the attributes of the detected obstacles and attributes in the detection results of the obstacles detected in the past; and managing identification numbers of the obstacles based on a first obstacle list including information of the detected obstacles and a second obstacle list including information of obstacles that have already been detected.

[0014] A program according to a sixth aspect of the present disclosure causes a computer to perform processing including acquiring depth information of a travel path of a mobile robot based on sensor data from a sensor that senses the travel path; detecting one or more obstacles present on the travel path based on the depth information; determining whether the attributes of the detected obstacles are dynamic or static; determining whether the obstacles detected as a single obstacle include two or more obstacles based on the detected obstacles and detection results of obstacles detected in the past; separating at least one obstacle from the obstacles determined to include two or more obstacles using the attributes of the detected obstacles and the attributes in the detection results of the obstacles detected in the past; managing identification numbers of the obstacles based on a first obstacle list including information of the detected obstacles and a second obstacle list including information of obstacles that have already been detected; and controlling the mobile robot based on information of the obstacles included in the second obstacle list. [Effects of the Invention]

[0015] The obstacle detection device, control device, obstacle detection method, control method, and program according to the present disclosure can maintain continuity of the identification numbers of detected obstacles even when multiple obstacles are in close proximity. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 2 is a block diagram illustrating a schematic configuration example of a control device according to the present disclosure. [Figure 2] 1 is a schematic diagram illustrating an example configuration of a system including a control device according to the present disclosure. [Figure 3] FIG. 2 is a block diagram illustrating an example of the configuration of a control server. [Figure 4] FIG. 2 is a block diagram showing an example of the configuration of an obstacle detection unit. [Figure 5] FIG. 10 is a diagram showing a specific example of a list of obstacles detected by an obstacle detection unit. [Figure 6] 10 shows a list of detected obstacles managed by an obstacle management unit. [Figure 7] 10 is a flowchart showing an operation procedure of the control server. [Figure 8] 10 is a flowchart showing the operation procedure of a combination check process. [Figure 9] 10 is a flowchart showing the operation procedure of a separation process. [Figure 10] FIG. 10 is a diagram showing obstacles detected in a frame and the frame immediately before the frame. [Figure 11] FIG. 10 is a diagram showing a grid map of an obstacle detected as one obstacle in a certain frame. [Figure 12] FIG. 10 is a diagram showing a grid map of obstacles detected in a frame immediately before a given frame. [Figure 13] FIG. 1 is a block diagram illustrating an example of the configuration of a computer device. DETAILED DESCRIPTION OF THE INVENTION

[0017] Prior to describing embodiments of the present disclosure, an overview of the present disclosure will be described. Fig. 1 is a block diagram showing a schematic configuration example of a control device according to the present disclosure. The control device 10 shown in Fig. 1 includes a depth information acquisition unit 11, an obstacle detection unit 12, a joining determination unit 13, a separation unit 14, an obstacle management unit 15, and a control unit 16. In the control device 10, the depth information acquisition unit 11, the obstacle detection unit 12, the joining determination unit 13, the separation unit 14, and the obstacle management unit 15 correspond to an obstacle detection device 20.

[0018] The depth information acquisition unit 11 acquires depth information of the traveling path of the mobile robot based on sensor data from a sensor that senses the traveling path of the mobile robot. The obstacle detection unit 12 detects one or more obstacles that exist on the traveling path of the mobile robot based on the acquired depth information. The obstacle detection unit 12 also determines whether the attribute of the detected obstacle is a moving obstacle or a static obstacle.

[0019] The combination determination unit 13 determines whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and the detection results of obstacles detected in the past. The separation unit 14 separates at least one obstacle from the obstacles determined to include two or more obstacles, using the attributes of the detected obstacle and the attributes in the detection results of obstacles detected in the past.

[0020] The obstacle management unit 15 manages the identification numbers of obstacles based on a first obstacle list containing information about detected obstacles and a second obstacle list containing information about obstacles that have already been detected. The control unit 16 controls the mobile robot based on the information about obstacles included in the second obstacle list.

[0021] In the present disclosure, the separation unit 14 can separate at least one obstacle from a set of obstacles determined to include two or more obstacles by using attributes of the obstacles determined to include two or more obstacles and attributes of detection results of obstacles individually detected in the past. By separating the obstacles using attributes, the obstacle management unit 15 can assign the same identification number to each obstacle even when several obstacles are close to each other. The control device 10 can, for example, allow the mobile robot to travel safely by controlling the mobile robot based on information about the obstacles detected by the obstacle detection device 20.

[0022] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. In addition, in the following drawings, the same or similar elements are designated by the same reference numerals, and duplicate explanations are omitted as necessary.

[0023] FIG. 2 is a schematic diagram showing an example configuration of a system including a control device according to the present disclosure. A first embodiment will be described using FIG. 2. In this embodiment, the control device is configured as a control server 110. The system 100 shown in FIG. 2 includes the control server 110, a sensor 130, and a mobile robot 150. Note that FIG. 2 illustrates one sensor 130 and one mobile robot 150. However, the number of sensors 130 and the number of mobile robots 150 are not limited to one each. The system 100 may include multiple sensors 130. The system 100 may also include multiple mobile robots 150.

[0024] The mobile robot 150 has, for example, a power source such as a motor, a drive mechanism, and a wireless communication device. The mobile robot 150 travels within a predetermined movement range 200. The movement range 200 is also called the travel path of the mobile robot 150. The mobile robot 150 is configured as a transport robot that transports luggage in a warehouse, for example. The mobile robot 150 is configured to be capable of wireless communication with the control server 110. The mobile robot 150 may be configured to be capable of autonomous movement within the movement range 200.

[0025] The sensor 130 is a sensor that senses the movement range of the mobile robot 150. The sensor 130 is installed, for example, on the ceiling of the movement range 200 of the mobile robot 150. The movement range 200 may be divided into multiple areas, and a sensor 130 may be installed in each divided area. The sensor 130 includes an imaging device that can acquire depth information. The sensor 130 is used to acquire image information and depth information. In the following description, an example in which the sensor 130 includes a stereo camera will be mainly described. The sensor 130 transmits sensor data, for example, image data from the stereo camera, to the control server 110.

[0026] The control server 110 acquires image data from the sensor 130. Based on the acquired image data, the control server 110 detects obstacles present in the movement range 200. Here, an obstacle means, for example, an object that may hinder the movement of the mobile robot 150. The control server 110 directly or indirectly controls the movement of the mobile robot 150 based on information about the detected obstacle. For example, the control server 110 controls the movement of the mobile robot 150 by transmitting information about the detected obstacle to the mobile robot 150. Alternatively, the control server 110 may generate movement path information for the mobile robot 150 according to the detected obstacle and transmit the movement path information to the mobile robot 150 to control the movement of the mobile robot 150.

[0027] 3 is a block diagram showing an example configuration of the control server 110. The control server 110 includes a depth information acquisition unit 111, an obstacle detection unit 112, a joining determination unit 113, a separation unit 114, an obstacle management unit 115, and an obstacle information transmission unit 116. In the control server 110, the depth information acquisition unit 111, the obstacle detection unit 112, the joining determination unit 113, the separation unit 114, and the obstacle management unit 115 configure an obstacle detection device 117. The control server 110 corresponds to the control device 10 shown in FIG. 1. The obstacle detection device 117 corresponds to the obstacle detection device 20 shown in FIG. 1.

[0028] The control server 110 may be physically configured using a computer device having one or more memories and one or more processors. In the control server 110, the one or more processors may execute processing in accordance with instructions read from the one or more memories, thereby realizing at least a portion of the functions of each unit in the control server 110. In this embodiment, the control server 110 does not necessarily have to be configured as a single computer device. The control server 110 may also be configured using multiple physically separated devices.

[0029] The depth information acquisition unit 111 acquires depth information of the movement range 200 of the mobile robot 150 based on image data acquired from the sensor 130. Here, the depth information indicates, for example, the height from the floor surface on which the mobile robot 150 moves. For example, in FIG. 2, when the mobile robot 150 moves on the xy plane, the depth information indicates the height in the z direction at each location in the movement range 200. The depth information acquisition unit 111 acquires, for example, point cloud values ​​(x, y, z) for pixels in the imaging area of ​​a stereo camera, and acquires depth information based on the acquired point cloud values. The depth information acquisition unit 111 may acquire image information including the brightness of each coordinate in addition to the depth information. The depth information acquisition unit 111 corresponds to the depth information acquisition unit 11 shown in FIG. 1.

[0030] The obstacle detection unit 112 detects one or more obstacles present in the movement range 200 based on the depth information acquired by the depth information acquisition unit 111. The obstacle detection unit 112 also determines whether the detected obstacle is a moving obstacle or a static obstacle. For example, the obstacle detection unit 112 generates a grid map obtained by dividing the movement range 200 into a plurality of grids. The obstacle detection unit 112 identifies, in the grid map, grids whose depth information is equal to or greater than a predetermined height as obstacle grids. For each grid, the obstacle detection unit 112 determines whether the attribute of the obstacle grid is a moving obstacle or a static obstacle. The obstacle detection unit 112 groups the identified obstacle grids into one or more obstacle grid groups and detects the grouped obstacle grid groups as obstacles. The obstacle detection unit 112 corresponds to the obstacle detection unit 12 shown in FIG. 1.

[0031] Fig. 4 is a block diagram showing an example configuration of the obstacle detection unit 112. In the example of Fig. 4, the obstacle detection unit 112 includes a grid map generation unit 121, a grid attribute determination unit 122, and a grouping unit 123. The grid map generation unit 121 obtains a representative value of the point cloud z values ​​of the corresponding pixel values ​​for each grid, i.e., for each area, in each of a plurality of areas obtained by dividing the movement range 200 into a grid. For each grid, the grid map generation unit 121 sets the representative value of the point cloud z values ​​as the depth information of the grid.

[0032] The grid map generation unit 121 may calculate, for each grid, the average value of the point cloud z values ​​of the pixels contained in the grid, and use the calculated average value as the depth information of the grid. Alternatively, the grid map generation unit 121 may use the maximum value of the point cloud z values ​​as the depth information of the grid. The number of pixels contained in one grid depends on the number of divided areas, i.e., the number of grids. The grid map generation unit 121 may calculate, for each grid, the difference in maximum brightness between the previous image information and the current image information, and use the difference in maximum brightness as the brightness difference of the grid.

[0033] The grid map generating unit 121 may perform a smoothing process on the depth information. For example, the grid map generating unit 121 may set a smoothing coefficient k (0≦k<1) and use (depth information in the immediately preceding frame)×k+(depth information in the current frame)×(1−k) as the depth information in the current frame. The grid map generating unit 121 may control the smoothing coefficient k according to a change in the image information of the road over time, for example, the brightness difference in each grid. Note that the “current frame” refers to the frame being processed, and does not necessarily refer to the frame at the time of processing execution or the latest frame.

[0034] For example, a grid having a brightness difference greater than a first threshold is considered to contain a dynamic object, i.e., a dynamic obstacle. In this case, the grid map generator 121 may set the smoothing coefficient to a relatively small value, for example, k=0.3. After setting the smoothing coefficient to k=0.3, the grid map generator 121 may maintain the smoothing coefficient at k=0.3 until a predetermined number of frames, for example, 90 frames, have elapsed.

[0035] On the other hand, a grid where the brightness difference is smaller than the second threshold is considered to contain a static object, i.e., a static obstacle. The second threshold may be smaller than the first threshold. If the brightness difference is smaller than the second threshold, the grid map generator 121 may set the smoothing coefficient k to a relatively large value, for example, k=0.99. If the brightness difference is between the first threshold and the second threshold, the grid map generator 121 may set the smoothing coefficient k to a predetermined value specified using a configuration file or the like.

[0036] When the depth information of a certain grid changes by more than a predetermined value toward the floor, the grid map generator 121 may set the smoothing coefficient k for that grid to a value smaller than the smoothing coefficient set when the temporal change in image information is equal to or smaller than a first threshold. For example, when the depth of a grid deepens, it is considered that an obstacle has passed through that grid and the depth has become floor height. For example, the grid map generator 121 may set the smoothing coefficient to a small value, such as k = 0.3, for a grid where a dynamic obstacle has passed and the height indicated by the depth information has changed to floor height, even if the brightness difference is lower than the first threshold. Setting the smoothing coefficient to a small value can weaken the smoothing of the grid where the dynamic obstacle has passed, thereby preventing the influence of the depth information when the obstacle was present from remaining for a long time.

[0037] The grid attribute determination unit 122 determines the grid attribute of each grid. Grid attributes include, for example, unknown, distorted, floor, and obstacle. For example, the grid attribute determination unit 122 determines the grid attribute of a grid in which no valid point cloud exists and the grid height indicates an invalid value as "unknown." The grid attribute determination unit 122 determines the grid attribute of a grid that is located at the edge of the imaging range and has significant distortion, such as a grid height that indicates a height deeper than the floor, as "distorted."

[0038] The grid attribute determination unit 122 determines the grid attribute of a grid whose grid height is equal to or greater than a predetermined height above the floor as "obstacle." Even if the grid attribute determination unit 122 has once determined an obstacle to be an "obstacle," if there are no other grids determined to be "obstacles" around the grid determined to be an "obstacle," the grid attribute determination unit 122 may consider the obstacle grid to be a false detection and correct the grid attribute from "obstacle" to another attribute, for example, "floor."

[0039] For a grid determined to be an obstacle grid, the grid attribute determination unit 122 determines whether the obstacle is a dynamic obstacle or a static obstacle using the determination result of the grid attribute in a past frame. For example, when the determination result of the grid attribute changes from "floor" to "obstacle," the grid attribute determination unit 122 may determine the obstacle as a "dynamic obstacle" until a predetermined number of frames, for example, 40 frames, have elapsed since the frame in which the grid attribute changed. If the grid attribute determination unit 122 has continuously determined the obstacle to be an obstacle, it may determine the obstacle as a "static obstacle."

[0040] The grouping unit 123 groups obstacle grids, i.e., grids determined as "obstacles" by the grid attribute determination unit 122, into obstacle grid groups. The grouping unit 123 performs a labeling process on the obstacle grids and assigns the same label number to each group of obstacle grids. The grouping unit 123 groups obstacle grids that have been assigned the same label into one obstacle grid group. For example, a union find method may be used for grouping.

[0041] For example, the grouping unit 123 tallies the number of obstacle grids determined to be dynamic obstacles and the number of obstacle grids determined to be static obstacles for a group of obstacle grids that have been assigned the same label. The grouping unit 123 calculates the ratio between the number of obstacle grids determined to be dynamic obstacles and the number of obstacle grids determined to be static obstacles, and determines the overall attribute of the grouped obstacle grid group based on the calculated ratio. For example, if the number of obstacle grids determined to be dynamic obstacles is greater than the number of obstacle grids determined to be static obstacles, the grouping unit 123 determines that the attribute of the grouped obstacle grid group as a whole is dynamic obstacles. On the other hand, if the number of obstacle grids determined to be static obstacles is greater than the number of obstacle grids determined to be dynamic obstacles, the grouping unit 123 determines that the attribute of the grouped obstacle grid group as a whole is static obstacles.

[0042] Here, when one obstacle approaches another obstacle, the two obstacles may be detected as a single obstacle due to the grouping described above. Furthermore, after two obstacles have been combined into a single obstacle, if one of the obstacles moves, the combined obstacle may separate into two obstacles and be detected as two obstacles again. When a combined obstacle separates into two obstacles and is detected as two obstacles, at least one of the separated obstacles may be detected as a different obstacle from the obstacle detected before the combination.

[0043] In this embodiment, the combination determination unit 113 determines whether an obstacle detected as a single obstacle by the obstacle detection unit 112 includes two or more obstacles based on the obstacle detected by the obstacle detection unit 112 and the detection results of obstacles detected in the past. For example, the combination determination unit 113 checks whether two or more obstacles are included in the detection results of obstacles detected in the past within the range of a circumscribing rectangle surrounding the detected obstacle. If, for example, the center coordinates of two or more obstacles in the past are included within the range of the circumscribing rectangle, the combination determination unit 113 determines that an obstacle detected as a single obstacle includes two or more obstacles in the past. The combination determination unit 113 corresponds to the combination determination unit 13 shown in FIG. 1.

[0044] The separation unit 114 separates at least one obstacle from obstacles previously determined to include two or more obstacles, using attributes of obstacles detected currently and in the past. For example, when the obstacles determined to include two or more obstacles are determined as static obstacles as a whole, the separation unit 114 determines whether to separate at least one obstacle from the obstacles previously determined to include two or more obstacles. When the obstacles determined to include two or more obstacles are determined as moving obstacles as a whole, the separation unit 114 calculates a change in the proportion of obstacles whose attributes are determined to be moving obstacles. Based on the calculated change, the separation unit 114 determines whether to separate at least one obstacle from the obstacles previously determined to include two or more obstacles. The proportion of obstacles whose attributes are determined to be moving obstacles indicates, for example, the proportion of the portion determined to be moving obstacles to all obstacles present in a predetermined range. When the change is equal to or less than a predetermined threshold, the separation unit 114 determines to separate at least one obstacle from the obstacles previously determined to include two or more obstacles.

[0045] The separation unit 114 calculates the sum of the number of obstacle grids whose attributes are determined to be dynamic and the number of obstacle grids whose attributes are determined to be static, for obstacles determined to include multiple obstacles. The separation unit 114 calculates the ratio of the number of obstacle grids whose attributes are determined to be dynamic to the calculated sum as a first ratio. Furthermore, the separation unit 114 extracts two or more obstacles that exist within a circumscribing rectangle that surrounds an obstacle determined to include multiple obstacles in a detection result of a previously detected obstacle. The separation unit 114 calculates the sum of the number of obstacle grids whose attributes are determined to be dynamic and the number of obstacle grids whose attributes are determined to be static, for the extracted obstacles. The separation unit 114 calculates the ratio of the number of obstacle grids whose attributes are determined to be dynamic to the calculated sum as a second ratio. The separation unit 114 calculates the difference between the first ratio and the second ratio as a change in the ratio of obstacles whose attributes are determined to be dynamic. If the absolute value of the difference is equal to or less than a predetermined threshold, the separator 114 determines to separate at least one obstacle from the obstacles determined to include two or more obstacles. The separator 114 corresponds to the separator 14 shown in FIG.

[0046] The obstacle detection unit 112 outputs a list of detected obstacles, i.e., a list of grouped obstacle grid groups, to the obstacle management unit 115. The list of grouped obstacle grid groups may include the obstacle grid groups separated by the separation unit 114. The list of obstacles detected by the obstacle detection unit 112 is also referred to as a first obstacle list. The obstacle management unit 115 manages a list of obstacles that have already been detected. The list of detected obstacles is also referred to as a second obstacle list. The obstacle management unit 115 manages the identification numbers of the obstacles based on the list of obstacles detected by the obstacle detection unit 112 and the list of obstacles that have already been detected.

[0047] In the following description, the list of obstacles detected by the obstacle detection unit 112 is also referred to as an obstacle detail information list. Furthermore, the list of detected obstacles managed by the obstacle management unit 115 is also referred to as an obstacle list. When an obstacle corresponding to an obstacle detected by the obstacle detection unit 112 is included in the obstacle list, the obstacle management unit 115 manages the detected obstacle by its identification number. Furthermore, the obstacle management unit 115 updates the information of the obstacle included in the obstacle list with the information of the detected obstacle. On the other hand, when an obstacle corresponding to an obstacle detected by the obstacle detection unit 112 is not included in the obstacle list, the obstacle management unit 115 assigns a new identification number to the detected obstacle and manages the detected obstacle with the new identification number. Furthermore, the obstacle management unit 115 adds information of the detected obstacle to the obstacle list as information of a new obstacle. The obstacle management unit 115 corresponds to the obstacle management unit 15 shown in FIG. 1 .

[0048] FIG. 5 is a diagram showing a specific example of a list of obstacles detected by the obstacle detection unit 112. In the example of FIG. 5, the obstacle detailed information list includes an index, the number of grids, the minimum coordinate value, the maximum coordinate value, and an attribute. "Index" indicates the index of the obstacle. "Number of grids" indicates the number of grids in the obstacle grid group. "Minimum coordinate value" indicates the minimum value of the coordinates of the circumscribing rectangle that surrounds the obstacle grid group. "Maximum coordinate value" indicates the maximum value of the coordinates of the circumscribing rectangle that surrounds the obstacle grid group. "Attribute" indicates whether the obstacle grid group as a whole is a dynamic obstacle or a static obstacle.

[0049] FIG. 6 shows a list of detected obstacles managed by the obstacle management unit 115. In the example of FIG. 6, the obstacle list includes an element number, an identifier (ID), center coordinates, presence, an update flag, and an index. "Element number" indicates the element number in the obstacle list. "ID" indicates the identification number of the obstacle. "Center coordinates" indicate the center coordinates of a circumscribing rectangle that surrounds the obstacle grid group.

[0050] "Existence" indicates whether an obstacle exists in the current frame. In other words, "existence" indicates whether an obstacle corresponding to an obstacle identified by an ID is included in the obstacle detail information list. "Update flag" indicates whether the information of an obstacle included in the obstacle list has changed in each frame, i.e., whether the obstacle information has been updated. The update flag is initialized to False for each frame, indicating no update.

[0051] The obstacle management unit 115 compares the coordinate information of the obstacle included in the obstacle detailed information list with the coordinate information of the obstacle included in the obstacle list, and determines whether or not the obstacle list includes an obstacle corresponding to the obstacle detected by the obstacle detection unit 112. The obstacle management unit 115 sequentially selects the obstacles included in the obstacle detailed information list, i.e., the obstacles detected by the obstacle detection unit 112, one by one. The obstacle management unit 115 compares the coordinate information of the selected obstacle with the coordinate information of the obstacle included in the obstacle list, and determines whether or not the obstacle corresponding to the selected obstacle is included in the obstacle list.

[0052] The obstacle management unit 115 may sort the obstacle detailed information list and the obstacle list according to the coordinates of the obstacles. For example, the obstacle management unit 115 sorts the obstacle detailed information list and the obstacle list in ascending order of the x-coordinate and the y-coordinate, respectively, with the upper left corner of the image as the origin (0,0). The obstacle management unit 115 sorts the obstacles included in the obstacle detailed information list in ascending order, for example, based on the minimum coordinate value included in the obstacle detailed information list shown in FIG. 5. The obstacle management unit 115 also sorts the obstacles included in the obstacle detailed information list in ascending order, for example, based on the center coordinates included in the obstacle list shown in FIG. 6. The obstacle management unit 115 may select obstacles one by one sequentially from the top of the obstacle detailed information list sorted in ascending order. Furthermore, in the comparison, the coordinate information of the selected obstacle is compared with the coordinate information of the obstacle in the obstacle list sorted in ascending order. This fixes the order of comparison with the obstacle list, making it possible to prevent the identification numbers of adjacent obstacles from being swapped.

[0053] If an obstacle corresponding to the selected obstacle is included in the obstacle list, the obstacle management unit 115 identifies the obstacle corresponding to the selected obstacle. In other words, the obstacle management unit 115 identifies which obstacle included in the obstacle list the selected obstacle corresponds to. For example, the obstacle management unit 115 identifies an obstacle in the obstacle list whose center coordinates are closest to the center coordinates of the selected obstacle as the obstacle corresponding to the selected obstacle. The obstacle management unit 115 updates the information of the obstacle identified in the obstacle list based on the information of the selected obstacle.

[0054] For example, the obstacle management unit 115 overwrites the "center coordinates" of the identified obstacle in the obstacle list with values ​​corresponding to the coordinates of the obstacle selected from the obstacle detailed information list. Also, the obstacle management unit 115 stores the index value of the obstacle selected from the obstacle detailed information list in the "index" of the identified obstacle in the obstacle list. In this way, the obstacle management unit 115 can manage the selected obstacle by the identification number of the obstacle that has already been detected. The obstacle management unit 115 sets the "update flag" of the identified obstacle to True, which indicates that the obstacle has been updated.

[0055] In the comparison, the obstacle management unit 115 may compare information about obstacles included in the obstacle list whose update flags are not set to "already updated" with information about the selected obstacle. This makes it possible to prevent information about an obstacle that has already been updated in the current frame in the obstacle list from being overwritten with information about another obstacle.

[0056] The obstacle management unit 115 outputs the obstacle detail information list and the obstacle list to the obstacle information transmission unit 116. The obstacle information transmission unit 116 controls the traveling of the mobile robot 150 by transmitting the obstacle detail information list and the obstacle list to the mobile robot 150. The obstacle information transmission unit 116 may transmit information about obstacles detected in the current frame from among the obstacles included in the obstacle list to the mobile robot 150. The control server 110 may generate travel path information for the mobile robot 150 based on the obstacle list acquired from the obstacle management unit 115, and transmit the generated travel path information to the mobile robot 150. The obstacle information transmission unit 116 corresponds to the control unit 16 shown in FIG. 1.

[0057] Next, the operation procedure will be explained. Fig. 7 is a flowchart showing the operation procedure of the control server 110. The operation procedure of the control server 110 corresponds to a control method. The depth information acquisition unit 111 acquires depth information of each location included in the movement range 200 of the mobile robot 150 based on image data of the movement range 200 acquired using the sensor 130 (step A1). In step A1, the depth information acquisition unit 111 acquires, for example, the height from the floor of each location as depth information.

[0058] The obstacle detection unit 112 detects obstacles present in the movement range 200 based on the depth information acquired in step A1 (step A2). In step A2, for example, the obstacle detection unit 112 generates a grid map obtained by dividing the movement range 200 into a plurality of grids based on the depth information. The obstacle detection unit 112 identifies, in the grid map, grids whose depth information is equal to or greater than a predetermined height as obstacle grids. Furthermore, the obstacle detection unit 112 groups the identified obstacle grids into one or more obstacle grid groups, and detects the grouped obstacle grid groups as obstacles.

[0059] The combining determination unit 113 and the separating unit 114 perform a combining check process on the detected obstacles (step A3). FIG. 8 is a flowchart showing the operation procedure of the combining check process. The combining determination unit 113 selects one obstacle from the obstacles detected in step A2 of FIG. 7, and determines whether the area of ​​the selected obstacle has previously contained two or more obstacles (step B1). In step B1, the combining determination unit 113 determines, for example, whether two or more obstacles existed within the range of a circumscribing rectangle surrounding the selected obstacle in the previous frame. If it is determined in step B1 that the area did not previously contain two or more obstacles, the processing for the selected obstacle ends.

[0060] If it is determined in step B1 that the selected obstacle contains two or more obstacles in the past, the separation unit 114 determines whether the attribute of the selected obstacle is a static obstacle as a whole (step B2). If it is determined in step B2 that the attribute is not a static obstacle as a whole, the separation unit 114 calculates the ratio of dynamic obstacle grids to the total number of obstacle grids for the current frame and the previous frame within the bounding rectangle surrounding the selected obstacle. Furthermore, the separation unit 114 calculates the amount of change in this ratio between the current frame and the previous frame (step B3). The separation unit 114 compares the calculated amount of change in this ratio with a predetermined threshold value and determines whether the amount of change in this ratio is equal to or less than the threshold value (step B4).

[0061] If the separation unit 114 determines in step B4 that the amount of change in the proportion of dynamic obstacle grids is equal to or less than the threshold value, it determines to perform separation processing to separate one or more obstacles from an obstacle containing two or more obstacles (step B5). If the separation unit 114 determines in step B2 that the obstacles as a whole are static, it proceeds to step B5 and determines to perform separation processing. If the separation unit 114 determines in step B4 that the amount of change in the dynamic obstacles is greater than the threshold value, it ends the processing. Steps B1 to B5 are repeatedly executed for each obstacle detected in step A2.

[0062] Returning to FIG. 7, the separation unit 114 performs the separation process on the obstacle detection system including two or more obstacles for which it has been determined that the separation process should be performed (step A4). FIG. 9 is a flowchart showing the operational procedure of the separation process. The separation unit 114 selects one of two or more obstacles present in the past obstacle detection results within the range of a circumscribing rectangle surrounding the obstacle detected in step A2 (step C1). The separation unit 114 determines whether the selected past obstacle is a static obstacle (step C2). If the selected past obstacle is determined to be not a static obstacle, that is, to be a moving obstacle, the separation unit 114 estimates the position of the selected obstacle in the current frame (step C3). Here, within the range of the circumscribing rectangle surrounding the obstacle detected in step A2 of FIG. 7, it can be estimated that the portion of the current obstacle grid that is not an obstacle grid of the past obstacle rectangle has moved from the previous frame to the present. By estimating the movement amount of the moving obstacle in this way, it is possible to more accurately identify the obstacle corresponding to the selected obstacle.

[0063] The separation unit 114 separates the obstacle grid group present at the position estimated in step C3 from obstacles consisting of two or more obstacles that correspond to the obstacle selected in step C1 in the current frame (step C4). If the separation unit 114 determines that the obstacle selected in step C2 is a static obstacle, it proceeds to step C4 and separates the static obstacle selected in step C1 from obstacles consisting of two or more obstacles. Steps C1 to C4 are repeatedly executed for each obstacle that previously existed within the circumscribing rectangle of the obstacle detected as a single obstacle.

[0064] There are four possible patterns in which multiple obstacles that were previously detected are detected as a single obstacle in the current frame. Pattern 1 is a pattern in which, when multiple adjacent static obstacles are located adjacent to each other, the adjacent multiple static obstacles are erroneously detected as a single obstacle due to fluctuations in depth values ​​caused by sensor measurement errors, etc. When adjacent static obstacles are detected as a single obstacle, the detected single obstacle is determined as a static obstacle as a whole. When the detected obstacle is determined as a static obstacle as a whole, the process proceeds from step B2 to step B5, where it is determined that separation processing should be performed. In this way, when adjacent static obstacles are erroneously detected as a single obstacle, the obstacle erroneously detected as a single obstacle can be separated into two or more obstacles based on past obstacle information in the separation processing.

[0065] Pattern 2 is a pattern in which a small, moving obstacle approaches a static obstacle, and the static obstacle and the moving obstacle are detected as a single obstacle. If the static obstacle is physically large and the moving obstacle is small, the detected obstacles are determined to be a static obstacle as a whole. In pattern 2, as in pattern 1, the process proceeds from step B2 to step B5, and it is determined in step B5 that separation processing should be performed. In this way, in the separation processing, static obstacles and small, moving obstacles can be separated from the detected obstacles based on past obstacle information.

[0066] In pattern 3, when a large dynamic obstacle passes near a static obstacle, the static obstacle and the dynamic obstacle are detected as a single obstacle. If the dynamic obstacle is physically large and the static obstacle is small, the detected obstacles are determined to be a dynamic obstacle as a whole. In pattern 3, the process proceeds from step B2 to step B3, where the change in the proportion of dynamic obstacle grids is calculated. If the static obstacle remains stationary, it is considered that the number of obstacle grids determined to be dynamic obstacles does not change significantly between the previous frame and the current frame. If the change in the proportion of dynamic obstacle grids is small, the process proceeds from step B4 to step B5, where it is determined to perform separation processing. In this way, for example, in the separation processing, static obstacles and dynamic obstacles can be separated from the detected obstacles based on past obstacle information.

[0067] In pattern 4, when a dynamic obstacle approaches a static obstacle and then the static obstacle starts moving and becomes a dynamic obstacle, the static and dynamic obstacles are detected as a single obstacle. For example, if a worker approaches a stationary cart and moves together with the cart, the worker and the cart are detected as a single obstacle. In pattern 4, as in pattern 3, the detected obstacles are determined to be dynamic obstacles as a whole, and processing proceeds from step B2 to step B3, where the change in the proportion of dynamic obstacle grids is calculated. However, when an obstacle such as a cart that was previously a static obstacle starts moving, the number of obstacle grids determined to be dynamic obstacles increases, and the proportion of dynamic obstacle grids is likely to change significantly between the previous frame and the current frame. If the change in the proportion of dynamic obstacle grids is large, it is determined that separation processing will not be performed. This prevents static obstacles that start moving together with dynamic obstacles from being separated.

[0068] Returning to FIG. 7 again, the obstacle management unit 115 manages the identification numbers of obstacles based on the list of obstacles detected in the current frame and the list of previously detected obstacles (step A5). For example, in step A5, the obstacle management unit 115 determines whether the obstacle detected in step A2 or the obstacle separated in step C5 is included in the list of previously detected obstacles. If it is determined that an obstacle corresponding to the detected obstacle is included in the list of previously detected obstacles, the obstacle management unit 115 assigns the obstacle identification number in the list of previously detected obstacles to the detected obstacle. Furthermore, the obstacle management unit 115 updates the information of the obstacle included in the list of previously detected obstacles with the information of the detected obstacle. If it is determined that an obstacle corresponding to the detected obstacle is not included in the list of previously detected obstacles, the obstacle management unit 115 assigns a new identification number to the detected obstacle. Furthermore, the obstacle management unit 115 adds the information of the detected obstacle to the list of previously detected obstacles as information of a new obstacle. Steps A1 to A5 correspond to the operation procedure of the obstacle detection device 117. The operation procedure of the obstacle detection device 117 corresponds to an obstacle detection method.

[0069] The obstacle information transmitting unit 116 controls the mobile robot 150 by transmitting information about the obstacles included in the list of detected obstacles to the mobile robot 150 (step A6). The control server 110 repeatedly performs steps A1 to A6 at predetermined time intervals.

[0070] A specific example will be described below. Fig. 10 is a diagram showing obstacles detected in a certain frame and the frame immediately before that. In Fig. 10, obstacle 201 indicates, for example, an obstacle detected in the current frame (a circumscribing rectangle of the detected obstacle). Obstacle 202 indicates a static obstacle detected in the frame immediately before the current frame (a circumscribing rectangle of the static obstacle). Obstacle 203 indicates a dynamic obstacle detected in the frame immediately before the current frame (a circumscribing rectangle of the dynamic obstacle).

[0071] In step B1 of Fig. 8, the combination determination unit 113 checks how many center coordinates of obstacles detected in the previous frame are present within the coordinate range of the circumscribing rectangle of the obstacle 201 detected in the current frame. In the example of Fig. 10, the coordinate range of the circumscribing rectangle of the obstacle 201 includes the center coordinates of the obstacle 202 and the center coordinates of the obstacle 203. If the coordinate range of the circumscribing rectangle of the obstacle 201 includes the center coordinates of the obstacle 202 and the obstacle 203, the combination determination unit 113 determines in step B1 that the circumscribing rectangle of the obstacle 201 contains the obstacles 202 and 203. In other words, the combination determination unit 113 determines that the obstacle 201 is an obstacle formed by combining two obstacles.

[0072] 11 is a diagram showing a grid map of an obstacle detected as one obstacle in a certain frame. For example, in the current frame, obstacle 201 includes a total of 25 static obstacle grids and a total of 29 dynamic obstacle grids. If the number of static obstacle grids is 25 and the number of dynamic obstacle grids is 29, the separation unit 114 calculates the ratio of the number of obstacle grids whose attribute is determined to be a dynamic obstacle to the total number of obstacle grids in the current frame by using 29 / (29+25)=0.537 in step B3 of FIG. 8.

[0073] 12 is a diagram showing a grid map of obstacles detected in the frame immediately before a given frame. For example, in the frame immediately before the current frame, obstacle 202 includes a total of 25 static obstacle grids. Also, obstacle 203 includes a total of 32 dynamic obstacle grids. If the number of static obstacle grids is 25 and the number of dynamic obstacle grids is 32, the separation unit 114 calculates the ratio of the number of obstacle grids whose attribute is determined to be a dynamic obstacle to the number of obstacle grids in the past frame by using 32 / (32+25)=0.561 in step B3 of FIG. 8.

[0074] In step B3, the separation unit 114 calculates the change in the ratio of the number of obstacle grids whose attribute is determined to be a moving obstacle to the number of obstacle grids, for example, the difference in the ratio, as the change in the ratio of obstacles whose attribute is determined to be a moving obstacle. In the examples of FIGS. 11 and 12, the absolute value of the difference is calculated as |0.537-0.561|=0.024. If the threshold value used in step B4 of FIG. 8 is 0.2, the separation unit 114 determines in step B4 that the change is equal to or less than the threshold value. If it is determined that the change is equal to or less than the threshold value, the separation unit 114 determines to separate one or more obstacles from the obstacle 201.

[0075] It is assumed that the separation unit 114 selected the obstacle 202 in step C1 of Fig. 9. The obstacle 202 is an obstacle that was determined to be a static obstacle in the frame immediately before the current frame. It is considered that the position of the obstacle 202 has not changed between the current frame and the frame immediately before. In step C5, the separation unit 114 separates the obstacle grid group corresponding to the obstacle 202 in Fig. 12 from the grid map of the obstacle 201 in Fig. 11 as the obstacle 202 in the current frame.

[0076] Next, it is assumed that the separation unit 114 selects the obstacle 203 in step C1 of Fig. 9. The obstacle 203 is an obstacle that has been determined to be a moving obstacle in the frame immediately before the current frame. In step C3, the separation unit 114 estimates the position of the obstacle 203 in the current frame. Furthermore, in step C4, the separation unit 114 searches for an obstacle grid group corresponding to the obstacle 203 in Fig. 12 in the grid map of the obstacle 201 in Fig. 11 based on the position estimated in step C3. In step C5, the separation unit 114 separates the obstacle grid group searched in step C4 as the obstacle 203 in the current frame.

[0077] The separation unit 114 adds information about the separated obstacle grid group to the obstacle detailed information list shown in Fig. 5, for example. Alternatively, the separation unit 114 updates information about an obstacle that includes multiple obstacles and is included in the obstacle detailed information list with information about the separated obstacle grid. The obstacle grid group separated as the obstacle 202 in Fig. 12 is managed in the obstacle management unit 115 by the identification number of the obstacle 202 that was previously detected. Furthermore, the obstacle grid group separated as the obstacle 203 in Fig. 12 is managed in the obstacle management unit 115 by the identification number of the obstacle 203 that was previously detected.

[0078] In this embodiment, the obstacle detection device 117 detects obstacles based on depth information acquired from a sensor 130, such as a stereo camera installed on the ceiling, in a warehouse environment where workers and the like are moving around. By using the information on the detected obstacles to control the mobile robot 150, it becomes possible for the mobile robot 150 to avoid obstacles on its travel path even when the obstacle is in the blind spot of the mobile robot 150, thereby improving safety and work efficiency.

[0079] In this embodiment, the grid attribute determination unit 122 of the obstacle detection unit 112 determines whether an obstacle grid is a moving obstacle or a static obstacle. The combination determination unit 113 determines whether an obstacle detected as a single obstacle by the obstacle detection unit 112 includes two or more obstacles. The separation unit 114 separates one or more obstacles from the obstacle detected as a single obstacle, using the attribute determination result by the grid attribute determination unit 122 and the attribute determination result of the obstacle grid in the past. In this way, it is possible to distinguish between moving obstacles and static obstacles among obstacles detected as a single obstacle, and separate the moving obstacles from the static obstacles.

[0080] For example, if a moving object approaches a stationary object and is detected as a single obstacle, the center coordinates of the detected obstacle may change significantly from the previous detection result, and the identification number of the obstacle may also change. Furthermore, if the combined obstacles are managed with a new identification number and then the moving object separates from the stationary object, the identification number of the obstacle may also change. In this embodiment, the separation unit 114 can separate one or more obstacles from an obstacle detected as a single obstacle using the attribute determination result. Therefore, even if two or more obstacles are close to each other, the continuity of the identification numbers of the detected obstacles can be maintained. Maintaining the continuity of obstacles is useful for future obstacle prediction and risk-sensitive probability control.

[0081] Next, a second embodiment of the present disclosure will be described. In this embodiment, the obstacle list, i.e., the list of detected obstacles managed by the obstacle management unit 115, includes, for each obstacle identification number, a non-update count indicating the time elapsed since the obstacle was last detected. The obstacle management unit 115 deletes from the obstacle list any obstacle whose non-update count indicates a time longer than a predetermined time.

[0082] More specifically, the obstacle management unit 115 sequentially selects obstacles one by one from the obstacle detail information list, which includes information on the obstacles detected by the obstacle detection unit 112 and the obstacles separated by the separation unit 114, and identifies the obstacle corresponding to the selected obstacle. For example, the obstacle management unit 115 identifies, in the obstacle list, the obstacle whose center coordinates are closest to the center coordinates of the selected obstacle as the obstacle corresponding to the selected obstacle. The obstacle management unit 115 updates the information on the obstacles identified in the obstacle list based on the information on the selected obstacle.

[0083] When all obstacles included in the obstacle detailed information list have been selected, the obstacle management unit 115 determines whether the obstacle list includes any obstacles that do not correspond to obstacles included in the obstacle detailed information list. In other words, the obstacle management unit 115 identifies obstacles that have not been detected in the current frame among the detected obstacles included in the obstacle list. The obstacle management unit 115 increments the "non-update count" of obstacles in the obstacle list that are determined not to correspond to obstacles included in the obstacle detailed information list.

[0084] The obstacle management unit 115 compares the non-update count of each obstacle included in the obstacle list with a predetermined threshold indicating a predetermined time. The predetermined threshold is defined, for example, as the number of processes per predetermined processing cycle. The obstacle management unit 115 identifies obstacles whose non-update count is greater than the predetermined threshold, i.e., obstacles whose non-update count indicates a time longer than the predetermined time. The obstacle management unit 115 deletes the identified obstacles from the obstacle list. For example, if the processing cycle is 100 ms and the non-update count threshold is 100, the obstacle management unit 115 deletes from the obstacle list any obstacle whose information has not been updated for 10 seconds. When the obstacle management unit 115 deletes an obstacle from the obstacle list, it deletes the obstacle information associated with the index of the deleted obstacle from the obstacle detailed information list.

[0085] In this embodiment, even if an obstacle is temporarily not detected, the obstacle management unit 115 retains information about the obstacle in the list of detected obstacles until the non-update count reaches a certain number. If an obstacle corresponding to an obstacle detected in the current frame is included in the list of detected obstacles, the obstacle management unit 115 manages the obstacle by the identification number of the obstacle included in the list of detected obstacles.

[0086] If the obstacle list only holds information about current or very recent obstacles, information about obstacles that have temporarily disappeared is deleted from the obstacle list. When the deleted obstacle reappears, it is detected as a new obstacle, and the obstacle identification number changes. In contrast, if information about obstacles that have disappeared is not deleted from the obstacle list, the temporarily disappeared obstacle can be managed with the same identification number as before. However, if information about obstacles that have not been detected for a long time is not deleted from the obstacle list, the number of matching operations with obstacles detected in the current frame becomes enormous, resulting in a high processing load. Furthermore, there is a possibility that a new obstacle may be mistakenly associated with another obstacle that was detected in the past.

[0087] In this embodiment, information about a temporarily disappeared obstacle is maintained in the list of detected obstacles until the number of non-updates reaches a certain number, and when the number of non-updates reaches the certain number, the information about the disappeared obstacle is deleted from the list of detected obstacles. This prevents the number of matches with detected obstacles from becoming overwhelming, and even if an obstacle temporarily disappears, if the disappeared obstacle reappears, the obstacle can be managed with the same identification number as before. In this embodiment, the continuity of the obstacle identification numbers can be maintained. This has the advantage of making it easier to predict future obstacles compared to when identification numbers change frequently.

[0088] Next, a description will be given of the physical configuration of the control server 110 and the obstacle detection device 117. Fig. 13 is a block diagram showing an example configuration of a computer device that can be used as the control server 110 or the obstacle detection device 117. The computer device 500 has a processor 510 such as a CPU (Central Processing Unit), a storage unit 520, a ROM (Read Only Memory) 530, a RAM (Random Access Memory) 540, a communication interface (IF: Interface) 550, and a user interface 560.

[0089] The communication interface 550 is an interface for connecting the computer device 500 to a communication network via wired communication means or wireless communication means, etc. The user interface 560 includes a display unit such as a display, and an input unit such as a keyboard, a mouse, and a touch panel.

[0090] The storage unit 520 is an auxiliary storage device that can store various types of data. The storage unit 520 does not necessarily have to be a part of the computer device 500, but may be an external storage device or a cloud storage connected to the computer device 500 via a network.

[0091] The ROM 530 is a non-volatile storage device. For example, a semiconductor storage device with a relatively small capacity, such as a flash memory, is used for the ROM 530. The programs executed by the CPU 510 can be stored in the storage unit 520 or the ROM 530. The storage unit 520 or the ROM 530 stores various programs that realize the functions of each unit of the control server 110 or the obstacle detection device 117.

[0092] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technology, compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0093] The RAM 540 is a volatile storage device. Various semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM 540. The RAM 540 can be used as an internal buffer for temporarily storing data and the like. The CPU 510 loads a program stored in the storage unit 520 or the ROM 530 into the RAM 540 and executes it. The CPU 510 executes the program, thereby realizing the functions of each unit in the control server 110 or the obstacle detection device 117. The CPU 510 may have an internal buffer for temporarily storing data and the like.

[0094] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0095] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but also to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0096] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0097] [Appendix 1] a depth information acquisition unit that acquires depth information of a travel path based on sensor data from a sensor that senses the travel path of the mobile robot; an obstacle detection unit that detects one or more obstacles present on the travel path based on the depth information and determines whether the detected obstacles are dynamic obstacles or static obstacles; a combination determination unit that determines whether the obstacles detected as one obstacle include two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; a separation unit that separates at least one obstacle from the obstacles determined to include two or more obstacles by using attributes of the detected obstacle and attributes in the detection results of the obstacles detected in the past; An obstacle detection device comprising: an obstacle management unit that manages the identification numbers of the obstacles based on a first obstacle list that includes information about the detected obstacles and a second obstacle list that includes information about obstacles that have already been detected.

[0098] [Appendix 2] The obstacle detection device described in Appendix 1, wherein the separation unit determines whether to separate at least one obstacle from the obstacles determined to include two or more obstacles based on the amount of change in the proportion of obstacles whose attributes are determined to be dynamic obstacles.

[0099] [Appendix 3] The obstacle detection device described in Appendix 2, wherein the separation unit decides to separate at least one obstacle from the obstacles determined to include two or more obstacles when the amount of change is equal to or less than a predetermined threshold.

[0100] [Appendix 4] The obstacle detection device according to any one of appendices 1 to 3, wherein the combination determination unit determines that the obstacle detected as a single obstacle includes two or more obstacles when the range of a circumscribing rectangle surrounding the detected obstacle includes two or more obstacles in the detection results of the obstacles detected in the past.

[0101] [Appendix 5] the depth information indicates a height from a floor surface of the travel path, The obstacle detection device according to any one of appendix 1 to 4, wherein the obstacle detection unit identifies, in a grid map obtained by dividing the roadway into a plurality of grids, grids whose depth information is equal to or greater than a predetermined height as obstacle grids, determines for each grid whether the attribute of the obstacle grid is a dynamic obstacle or a static obstacle, groups the identified obstacle grids into one or more obstacle grid groups, and detects the grouped obstacle grid groups as the obstacles.

[0102] [Appendix 6] the separation unit calculates, for the obstacle detected as the single obstacle, a first ratio indicating the ratio of the number of obstacle grids whose attributes are determined to be dynamic to the sum of the number of obstacle grids whose attributes are determined to be dynamic and the number of obstacle grids whose attributes are determined to be static, and calculates, for two or more obstacles present within a circumscribing rectangle surrounding the detected obstacle in the detection results of the previously detected obstacle, a second ratio indicating the ratio of the number of obstacle grids whose attributes are determined to be dynamic to the sum of the number of obstacle grids whose attributes are determined to be dynamic and the number of obstacle grids whose attributes are determined to be static, and decides to separate at least one obstacle from the obstacles determined to include the two or more obstacles if an absolute value of a difference between the first ratio and the second ratio is equal to or less than a predetermined threshold.

[0103] [Appendix 7] 7. The obstacle detection device according to claim 5, wherein the obstacle detection unit smoothes the depth information for each grid using a smoothing coefficient that is set in accordance with a time change in image information of the road.

[0104] [Appendix 8] The obstacle detection device described in Appendix 7, wherein, when the change over time of the image information in a certain grid is greater than a first threshold, the obstacle detection unit sets the smoothing coefficient in that grid to a value smaller than the smoothing coefficient set when the change over time of the image information is equal to or smaller than the first threshold, until a predetermined time has elapsed since it was determined that the change over time is greater than the first threshold.

[0105] [Appendix 9] The obstacle detection device described in Appendix 8, wherein, when the depth information in a grid changes by more than a predetermined value in the direction toward the floor, the obstacle detection unit sets the smoothing coefficient in that grid to a value smaller than the smoothing coefficient set when the change over time in the image information is less than a first threshold value.

[0106] [Appendix 10] 10. The obstacle detection device according to any one of claims 1 to 9, wherein the sensor is installed on a ceiling of the travel path.

[0107] [Appendix 11] An obstacle detection device according to any one of appendices 1 to 10; a control unit that controls the mobile robot based on information about obstacles included in the second obstacle list.

[0108] [Appendix 12] acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; An obstacle detection method that manages the identification numbers of the obstacles based on a first obstacle list containing information about the detected obstacles and a second obstacle list containing information about obstacles that have already been detected.

[0109] [Appendix 13] acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; managing the identification numbers of the obstacles based on a first obstacle list including information on the detected obstacles and a second obstacle list including information on obstacles that have already been detected; A control method for controlling the mobile robot based on information about obstacles included in the second obstacle list.

[0110] [Appendix 14] acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and attributes of the detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; A program that causes a computer to execute a process that includes managing the identification numbers of the obstacles based on a first obstacle list that includes information about the detected obstacles and a second obstacle list that includes information about obstacles that have already been detected.

[0111] [Appendix 15] acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; managing the identification numbers of the obstacles based on a first obstacle list including information on the detected obstacles and a second obstacle list including information on obstacles that have already been detected; A program that causes a computer to execute a process including controlling the mobile robot based on information about obstacles included in the second obstacle list.

[0112] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 10 that are dependent on Supplementary Notes 1 may also be dependent on Supplementary Notes 12, 13, 14, and 15 in the same dependency relationship as Supplementary Notes 2 to 10. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0113] 10: Control device 11: Depth information acquisition section 12: Obstacle detection unit 13: Connection determination section 14: Separation part 15: Obstacle Management Department 16: Control unit 20: Obstacle detection device 100: System 110: Control server 111: Depth information acquisition unit 112: Obstacle detection unit 113: Connection determination section 114: Separation part 115: Obstacle Management Department 116: Obstacle information transmission unit 117: Obstacle detection device 121: Grid map generation unit 122: Grid attribute determination unit 123: Grouping section 130: Sensor 150: Mobile robot 200:Movement range

Claims

1. a depth information acquisition unit that acquires depth information of a travel path based on sensor data from a sensor that senses the travel path of the mobile robot; an obstacle detection unit that detects one or more obstacles present on the travel path based on the depth information and determines whether the detected obstacles are dynamic obstacles or static obstacles; a combination determination unit that determines whether the obstacles detected as one obstacle include two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; a separation unit that separates at least one obstacle from the obstacles determined to include two or more obstacles, using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; an obstacle management unit that manages the identification numbers of the obstacles based on a first obstacle list that includes information about the detected obstacles and a second obstacle list that includes information about obstacles that have already been detected.

2. 2. The obstacle detection device according to claim 1, wherein the separation unit determines whether to separate at least one obstacle from the obstacles determined to include two or more obstacles, based on an amount of change in a proportion of obstacles whose attributes are determined to be dynamic obstacles.

3. The obstacle detection device according to claim 2 , wherein the separation unit determines to separate at least one obstacle from the obstacles determined to include two or more obstacles when the amount of change is equal to or less than a predetermined threshold value.

4. 4. The obstacle detection device according to claim 1, wherein the combination determination unit determines that the obstacle detected as a single obstacle includes two or more obstacles when two or more obstacles are included in the detection results of the obstacles detected in the past within a range of a circumscribing rectangle surrounding the detected obstacle.

5. the depth information indicates a height from a floor surface of the travel path, 4. The obstacle detection device according to claim 1, wherein the obstacle detection unit identifies, as obstacle grids, grids whose depth information is equal to or greater than a predetermined height in a grid map obtained by dividing the roadway into a plurality of grids, determines, for each grid, whether the attribute of the obstacle grid is a dynamic obstacle or a static obstacle, groups the identified obstacle grids into one or more obstacle grid groups, and detects the grouped obstacle grid groups as the obstacles.

6. An obstacle detection device according to any one of claims 1 to 3; a control unit that controls the mobile robot based on information about obstacles included in the second obstacle list.

7. acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; An obstacle detection method that manages the identification numbers of the obstacles based on a first obstacle list containing information about the detected obstacles and a second obstacle list containing information about obstacles that have already been detected.

8. acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; managing the identification numbers of the obstacles based on a first obstacle list including information on the detected obstacles and a second obstacle list including information on obstacles that have already been detected; A control method for controlling the mobile robot based on information about obstacles included in the second obstacle list.

9. acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and attributes of the detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; A program that causes a computer to execute a process that includes managing the identification numbers of the obstacles based on a first obstacle list that includes information about the detected obstacles and a second obstacle list that includes information about obstacles that have already been detected.

10. acquiring depth information of the travel path based on sensor data from a sensor that senses the travel path of the mobile robot; Detecting one or more obstacles present on the travel path based on the depth information; determining whether the detected obstacle is a dynamic obstacle or a static obstacle; determining whether the obstacle detected as one obstacle includes two or more obstacles based on the detected obstacle and detection results of obstacles detected in the past; Separating at least one obstacle from the obstacles determined to include two or more obstacles using attributes of the detected obstacle and attributes of the detection results of the obstacles detected in the past; managing the identification numbers of the obstacles based on a first obstacle list including information on the detected obstacles and a second obstacle list including information on obstacles that have already been detected; a program for causing a computer to execute a process including controlling the mobile robot based on information about obstacles included in the second obstacle list;

Images