AGV control device and AGV control method

Abstract

To provide an AGV control technology that appropriately avoids an obstacle with a reduced cost.SOLUTION: An AGV control device comprises: an image recognition unit that forms segmentation information by recognizing an object in image information acquired from a front of an AGV by semantic segmentation; a cost map forming unit that forms a cost map from point cloud information around the AGV; an obstacle detection unit that detects an obstacle on an AGV travel route from the segmentation information and the point cloud information; a cost map correction unit that corrects a cost map so that a cost of an area where the obstacle is present is increased when a degree of necessity for avoiding the obstacle detected by the obstacle detection unit is high; a travel route calculation unit that calculates and updates the travel route based on the corrected cost map when the obstacle is present on the travel route; and a travel control unit that controls traveling of the AGV based on the travel route updated by the travel route calculation unit.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an AGV (Automatic Guided Vehicle) control device and an AGV control method. [Background technology]

[0002] Generally, AGVs detect obstacles on their travel route from the front, making it difficult to obtain depth information about the obstacles. Patent Document 1, a conventional technology, discloses a technology in which information obtained from a camera installed in front of a vehicle is converted into a bird's-eye view image using a convolutional neural network, and the state of the front in the depth direction is acquired. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-211900 Summary of the Invention [Problem to be solved by the invention]

[0004] The convolutional neural network in the above-mentioned prior art requires learning of kernel parameters and warp parameters. In training this convolutional neural network, annotation data of the forward image is required to train the kernel parameters, and a viewpoint transformation map is required to train the warp parameters.

[0005] However, creating a viewpoint transformation map is very costly. In addition, since the two parameters are learned separately, the learning cost is high, and when the kernel parameter is updated, the warp parameter must be re-learned.

[0006] The present invention has been made in view of the above, and provides an AGV control technology that appropriately avoids obstacles at a reduced cost. [Means for solving the problem]

[0007] One aspect of the present invention that solves the above-mentioned problems and achieves the object is an AGV control device that controls the travel of an AGV based on a cost map, and includes: an image recognition unit that recognizes objects in image information acquired from the front of the AGV using semantic segmentation and creates segmentation information; a cost map creation unit that creates a cost map from point cloud information around the AGV; an obstacle detection unit that detects obstacles on the travel path of the AGV from the segmentation information and the point cloud information; a cost map correction unit that corrects the cost map so that the cost of the area where the obstacle exists is higher if the degree of necessity to avoid the obstacle detected by the obstacle detection unit is high; a travel path calculation unit that calculates and updates the travel path based on the corrected cost map if the obstacle exists on the travel path; and a travel control unit that controls the travel of the AGV based on the travel path updated by the travel path calculation unit.

[0008] In the above AGV control device, the travel route calculation unit may calculate and update the travel route depending on the depth of the obstacle.

[0009] Alternatively, one aspect of the present invention that solves the above-mentioned problems and achieves the object is an AGV control method that controls the travel of an AGV based on a cost map, the AGV control method including: recognizing objects in image information acquired from the front of the AGV by semantic segmentation and creating segmentation information; creating a cost map from point cloud information around the AGV; detecting obstacles on the travel path of the AGV from the segmentation information and the point cloud information; modifying the cost map so that the cost of the area where the obstacle exists is higher if the detected obstacle has a high degree of necessity for avoidance; calculating and updating the travel path based on the modified cost map if the obstacle exists on the travel path; and controlling the travel of the AGV based on the updated travel path.

[0010] In the above-described AGV control method, the updating may be performed by calculating a travel path depending on a depth of the obstacle. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an AGV control technology that can appropriately avoid obstacles at a reduced cost. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 shows examples of objects with sparse shapes, where FIG. 1(A) shows a mesh fence and FIG. 1(B) shows a grating. [Figure 2] Figure 2 shows the driving route of an AGV using point cloud information from LiDAR, where Figure 2(A) shows the driving route of the AGV when a mesh-like fence, which is a sparsely shaped object, is present on the driving route, and Figure 2(B) shows the driving route of the AGV when an object that is not sparsely shaped is present on the driving route. [Figure 3] FIG. 3 is a diagram showing the travel route (without obstacles) of the AGV set based on point cloud information from the LiDAR. [Figure 4]FIG. 4 is a diagram showing a travel route (with obstacles) of an AGV set based on point cloud information from a LiDAR. [Figure 5] FIG. 5 is a diagram showing a travel route in which an avoidance route is set for the route shown in FIG. [Figure 6] FIG. 6 is a diagram showing a travel path when the avoidance path results in a collision with an object. [Figure 7] FIG. 7 is a diagram showing how the AGV observes the depth of an object when it enters the avoidance path. [Figure 8] FIG. 8 is a diagram showing how the avoidance path is corrected. [Figure 9] FIG. 9 is a diagram showing an example of performing image recognition of a forward direction using a monocular camera. [Figure 10] FIG. 10 is a block diagram showing the configuration of the AGV control device according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an AGV control method according to the embodiment. [Figure 12] FIG. 12(A) is a diagram showing an example of image information acquired by a camera, and FIG. 12(B) is a diagram showing the image recognition result (segmentation information) based on the image information shown in FIG. 12(A). [Figure 13] FIG. 13 is a diagram showing the result of integrating the segmentation information and the three-dimensional map information. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. However, the present invention should not be construed as being limited by the following embodiments.

[0014] <Prerequisite technology> First, before explaining the present invention, the underlying technology will be explained. AGVs, which operate autonomously by obtaining information about the surrounding environment using LiDAR (Light Detection and Ranging), create a cost map of the area around the AGV and set a driving route that avoids areas with high costs on the cost map. However, LiDAR installed on AGVs may not detect objects with sparse shapes, and the cost of the object may be evaluated as low on the cost map even though it actually exists. FIG. 1 shows examples of objects with sparse shapes, where FIG. 1(A) shows a mesh fence and FIG. 1(B) shows a grating.

[0015] Figure 2 shows the driving route of an AGV using point cloud information from LiDAR, where Figure 2(A) shows the driving route of the AGV when a mesh-like fence, which is a sparsely shaped object, is present on the driving route, and Figure 2(B) shows the driving route of the AGV when an object that is not sparsely shaped is present on the driving route. As shown in Figure 2(B), LiDAR detects objects that are not sparsely shaped as obstacles. However, as shown in FIG. 2(A), LiDAR may fail to detect objects with sparse shapes. If an object is not detected, the cost of the area where the object exists will be evaluated as low, and the AGV may set a travel route in the area where the object exists.

[0016] FIG. 3 is a diagram showing the travel route (without obstacles) of the AGV set based on point cloud information from the LiDAR. As shown in Figure 3, if there are no obstacles on the travel route, the information on the start and end points is provided on a cost map created in advance from surrounding information acquired by LiDAR, an appropriate travel route connecting the start and end points is calculated, and the AGV travels autonomously along that route.

[0017] FIG. 4 is a diagram showing a travel route (with obstacles) of an AGV set based on point cloud information from a LiDAR. As shown in Figure 4, if an obstacle exists on the driving route, the cost of the obstacle area on the cost map increases based on the point cloud information from the LiDAR.

[0018] FIG. 5 is a diagram showing a travel route in which an avoidance route is set for the route shown in FIG. FIG. 6 is a diagram showing a travel path when the avoidance path results in a collision with an object. As shown in Figure 5, once the avoidance path is set, the AGV performs the avoidance operation. However, LiDAR only provides two-dimensional information capturing an object from the front, and does not provide depth information for the object. Therefore, as shown in Figure 6, the AGV may collide with an object even though an avoidance path has been set. Therefore, in order to avoid obstacles while driving using point cloud information from LiDAR, it is necessary to sequentially calculate the driving route even while driving, which is computationally expensive.

[0019] FIG. 7 is a diagram showing how the AGV observes the depth of an object when it enters the avoidance path. FIG. 8 is a diagram showing how the avoidance path is corrected. The AGV observes the depth of the object during the avoidance operation as shown in Figure 7, and corrects the avoidance route as shown in Figure 8 to avoid the obstacle while returning to the original route.

[0020] In this way, AGVs that avoid obstacles using point cloud information from LiDAR must perform sequential calculations during the obstacle avoidance operation to correct the driving path and control the AGV, which results in high computational costs and makes it difficult to drive smoothly.

[0021] FIG. 9 is a diagram showing an example of performing image recognition of a forward direction using a monocular camera. In Figure 9, the AGV recognizes an object ahead, estimates an avoidance route, and controls its travel. However, the image information from the monocular camera does not provide depth information about the object ahead. Therefore, after avoiding an obstacle, the travel route must be calculated sequentially until the vehicle returns to the original travel route, which results in high calculation costs.

[0022] As described above, the learning cost for obstacle avoidance is high in the conventional technology, and the calculation cost for obstacle avoidance is high in the above-mentioned underlying technology. In the following embodiment, an AGV control technology that appropriately avoids obstacles on the travel route while keeping learning costs and calculation costs low will be described.

[0023] <Embodiment 1> FIG. 10 is a block diagram showing the configuration of the AGV control device 10 according to this embodiment. Each component included in the AGV control device 10 shown in FIG. 10 can be realized by a processor such as an MPU (Micro-Processing Unit) and a CPU (Central Processing Unit), and a recording medium such as a semiconductor memory and a magnetic disk. FIG. 11 is a flowchart showing the AGV control method according to this embodiment.

[0024] An AGV (vehicle or self-propelled machine) that is the control target of the AGV control device 10 shown in FIG. The camera 20 captures an image of the front of the AGV, which is the object of control of the AGV control device 10, and transmits the image information to the AGV control device 10. The LiDAR 30 acquires information about the surrounding environment of the AGV and transmits point cloud information to the AGV control device 10. Although the camera 20 and the LiDAR 30 are provided on the AGV, the AGV control device 10 may be provided inside the AGV to be controlled, or may be provided outside the AGV to be controlled.

[0025] The AGV control device 10 shown in FIG. 10 includes an image recognition unit 101, a point cloud information processing unit 102, a recognition result integration unit 103, a cost map creation unit 104, an obstacle detection unit 105, a cost map correction unit 106, a travel path calculation unit 107, a travel path storage unit 108, a travel control unit 109, a calibration information storage unit 111, a map information storage unit 112, an object information DB unit 113, and a travel path information storage unit 114.

[0026] The operation of the AGV control device 10 shown in FIG. 10 is roughly divided into a map creation and travel route creation process in advance, and a travel route creation process during travel. In the preliminary map creation and driving route creation process, a point cloud map obtained from point cloud information from the LiDAR 30 is combined with segmentation information obtained from image information from the camera 20 to improve the recognition accuracy of sparsely shaped objects, and then a cost map (map) and driving route are created. In the process of creating a travel route while traveling, when an obstacle is detected while traveling, the cost map is corrected based on the size of the detected obstacle, and the travel route is updated.

[0027] <Preliminary map creation and route creation process (pre-processing)> First, as a preliminary process, the user manually drives the AGV, which is the object of control, along the planned travel route. The AGV collects image information and point cloud information around the planned route and creates a cost map (map) of the area around the planned route. Here, the map of the area around the planned travel route is created using SLAM (Simultaneous Localization and Mapping) or the like. Thereafter, a driving route for autonomous driving is created based on a map of the area surrounding the planned driving route. FIG. 12(A) is a diagram showing an example of image information acquired by camera 20, and FIG. 12(B) is a diagram showing the image recognition result (segmentation information) based on the image information shown in FIG. 12(A).

[0028] The image recognition unit 101 performs semantic segmentation, which is a deep learning area division method, on the image information acquired by the camera 20, recognizes objects in the image information, and creates segmentation information (image recognition processing S1).

[0029] The point cloud information processing unit 102 performs point cloud information processing to expand the point cloud information from the LiDAR 30 into a three-dimensional space, thereby creating three-dimensional map information (point cloud information processing S2).

[0030] The recognition result integration unit 103 uses the calibration parameters stored in the calibration information storage unit 111 to integrate the segmentation information and the three-dimensional map information to create integrated information (recognition result integration process S3).

[0031] The calibration information storage unit 111 stores calibration information (calibration parameters) for image information and point cloud information. Calibration of the image information and the point cloud information is performed in advance to connect the positional relationship between the image information and the point cloud information.

[0032] The cost map creating unit 104 creates a cost map from the three-dimensional map information (cost map creating process S4). The map information storage unit 112 stores the cost map.

[0033] The obstacle detection unit 105 detects, from the integrated information, an object (obstacle) that may hinder the travel of the AGV on the cost map (obstacle detection process S5). The obstacle detection unit 105 checks whether there is an overdetection or overdetection of an obstacle, thereby detecting whether the cost map needs to be corrected. Overdetection is a type of erroneous detection in which a non-obstacle that temporarily exists in the image capture target space and is captured in the image when the camera 20 acquires image information is detected as an obstacle. The obstacle detection unit 105 can use the segmentation information created by the image recognition unit 101 to detect objects with sparse shapes that are difficult to detect using LiDAR, such as fences or gratings. The obstacle detection unit 105 transmits the obstacle detection result to the driving route calculation unit 107 when the cost map does not need to be corrected (S5: No correction required), and transmits the obstacle detection result to the cost map correction unit 106 when the cost map needs to be corrected (S5: Correction required).

[0034] The cost map correction unit 106 corrects the cost of the target space according to the degree of necessity of avoiding obstacles on the travel route, based on the observed information and the information in the object information DB 113 (cost map correction process S6). The observed information, both the information from the camera 20 and the point cloud information from the LiDAR 30, is two-dimensional information.

[0035] Here, the degree of necessity of avoidance is determined based on both the degree to which the obstacle needs to avoid the obstacle and the degree to which the AGV needs to avoid the obstacle. The degree to which the obstacle needs to be avoided is set high when the obstacle is a human being, a fragile object, or a green belt in a factory or the like. The degree to which the AGV needs to avoid an obstacle is set high when the AGV may become unable to move or when the AGV may be damaged by colliding with an obstacle. The AGV may become unable to move if the obstacle is a grating, a gutter, or a step. If the AGV's travel route is set over such an obstacle, the AGV may become unable to travel. Collisions with obstacles can cause damage to AGVs and other injuries if the obstacle is prone to crumbling or falling over upon impact. If the AGV's travel route is set over such an obstacle, the AGV may be damaged by colliding with the obstacle.

[0036] The cost map correcting unit 106 has an internal table that associates the degree of avoidance necessity with the cost in the cost map, determines the cost in the cost map from the degree of avoidance necessity, and corrects the cost map. Specifically, when the degree of necessity of avoidance is high, the cost in the cost map is set high, and a travel route is set so as to move away from the obstacle. Furthermore, if there is an overdetection, the cost map correction unit 106 corrects the cost map so as to lower the cost of the part where the overdetection occurred. The map information storage unit 112 stores the corrected cost map.

[0037] The object information DB 113 stores at least the labels of objects that may become obstacles and the required avoidance degrees. The cost map correction unit 106 searches for the segmentation information and the corresponding label, and thereby the cost map correction unit 106 can obtain the necessary degree of avoidance of an obstacle on the travel route.

[0038] The travel route calculation unit 107 calculates and updates the travel route of the AGV based on the cost map that does not require correction or the corrected cost map (travel route calculation process S7). Here, the travel route is calculated using a global route planning algorithm such as Dijkstra's algorithm or A-star.

[0039] The travel route storage unit 108 stores the travel route updated by the travel route calculation unit 107 (travel route storage process S8). The travel route information storage unit 114 stores the travel route from the travel route storage unit 108 .

[0040] <Route creation process while driving> First, if no obstacles appear on the travel route prepared in advance, the AGV control device 100 controls the AGV, which is the control target of the AGV control device 100, according to the travel route prepared in advance. The case where an obstacle appears on the travel route will be described below.

[0041] As in the pre-processing, the image recognition unit 101 performs semantic segmentation on the image information acquired by the camera 20, recognizes objects in the image information, and creates segmentation information (image recognition processing S1).

[0042] Similar to the pre-processing, the point cloud information processing unit 102 performs point cloud information processing to expand the point cloud information from the LiDAR 30 into a three-dimensional space, thereby creating three-dimensional map information (point cloud information processing S2).

[0043] Similar to the pre-processing, the recognition result integration unit 103 uses the calibration parameters stored in the calibration information storage unit 111 to integrate the segmentation information and the 3D map information to create integrated information (recognition result integration process S3).

[0044] FIG. 13 is a diagram showing the result of integrating the segmentation information and the three-dimensional map information. FIG. 13 shows the distance to an object and the size of the object in the image information.

[0045] The obstacle detection unit 105 detects, from the integrated information, objects (obstacles) that may hinder the travel of the AGV on the cost map, similar to the pre-processing (obstacle detection process S5). The obstacle detection unit 105 checks whether there is an overdetection or overdetection of an obstacle, thereby detecting whether the cost map needs to be corrected. The obstacle detection unit 105 transmits the obstacle detection result to the driving route calculation unit 107 when the cost map does not need to be corrected (S5: No correction required), and transmits the obstacle detection result to the cost map correction unit 106 when the cost map needs to be corrected (S5: Correction required).

[0046] The cost map correction unit 106 corrects the cost of the target space according to the degree of necessity of avoiding obstacles on the travel route, based on the observed information (two-dimensional information) and the information in the object information DB 113 (cost map correction process S6). When an obstacle that has not been detected by the obstacle detection unit 105 is detected, the cost map correction unit 106 corrects the cost map according to the degree of necessity of avoiding the obstacle. The cost map correcting unit 106 has an internal table that associates the degree of avoidance necessity with the cost in the cost map, and determines the cost in the cost map according to this table to correct the cost map. This table is set so that the cost in the cost map is high when the degree of necessity of avoidance is high.

[0047] The driving route calculation unit 107 creates a local operation plan based on the corrected cost map, and updates the driving route by calculating a driving route that avoids obstacles using a global route planning algorithm such as Dijkstra's algorithm or A-star (driving route calculation process S7).

[0048] The travel route storage unit 108 stores the travel route updated by the travel route calculation unit 107 (travel route storage process S8). The travel route information storage unit 114 stores the travel route from the travel route storage unit 108 .

[0049] The travel control unit 109 controls the travel of the AGV by outputting operation commands and the like to the AGV, which is the control target of the AGV control device 100, according to the travel route updated by the travel route calculation unit 107.

[0050] According to this embodiment, the cost map is corrected to eliminate the need for sequential avoidance calculations, enabling the AGV to travel stably. Therefore, according to this embodiment, it is possible to control the AGV so as to appropriately avoid obstacles while keeping costs down.

[0051] <Embodiment 2> In the first embodiment, the cost on the cost map is set according to the degree of necessity of avoiding an obstacle, but the present invention is not limited to this. The cost map may be modified according to the depth of the obstacle in addition to the degree of necessity of avoiding the obstacle. In the following description, only the parts that are different from the first embodiment will be described, and the description of the first embodiment will be used for the other configurations.

[0052] The cost map correction unit 106 corrects the cost of the target space based on the observed information (two-dimensional information) and the information in the object information DB 113. The object information DB 113 stores at least the labels of objects that may become obstacles, the necessary avoidance levels, and three-dimensional information (size) of the objects. The three-dimensional information (size) is input in advance by the user, etc. Additionally, the three-dimensional information (size) includes the depth of the obstacle. When an obstacle that has not been detected by the obstacle detection unit 105 is detected, the cost map correction unit 106 corrects the cost map so as to increase the cost of the surrounding space including the depth of the obstacle. Furthermore, when overdetection is detected, the cost map correction unit 106 corrects the cost map so as to lower the cost of the portion where overdetection occurred.

[0053] In addition, when the cost map correction unit 106 searches the object information DB 113 based on the segmentation information and multiple objects are detected, the cost map correction unit 106 may adopt the object with the greatest depth among the multiple objects.

[0054] According to this embodiment, the same effects as those of the first embodiment can be obtained. That is, according to this embodiment, the cost map is corrected to eliminate the need for sequential avoidance calculations, enabling the AGV to travel stably. Therefore, according to this embodiment, it is possible to control the AGV so as to appropriately avoid obstacles while keeping costs down.

[0055] In this embodiment, if the amount of avoidance when avoiding an obstacle is set according to the depth, the AGV can avoid making large turns when avoiding the obstacle and can make small turns instead.

[0056] The present invention is not limited to the above-described embodiment, but also includes various modifications in which components are added, deleted, or converted from the above-described configuration. [Explanation of symbols]

[0057] 10 AGV control device 101 Image Recognition Unit 102 Point cloud information processing unit 103 Recognition result integration section 104 Cost Map Creation Department 105 Obstacle detection unit 106 Cost map correction section 107 Travel route calculation unit 108 Travel Route Storage Unit 109 Travel control unit 111 Calibration information storage unit 112 Map information storage unit 113 Object information DB section 114 Travel route information storage unit

Claims

1. An AGV control device that detects an obstacle on a travel route of an AGV based on a cost map and controls travel of the AGV to avoid the obstacle, an object information database unit that stores in advance the degree of necessity of avoiding the obstacle; an image recognition unit that recognizes objects in image information acquired from the front of the AGV by semantic segmentation and creates segmentation information; a cost map creation unit that creates the cost map from point cloud information around the AGV; an obstacle detection unit that detects the obstacle from the segmentation information and the point cloud information; a cost map correcting unit that includes a table that associates the degree of avoidance necessity with a cost in the cost map, and corrects the cost of an area where the obstacle exists based on the table and the degree of avoidance necessity stored in the object information database unit; a travel route calculation unit that calculates and updates a travel route based on the corrected cost map; a travel control unit that controls the travel of the AGV based on the travel route updated by the travel route calculation unit.

2. The AGV control device according to claim 1 , wherein the travel route calculation unit calculates and updates the travel route depending on the depth of the obstacle.

3. An AGV control method for detecting an obstacle on a travel route of an AGV based on a cost map and controlling travel of the AGV to avoid the obstacle, Recognizing objects in image information acquired from the front of the AGV by semantic segmentation and creating segmentation information; Creating the cost map from point cloud information around the AGV; detecting the obstacle from the segmentation information and the point cloud information; correcting the cost of the area in which the obstacle exists based on information associating the degree of necessity of avoiding the obstacle with the cost in the cost map; Calculating and updating a travel path based on the modified cost map; and controlling the travel of the AGV based on the updated travel route.

4. The AGV control method according to claim 3, wherein the updating is performed by calculating a travel route depending on the depth of the obstacle.

Images