Method, device, and program for guiding a transport vehicle

Abstract

To appropriately control the guidance and travel of a self-propelled material handling vehicle even in situations where environmental characteristics change. In a warehouse (100) which is a structure having a ceiling and in which the items stored therein are subject to change, the travel of a transport vehicle (9) is controlled using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front.

Description

[Technical Field]

[0001] The present invention relates to a technique for guiding a transport vehicle. [Background technology]

[0002] As a conventional technology for automatically driving a transport vehicle within a facility, a transport system is known which includes a facility, a plurality of reflectors installed within the facility, a transport vehicle that drives within the facility, a laser scanner installed on the transport vehicle that projects laser light around the transport vehicle and detects the laser light reflected by the reflectors, a measurement sensor installed on the transport vehicle that measures the environment around the transport vehicle, and a control device installed on the transport vehicle that controls the transport vehicle, and the transport vehicle is controlled by the control device to drive by switching between laser guidance, in which the transport vehicle drives automatically based on a current position determined using information from the laser scanner, and SLAM guidance, in which the transport vehicle drives automatically based on a current position determined by SLAM using information from the measurement sensor, under specified conditions (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-140490 Summary of the Invention [Problem to be solved by the invention]

[0004] In a warehouse, for example, the environmental characteristics often change significantly, are complex, and occur frequently due to frequent changes in the loads, freight, and equipment stored there, as well as the passage of workers and other transport vehicles. In situations where the environmental characteristics change, the stability and accuracy of the SLAM (Simultaneous Localization and Mapping) navigation of the transport vehicle may be significantly reduced, resulting in a loss of self-position and the inability to continue navigation.

[0005] Therefore, in one aspect, the present invention aims to provide a technique that makes it possible to appropriately control the guided travel of a self-propelled material handling vehicle even in situations where environmental characteristics change. [Means for solving the problem]

[0006] In order to solve the above problem, the method for guiding a transport vehicle according to the present invention may be configured to control the travel of a transport vehicle in a structure having a ceiling and in which items stored inside are subject to change using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front.

[0007] In the method for guiding a delivery vehicle according to the present invention, the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front may be combined.

[0008] The method for guiding a transport vehicle according to the present invention may also be configured so that the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front are synthesized based on a combination of a three-dimensional point cloud corresponding to a continuous vertical ceiling surface extracted from the three-dimensional point cloud data of the ceiling and a three-dimensional point cloud corresponding to a continuous vertical floor surface extracted from the three-dimensional point cloud data of the front, which are in a corresponding relationship between the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front.

[0009] In the method for guiding a transport vehicle according to the present invention, the structure may be a warehouse and the transport vehicle may be a forklift.

[0010] In addition, the transport vehicle guidance device of the present invention may have a means for controlling the travel of the transport vehicle in a structure having a ceiling and in which items stored therein are subject to change, using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front.

[0011] The transport vehicle guidance device according to the present invention may be configured such that the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front are combined.

[0012] The transport vehicle guidance device of the present invention may be configured to synthesize the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front based on a combination of a three-dimensional point cloud corresponding to a continuous vertical ceiling surface extracted from the three-dimensional point cloud data of the ceiling, which are in a corresponding relationship between the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front, and a three-dimensional point cloud corresponding to a continuous vertical floor surface extracted from the three-dimensional point cloud data of the front.

[0013] In addition, the guidance program for a transport vehicle according to the present invention may be configured to cause an information processing device to execute a process for controlling the travel of a transport vehicle within a structure having a ceiling and in which items stored therein are subject to change, using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front.

[0014] The transport vehicle guidance program according to the present invention may be configured to combine the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front.

[0015] The guidance program for a transport vehicle according to the present invention may be configured to synthesize the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front based on a combination of a three-dimensional point cloud corresponding to a continuous vertical ceiling surface extracted from the three-dimensional point cloud data of the ceiling, which are in a corresponding relationship between the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front, and a three-dimensional point cloud corresponding to a continuous vertical floor surface extracted from the three-dimensional point cloud data of the front. [Effects of the Invention]

[0016] According to one aspect of the present invention, it is possible to appropriately control the guided travel of a self-propelled material handling vehicle even in situations where environmental characteristics change. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic diagram illustrating an overall configuration of a transport vehicle guidance device according to an embodiment of the present invention. [Figure 2]2 is a plan view schematically showing a range in which a transport vehicle equipped with the transport vehicle guidance device of FIG. 1 can move. FIG. [Figure 3] FIG. 2 is a diagram illustrating a framework of SLAM in the driving control unit of FIG. 1. [Figure 4] 2 is a diagram illustrating a matching algorithm based on geometric features in the driving control unit of FIG. 1. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In the following embodiment, a case will be described in which a transport vehicle guidance device according to the present invention is mounted on a transport vehicle 9, a transport vehicle guidance program according to the present invention is executed in the transport vehicle guidance device, and a transport vehicle guidance method according to the present invention is applied to control the guided travel of the transport vehicle 9.

[0019] The transport vehicle 9 may be a vehicle that transports, conveys, or carries luggage or cargo, for example, a self-propelled loading and unloading vehicle equipped with a running mechanism and a mast that raises and lowers forks, etc., and specifically may be a forklift.

[0020] The transport vehicle 9 is configured to be capable of automatic travel (in other words, autonomous travel or automatic operation), and therefore may be unmanned, but may also have a worker on board.

[0021] In the following embodiment, a situation in which a transport vehicle 9 travels within a warehouse 100 will be described as an example.

[0022] The warehouse 100 is an example of a structure with a ceiling in the present invention. Specifically, the warehouse 100 is a structure with a ceiling as well as walls 104 and columns 105, a structure in which items stored therein (e.g., luggage, freight, equipment) change, and a structure through which workers and vehicles pass.

[0023] The transport vehicle 9 may pick up the luggage 300 stored inside the warehouse 100 (i.e., indoors) and transport the picked-up luggage 300 to a transport vehicle 200 (e.g., a truck) parked outside the warehouse 100 (i.e., outdoors) and waiting to be loaded, or may pick up luggage loaded on a transport vehicle 200 (e.g., a truck) parked outside the warehouse 100 (i.e., outdoors) and transport the picked-up luggage into the warehouse 100 and place it in a predetermined position (see FIG. 2). In other words, the transport vehicle 9 may operate at least inside the warehouse 100, which is indoors, and travel back and forth between the inside of the warehouse 100, which is indoors, and the transport vehicle 200 outdoors.

[0024] (Overall composition) FIG. 1 is a schematic diagram of the overall configuration of a transport vehicle guidance device 1 according to an embodiment, as an example of a specific configuration aspect of a transport vehicle guidance device according to the present invention.

[0025] The transport vehicle guidance device 1 according to the embodiment includes a ceiling detection unit 2, a forward detection unit 3, and an automatic control unit 5.

[0026] The ceiling detection unit 2 is a mechanism for detecting the ceiling above the transport vehicle 9 (i.e., the ceiling of the warehouse 100). The ceiling detection unit 2 is configured as a three-dimensional LIDAR (LIDAR: Laser Imaging Detection And Ranging).

[0027] In the description of the present invention, the "ceiling of the warehouse 100" refers to the ceiling above the transport vehicle 9 located within the warehouse 100, that is, the ceiling that is observed / detected within the warehouse 100.

[0028] The ceiling detection unit 2 emits a laser upward (specifically, toward the ceiling above) and receives the light reflected off the surface of the object (specifically, the ceiling) that the laser reaches, thereby calculating the distance from an arbitrary point (for example, a 3D LIDAR sensor as the ceiling detection unit 2) to each point on the surface of the object (ceiling), and the direction of each point on the surface of the object (ceiling) from the arbitrary point (for example, a 3D LIDAR sensor as the ceiling detection unit 2) as the base point. The point (location) where the laser reaches is a point that represents part of the surface of the object (ceiling).

[0029] The ceiling detection unit 2 is installed in a location where it is easy to irradiate a laser above the transport vehicle 9 (and easy to receive reflected light), specifically, where there are no obstacles between the 3D LIDAR sensor and above the transport vehicle 9 and the space above the transport vehicle 9 is open.

[0030] The ceiling detection unit 2 calculates the coordinates (specifically, for example, actual distances) of each point on the surface of an object (specifically, the ceiling) in a three-dimensional orthogonal coordinate system with an arbitrary point (for example, an arbitrary point in the warehouse 100; referred to as the "point cloud coordinate reference point") as the origin. The ceiling detection unit 2 acquires three-dimensional point cloud data (referred to as "ceiling three-dimensional point cloud data"), which is a collection of multiple points each having coordinates (specifically, for example, actual distances) in the three-dimensional orthogonal coordinate system with the point cloud coordinate reference point as the origin, relating to the shape of the ceiling of the warehouse 100.

[0031] The ceiling detection unit 2 outputs the ceiling three-dimensional point cloud data to the automatic control unit 5.

[0032] The forward detection unit 3 is a mechanism for detecting objects present in front of the transport vehicle 9. The forward detection unit 3 mainly detects objects on the floor and structures continuous with the floor inside the warehouse 100. The forward detection unit 3 is configured as a three-dimensional LIDAR.

[0033] The forward detection unit 3 emits a laser forward and receives the light reflected off the surface of an object that the laser reaches, thereby calculating the distance from an arbitrary point (for example, a 3D LIDAR sensor as the forward detection unit 3) to each point on the surface of the object, and the direction of each point on the surface of the object from the arbitrary point (for example, a 3D LIDAR sensor as the forward detection unit 3) as the base point. The point (location) where the laser reaches is a point that represents part of the surface of the object.

[0034] The forward detection unit 3 is installed in a location where it is easy to irradiate a laser ahead of the transport vehicle 9 (and easy to receive reflected light), specifically, in a location where there are no obstacles between the 3D LIDAR sensor and the front of the transport vehicle 9 and the space ahead of the transport vehicle 9 is open.

[0035] The forward detection unit 3 may be configured to detect objects present on the left and right sides of the delivery vehicle 9 in addition to those ahead of the delivery vehicle 9. In this case, the forward detection unit 3 irradiates a laser also to the left and right sides and receives light reflected from the surface of the object that the laser reaches, and calculates the distance, direction, etc., of the objects present on the left and right sides (specifically, the surfaces of the objects). Therefore, it can be said that the forward detection unit 3 is a mechanism for detecting objects present at least ahead of the delivery vehicle 9.

[0036] The forward detection unit 3 calculates the coordinates of each point on the surface of the object in a three-dimensional orthogonal coordinate system with the point cloud coordinate reference point as the origin (specifically, for example, actual distance). The forward detection unit 3 acquires three-dimensional point cloud data (referred to as "forward three-dimensional point cloud data"), which is a collection of multiple points each having coordinates (specifically, for example, actual distances) in the three-dimensional orthogonal coordinate system with the point cloud coordinate reference point as the origin, that is, (at least) relating to objects on the floor and structures connected to the floor.

[0037] The forward detection unit 3 outputs forward three-dimensional point cloud data to the automatic control unit 5.

[0038] (Functional configuration) The automatic control unit 5 controls at least the traveling of the transport vehicle 9. Specifically, the automatic control unit 5 controls at least the traveling of the transport vehicle 9 within the warehouse 100.

[0039] The automatic control unit 5 may also control the cargo handling operation of the transport vehicle 9. For example, the automatic control unit 5 may also control the operation of the transport vehicle 9 to pick up the luggage 300 stored in the warehouse 100 (i.e., indoors), and to transport the picked-up luggage 300 to a transport vehicle 200 (e.g., a truck) parked and waiting outside the warehouse 100 (i.e., outdoors), and then load the transported luggage 300 onto the loading platform 201 of the transport vehicle 200.

[0040] The automatic control unit 5 may also be configured to control the operation of the transport vehicle 9 to pick up cargo loaded on the loading platform 201 of a transport vehicle 200 (e.g., a truck) parked outside (i.e., outdoors) the warehouse 100, and to transport the picked-up cargo into the warehouse 100 and then place the transported cargo in a designated location within the warehouse 100.

[0041] The automatic control unit 5 includes a control unit 6 configured with a processor such as a CPU (Central Processing Unit) that performs arithmetic processing, a volatile storage unit 51 configured with volatile memory such as RAM (Random Access Memory) that provides a working area (memory), and a non-volatile storage unit 52 configured with non-volatile memory such as ROM (Read Only Memory) that provides a memory area (storage). If necessary, an auxiliary storage device such as an SSD (Solid State Drive) may be provided.

[0042] The automatic control unit 5 may be configured by, for example, a computer including a processor, a volatile memory, and a non-volatile memory.

[0043] A transport vehicle guidance program 53, which is a computer program that is read and executed by the control unit 6 configured by a processor, is stored in the non-volatile memory unit 52 (or an auxiliary storage device provided as needed). When the transport vehicle guidance program 53 is executed in the control unit 6 configured by a processor, the computer functions and at least the travel control of the transport vehicle 9 by the automatic control unit 5 is realized.

[0044] The automatic control unit 5, for example, controls the automatic (in other words, autonomous) driving of the transport vehicle 9 so as to perform a series of tasks related to picking up luggage, and also controls the automatic (in other words, autonomous) driving of the transport vehicle 9 so as to perform a series of tasks related to loading luggage.

[0045] The non-volatile storage unit 52 (or an auxiliary storage device provided as needed) further stores data on a combination of a start point and an end point of the operation of the transport vehicle 9 within the warehouse 100 (for example, including at least the space between the entrance / exit 101 of the warehouse 100 and the location where the cargo is stored within the warehouse 100). Note that the data on the combination of the start point and end point of the operation may include one or more waypoints / passing points as needed.

[0046] The combination data of the starting point and ending point of the transport vehicle 9's operation (including intermediate / passing points, if necessary) may be given as a combination of point coordinates (specifically, for example, actual distances) in a three-dimensional Cartesian coordinate system with the point cloud coordinate reference point as the origin.

[0047] The combination data of the starting point and ending point of the transport vehicle 9's operation (including waypoints / passing points, if necessary) may be stored in advance in the non-volatile memory unit 52 (or an auxiliary memory device provided as necessary) before the transport vehicle 9 operates, or may be input, updated, and stored at the appropriate time (in other words, at any time) during the operation of the transport vehicle 9.

[0048] The control unit 6 includes a travel control unit 61 .

[0049] The driving control unit 61 receives input of a data signal (referred to as a "ceiling LIDAR data signal") corresponding to three-dimensional point cloud data (i.e., ceiling three-dimensional point cloud data) of the ceiling above the transport vehicle 9 (i.e., the ceiling of the warehouse 100), which is output and transmitted from the ceiling detection unit 2.

[0050] The driving control unit 61 also receives input of a data signal (referred to as a "forward LIDAR data signal") that corresponds to three-dimensional point cloud data of objects located (at least) in front of the transport vehicle 9 (i.e., forward three-dimensional point cloud data), which is output and transmitted from the forward detection unit 3.

[0051] (Drive control) The method for guiding a transport vehicle according to the embodiment controls the travel of the transport vehicle 9 using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front inside a warehouse 100, which is a structure having a ceiling and in which items stored inside are subject to change.

[0052] In addition, the transport vehicle guidance device 1 in the embodiment has a driving control unit 61 as a means for controlling the driving of the transport vehicle 9 using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front inside the warehouse 100, which is a structure having a ceiling and in which items stored inside are subject to change.

[0053] In addition, the transport vehicle guidance program of the embodiment is configured to cause the transport vehicle guidance device 1, which is an information processing device, to execute a process of controlling the travel of the transport vehicle 9 using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of the front inside the warehouse 100, which is a structure having a ceiling and in which the items stored inside are subject to change.

[0054] In the following description, the transport vehicle 9 is assumed to be a forklift. In the following description, specifically, the travel of a forklift as the transport vehicle 9 within the warehouse 100 (that is, indoors) will be described.

[0055] The transport vehicle 9 may pick up the luggage 300 stored in the loading area 103 inside the warehouse 100 (i.e., indoors) and transport the picked-up luggage 300 to the loading platform 201 of the transport vehicle 200 (e.g., truck) parked and waiting at the loading area 400 outside the warehouse 100 (i.e., outdoors) and place it thereon, or may pick up luggage loaded on the loading platform 201 of the transport vehicle 200 (e.g., truck) parked at the loading area 400 outside the warehouse 100 (i.e., outdoors) and transport the picked-up luggage into the warehouse 100 and place it at a predetermined position in the loading area 103 (see FIG. 2). That is, the transport vehicle 9 may operate at least inside the warehouse 100, which is indoors, and travel back and forth between the inside of the warehouse 100, which is indoors, and the transport vehicle 200 outdoors.

[0056] The transport vehicle 9 (specifically, the travel control unit 61) uses the ceiling LIDAR data signal output from the ceiling detection unit 2 including a 3D LIDAR sensor, specifically, based on the ceiling 3D point cloud data acquired by the ceiling detection unit 2 including the 3D LIDAR sensor, to estimate its current position while simultaneously generating a ceiling map regarding the shape of the ceiling of the warehouse 100 using SLAM (Simultaneous Localization And Mapping: self-position estimation and map construction).

[0057] The transport vehicle 9 (specifically, the travel control unit 61) further uses the forward LIDAR data signal output from the forward detection unit 3 including a 3D LIDAR sensor, specifically, based on the forward 3D point cloud data acquired by the forward detection unit 3 including a 3D LIDAR sensor, to simultaneously generate an environment map / background map mainly relating to the shapes of objects on the floor and structures connected to the floor (for example, the wall 104 and the pillar 105 in the example shown in Figure 2) while estimating its current position and avoiding obstacles in front of it, using SLAM.

[0058] The estimation of the vehicle's current position and the generation of the ceiling map may be performed, for example, by using a ceiling map that has already been generated (for example, at the initial position of the transport vehicle 9) based on ceiling 3D point cloud data, which is geometric shape data of the ceiling of the warehouse 100 and is input as a ceiling LIDAR data signal, to estimate the vehicle's own position, and new shape data may be added based on this to expand the area of ​​the ceiling map.

[0059] The estimation of the vehicle's current position, and in particular the avoidance of obstacles ahead, as well as the generation of the environmental map / background map, may be performed, for example, by using an environmental map / background map that has already been generated (for example, at the initial position of the transport vehicle 9) based on forward three-dimensional point cloud data, which is geometric shape data of objects mainly on the floor and structures continuous with the floor in the warehouse 100, input as forward LIDAR data signals, and new shape data may be added based on this to expand the area of ​​the environmental map / background map.

[0060] With regard to the combined data of the starting point and ending point of the operation of the transport vehicle 9 (including, if necessary, waypoints / passing points), if the data is given as a combination of point coordinates (specifically, for example, actual distances) in a three-dimensional Cartesian coordinate system with the point cloud coordinate reference point as the origin, the starting point and ending point (and also, waypoints / passing points) may be converted into planar position coordinates on the ceiling shape data (i.e., ceiling map) generated using the ceiling LIDAR data signal (in other words, correlated with each other) and used.

[0061] The values ​​of the planar position coordinates in the ceiling shape data (i.e., ceiling map) generated using the ceiling LIDAR data signal are specifically coordinate values ​​related to the horizontal plane in a three-dimensional Cartesian coordinate system with the point cloud coordinate reference point as the origin.

[0062] That is, the self-position of the delivery vehicle 9 is represented by planar position coordinates on the ceiling map that correspond to directly above the delivery vehicle 9 in the vertical direction. In other words, if the two horizontal axes in a three-dimensional orthogonal coordinate system with the point cloud coordinate reference point as the origin are the X-axis and Y-axis and the vertical axis is the Z-axis, the X-coordinate and Y-coordinate of the self-position of the delivery vehicle 9 are represented by the X-coordinate and Y-coordinate that correspond to directly above the delivery vehicle 9 in the vertical direction on the ceiling map.

[0063] The travel control unit 61 of the control unit 6 controls the transport vehicle 9 to travel from the starting point to the end point of its operation in accordance with a sequence for transporting the cargo 300 in the warehouse 100 and by using the ceiling map and the environmental map / background map. At this time, the transport vehicle 9 transports the pallet 301 on which the cargo 300 is loaded, as necessary.

[0064] (3D point cloud data synthesis) The method for guiding a transport vehicle in accordance with the embodiment is such that the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data are synthesized based on a combination of a three-dimensional point cloud corresponding to a continuous vertical ceiling surface extracted from the ceiling three-dimensional point cloud data, which are in a corresponding relationship between the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data, and a three-dimensional point cloud corresponding to a continuous vertical floor surface extracted from the forward three-dimensional point cloud data.

[0065] The ceiling three-dimensional point cloud data output from the ceiling detection unit 2 including a three-dimensional LIDAR sensor and input to the driving control unit 61 as a ceiling LIDAR data signal and the forward three-dimensional point cloud data output from the forward detection unit 3 including a three-dimensional LIDAR sensor and input to the driving control unit 61 as a forward LIDAR data signal are combined and used for control by the driving control unit 61. The combination of the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data is, in other words, the combination of a ceiling map and an environment map / background map.

[0066] The ceiling 3D point cloud data and the front 3D point cloud data may be synthesized, for example, by treating the 3D point clouds corresponding to a predetermined vertical plane as 3D point clouds that have a corresponding relationship in these 3D point cloud data.

[0067] In this case, the detection range of the 3D LIDAR sensor included in the ceiling detection unit 2 is adjusted so that it can capture / detect not only the ceiling of the warehouse 100 but also the vertical surface of the structure extending downward in continuity with the ceiling. Note that with regard to the vertical surface of the structure extending downward in continuity with the ceiling of the warehouse 100, it is not necessary to capture / detect the entire structure, specifically, for example, the entire structure up to the floor inside the warehouse 100, but it is sufficient to capture / detect a portion that is continuous with the ceiling of the warehouse 100.

[0068] Furthermore, the detection range of the 3D LIDAR sensor included in the forward detection unit 3 is adjusted so that it can capture / detect not only objects on the floor of the warehouse 100 but also the vertical surfaces of structures extending upward and continuous with the floor (note that no special adjustment is required if the adjustment to enable objects on the floor of the warehouse 100 to be captured / detected also enables the vertical surfaces continuing with the floor to be captured / detected). Note that with regard to the vertical surfaces of structures extending upward and continuous with the floor of the warehouse 100, it is not necessary to capture / detect the entire structure, specifically, for example, the entire structure up to the ceiling of the warehouse 100, but it is sufficient that a portion continuing with the floor of the warehouse 100 be captured / detected.

[0069] From the ceiling 3D point cloud data based on capture / detection by the ceiling detection unit 2, a 3D point cloud is extracted that corresponds to a vertical or approximately vertical surface (referred to as a "ceiling-continuous vertical surface") on the surface / side of a part of the structure that extends downward in continuity with the ceiling of the warehouse 100 and is continuous with the ceiling.

[0070] The expression "approximately vertical surface" is used in consideration of the fact that the surface / side of wall 104 of warehouse 100 can be considered as a vertical surface even when, for example, windows, reinforcing members, or walkways (catwalks) are arranged on wall 104, and that the surface / side of pillar 105 of warehouse 100 can be considered as a vertical surface even when, for example, various switches or reinforcing members are arranged on pillar 105.

[0071] From the forward three-dimensional point cloud data based on capture / detection by the forward detection unit 3, a three-dimensional point cloud is extracted that corresponds to a vertical or approximately vertical surface (referred to as a "floor-continuous vertical surface") on the surface / side of a part of the structure that extends upward and is continuous with the floor inside the warehouse 100.

[0072] In this case, examples of vertical surfaces (i.e., continuous vertical ceiling surfaces and continuous vertical floor surfaces) used in combining the ceiling 3D point cloud data and the front 3D point cloud data include the surfaces / side surfaces of the walls 104 and the surfaces / side surfaces of the pillars 105 of the warehouse 100.

[0073] The extraction of a three-dimensional point cloud corresponding to a continuous vertical ceiling surface from the ceiling three-dimensional point cloud data may be performed using, for example, a trained model that has been trained in advance to extract a three-dimensional point cloud corresponding to a continuous vertical ceiling surface in the ceiling three-dimensional point cloud data in response to input ceiling three-dimensional point cloud data, using as training data three-dimensional point clouds corresponding to the surfaces / sides of walls and pillars that are structures that extend downward in continuity with the warehouse ceiling.

[0074] The extraction of the three-dimensional point cloud corresponding to the continuous vertical plane of the floor from the forward three-dimensional point cloud data may be performed using, for example, a trained model that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous vertical plane of the floor in the forward three-dimensional point cloud data in response to input of the forward three-dimensional point cloud data, using as training data three-dimensional point clouds corresponding to the surfaces / sides of walls and the surfaces / sides of columns that are structures that extend upward in continuity with the warehouse floor.

[0075] If the planar position coordinate values ​​of the three-dimensional point cloud corresponding to the continuous vertical ceiling surface extracted from the ceiling three-dimensional point cloud data (i.e., the X and Y coordinate values ​​when the two horizontal axes are the X and Y axes and the vertical axis is the Z axis in a three-dimensional Cartesian coordinate system with the point cloud coordinate reference point as the origin) match or nearly match the planar position coordinate values ​​of the three-dimensional point cloud corresponding to the continuous vertical floor surface extracted from the front three-dimensional point cloud data, it is determined that the three-dimensional point cloud corresponding to the continuous vertical ceiling surface and the three-dimensional point cloud corresponding to the continuous vertical floor surface are point clouds relating to the same structure and are three-dimensional point clouds that have a corresponding relationship in the two three-dimensional point cloud data.

[0076] Whether or not the three-dimensional point clouds correspond to each other is determined based on the X coordinate value and the Y coordinate value. Therefore, the three-dimensional point cloud corresponding to the continuous vertical ceiling surface extracted from the ceiling three-dimensional point cloud data and the three-dimensional point cloud corresponding to the continuous vertical floor surface extracted from the front three-dimensional point cloud data do not need to be continuous in the vertical direction. In the vertical direction, there may be a blank area where no three-dimensional point cloud exists between the bottom end of the three-dimensional point cloud corresponding to the continuous vertical ceiling surface and the top end of the three-dimensional point cloud corresponding to the continuous vertical floor surface.

[0077] Based on a combination of three-dimensional point clouds (one or more combinations) that correspond to each other in the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data, the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data are synthesized, and ultimately the ceiling map and the environment map / background map are synthesized.

[0078] Furthermore, the above synthesis makes it possible to synthesize ceiling 3D point cloud data and forward 3D point cloud data acquired at different times, and ultimately makes it possible to synthesize a ceiling map and an environment map / background map.

[0079] After the ceiling 3D point cloud data and the forward 3D point cloud data are synthesized, i.e., after the ceiling map and the environmental map / background map are synthesized, the data is used to control the traveling of the transport vehicle 9 by the traveling control unit 61, so that the estimation of self-position based mainly on the ceiling 3D point cloud data (ceiling map) and the avoidance of obstacles based mainly on the forward 3D point cloud data (environmental map / background map) function as an integrated task and are executed in a state where consistency is always maintained.

[0080] Furthermore, when combining the ceiling three-dimensional point cloud data and the forward three-dimensional point cloud data, attention is paid to the three-dimensional point cloud corresponding to the continuous vertical plane of the ceiling and the three-dimensional point cloud corresponding to the continuous vertical plane of the floor, that is, attention is paid to structures that are continuous with the ceiling of the warehouse 100 and also with the floor inside the warehouse 100. Such structures that are continuous with the ceiling of the warehouse 100 and also with the floor inside the warehouse 100 are considered to be permanent structures of the warehouse 100, such as the walls 104 and pillars 105. For this reason, by estimating the self-position using shape data of the permanent structures in addition to shape data of the ceiling of the warehouse 100 itself, the travel of the transport vehicle 9 can be appropriately controlled even in situations where environmental characteristics change.

[0081] A vertical plane that exists continuously for a predetermined period (referred to as a "continuous existence period") or longer in the 3D point cloud data acquired in a time series may be selected as a continuous ceiling vertical plane or a continuous floor vertical plane. By selecting a vertical plane that exists continuously for a period or longer as a continuous ceiling vertical plane or a continuous floor vertical plane, it is possible to prevent luggage or cargo stored in the warehouse 100, which may fluctuate, from being selected as a continuous ceiling vertical plane or a continuous floor vertical plane, and ultimately to perform stable and appropriate synthesis of the 3D point cloud data.

[0082] The continuous existence period is not limited to a specific length of time, but may be set appropriately to any period within the range of, for example, about one week to about one month.

[0083] (SLAM framework) The framework of the SLAM (Simultaneous Localization And Mapping) algorithm is broadly composed of two processing systems: the "front end" and the "back end" (see Figure 3).

[0084] The front end performs sensor-dependent processing based on raw data (specifically, 3D point cloud data) acquired from the 3D LIDAR sensor. Specifically, the front end performs initial estimation of the environment around the transport vehicle 9 and the movement of the transport vehicle 9.

[0085] The main tasks of the front end are "motion estimation" and "obstacle position estimation."

[0086] Motion estimation is a process of estimating how much (in other words, how) the delivery vehicle 9 has moved by comparing continuous sensor data (specifically, 3D point cloud data) in time series when the delivery vehicle 9 is moving. This is achieved by "scan matching" and "feature point matching." The positions of surrounding obstacles are also simultaneously determined to ensure the delivery vehicle 9 can move safely.

[0087] Scan matching is a process of comparing successive sensor data (specifically, 3D point cloud data) and estimating and adjusting the position and posture (specifically, at least the orientation) of the transport vehicle 9 so that the shape and characteristics of the current sensor data match those of the immediately preceding sensor data.

[0088] Feature point matching is a process of estimating and adjusting the position and attitude (direction) of the transport vehicle 9 by tracking feature points in sensor data (specifically, three-dimensional point cloud data).

[0089] Obstacle position estimation is a process of determining the positions of surrounding obstacles from acquired sensor data (specifically, 3D point cloud data). In obstacle position estimation, for example, the outlines of walls and obstacles are extracted from the acquired sensor data and plotted on a map.

[0090] In the present invention, sensor data in a range equivalent to ceiling three-dimensional point cloud data (ceiling map) is used to estimate and adjust the position and attitude (specifically, at least the direction) of the transport vehicle 9, and sensor data in a range equivalent to forward three-dimensional point cloud data (environment map / background map) is used to estimate the position of an obstacle.

[0091] The front end also detects loop closures by detecting when the transport vehicle 9 returns to a location it has previously passed through and correlating the past location information with the current location information. Detecting loop closures provides clues for correcting errors in the map, thereby reducing errors in location estimation.

[0092] The front end relies on sensors and performs basic processing to capture the vehicle's own position and surrounding environment in real time. The front end processing outputs estimates of the vehicle's continuous position and attitude (direction) and simple map information based on them.

[0093] In the back end, the movement, position, and attitude (direction) of the delivery vehicle 9 obtained in the front end, as well as the observation results, are integrated, and optimization processing is performed to finally obtain an accurate map and the self-position of the delivery vehicle 9. In the back end, general-purpose calculation processing that is not dependent on sensors is performed.

[0094] The main tasks of the backend are "pose graph construction" and "graph optimization." Specifically, the backend performs graph optimization based on the pose graph, and reconstructs a consistent map as a whole and the self-position of the transport vehicle 9.

[0095] Pose graph construction is a process of constructing a pose graph that represents the movement path and observation results of the transport vehicle 9 as nodes (points) and edges (lines / edges). The nodes represent the position and posture (pose) of the transport vehicle 9 at a certain point in time, and the edges represent the relationship between the movements and observations (in other words, constraints, information about the relative relationship between positions and orientations that holds between 3D point cloud data at different points in time).

[0096] Graph optimization is an optimization process for ensuring consistency of errors on the pose graph. For example, if an error such as "the actual position was closer" is found by loop closure, this is reflected in the entire graph structure to correct the position estimation error overall. This improves the consistency of the position and posture (orientation) of the transport vehicle 9 and the map information.

[0097] The back end is responsible for reconstructing all of the transport vehicle 9's movement history from the past to the present and the map in a consistent form based on the information obtained by the front end.

[0098] As a result of the front-end and back-end processing, a posture graph and map information of the transport vehicle 9 are obtained. These become the basis for the transport vehicle 9 to act and work autonomously in an unknown environment.

[0099] The SLAM algorithm allows the front-end and back-end to work together to build a highly accurate map while the delivery vehicle 9 moves and explores autonomously, even in unknown environments. The resulting pose graph and map information are used for navigation of the delivery vehicle 9.

[0100] (scan matching algorithm) The accuracy of trajectory estimation and map construction of the transport vehicle 9 in SLAM depends heavily on the performance of the scan matching algorithm.

[0101] Scan matching involves matching the 3D point cloud data acquired by the delivery vehicle 9 (specifically, the 3D LIDAR sensor) with the previous frame or an existing map to find the best match. This process estimates the current position of the delivery vehicle 9, which forms the basis for motion estimation in the front end of the SLAM framework.

[0102] The results of scan matching are used as the initial values ​​for the position and attitude (orientation) of the transport vehicle 9 in the optimization process in the backend, and are also incorporated into the pose graph as observation constraints between multiple 3D point cloud data. Therefore, the accuracy and calculation efficiency of scan matching have a significant impact on the consistency and real-time performance of the map information that is ultimately constructed.

[0103] Scan matching algorithms are classified into three types based on their approach: "matching algorithms based on iterative closest points (ICP)," "matching algorithms based on geometric features," and "matching algorithms based on mathematical features."

[0104] 1) Matching algorithm based on ICP The ICP-based matching algorithm is a method for aligning three-dimensional point clouds at successive points in time, and is a method for estimating the transformation of the position and attitude (orientation) of the transport vehicle 9 by repeatedly minimizing the distance error between corresponding points in the three-dimensional point cloud data.

[0105] ICP has the problem that it easily falls into a local optimum when it uses an iterative optimization method to find the optimal transformation matrix (change in position and orientation) between two 3D point clouds.

[0106] Therefore, in the present invention, the following configuration is incorporated to improve the accuracy and robustness of ICP. A) Optimization of point selection strategy b) Improved matching accuracy of corresponding points C) Removal of outliers (mismatches) d) Weighting for each corresponding point E) Minimization of the cost function

[0107] The following algorithm may also be introduced: PP-ICP (Point-to-Plane ICP) By minimizing the distance between a point and a surface as the error, rather than the distance between corresponding points, more accurate registration can be achieved in an environment with a smooth surface.

[0108] PL-ICP (Point-to-Line ICP) The error cost is the distance between a point and a line, and more accurate registration can be achieved in an environment where linear features such as indoor structures (e.g., walls, pillars, etc.) are dominant.

[0109] GICP (Generalized ICP) By integrating PP-ICP and PL-ICP to handle point clouds within a probabilistic framework, the relationships between corresponding points can be modeled more flexibly and accurately.

[0110] NICP (Normal ICP) By taking into account local geometric information such as the normal vectors and curvature of the point cloud, it is possible to correspond to the curved shape of the environment and perform more accurate registration.

[0111] ·IMLS-ICP(Iterative Closest Point with Implicit Moving Least Squares) By assigning weights to each point and selecting representative points for matching, more reliable registration can be achieved while reconstructing smooth surfaces based on local normal directions.

[0112] ·BLAM(Bayesian Localization And Mapping) First, a rough pose transformation is estimated using GICP, and then the closest point is obtained and a high-precision alignment is performed again using GICP, which can improve the map consistency and allow better loop closure detection.

[0113] LiTAMIN By incorporating the Kullback-Leibler (KL) divergence, which is the difference between probability distributions, into the cost function, it is possible to compress the information in 3D point cloud data while maintaining registration accuracy and improving processing speed. In addition, spatial structuring using KD trees and voxel grids can achieve both computational efficiency and robustness.

[0114] In addition, the 3D LIDAR included in the ceiling detection unit 2 for detecting the ceiling above the transport vehicle 9 (i.e., the ceiling of the warehouse 100) may be implemented with an ICP that does not handle color information as a navigation LIDAR, and the 3D LIDAR included in the forward detection unit 3 for detecting objects in front of the transport vehicle 9 may be implemented with a color ICP algorithm that handles 3D point clouds including color information.

[0115] In addition, SLAM results based on 3D point cloud data from the ceiling and 3D point cloud data from the front are integrated to scan and identify the entire environment in three dimensions, enabling highly accurate self-position estimation and map generation even in wide-area and complex environments, improving operational reliability.

[0116] 2) Matching algorithm based on geometric features A matching algorithm based on geometric features is a method for estimating the position and attitude (orientation) of the transport vehicle 9 by extracting geometrically meaningful features such as edges and planes from a three-dimensional point cloud and aligning frames based on their correspondence.

[0117] LOAM (Lidar Odometry And Mapping), a type of scan matching based on geometric features, extracts geometrically significant feature points such as edges (specifically, sharp changes in curvature) and planes (specifically, smooth areas) from 3D point cloud data, and performs matching with past frames based on these features. Through matching, the position and posture (orientation) of the transport vehicle 9 in the current frame are estimated, achieving continuous mapping.

[0118] In the present invention, improvements are made to achieve higher accuracy and stability.

[0119] Specifically, prior to feature extraction, a three-dimensional point cloud corresponding to (or equivalent to) the ceiling of the warehouse 100 is separated, thereby reducing the influence of irrelevant features on matching.

[0120] Separation of the three-dimensional point cloud corresponding to / equivalent to the ceiling of warehouse 100 may be performed, for example, by separating point clouds above a predetermined height (in other words, altitude) determined taking into account the actual structure of warehouse 100.

[0121] Additionally, the 3D points corresponding to (i.e., equivalent to) the ground may be removed, further reducing the influence of irrelevant features on the matching.

[0122] The removal of the three-dimensional point cloud corresponding to / equivalent to the ground may be performed, for example, by removing point clouds below a predetermined height (in other words, altitude).

[0123] The system extracts and utilizes ceiling features from reference objects in the environment to effectively suppress drift (error accumulation) in self-location estimation during long-term operation. The ceiling features function as stable landmarks, increasing the reliability of estimation of the position and attitude (orientation) of the transport vehicle 9. This process consists of the following modules (see Figure 4).

[0124] Point cloud division module: Classifies 3D point cloud data into regions. Feature Extraction Module: Detects geometric features such as edges and planes. · LIDAR measurement module: Obtains observations (current environment) based on extracted features. LIDAR mapping module: Matches observations with historical maps to update the maps. · Transformation fusion module: Integrates matching results to calculate the transformation amount. Position and attitude determination module: outputs the accurate position and attitude of the transport vehicle 9.

[0125] This invention also incorporates a point-based rendering technique called Surfel (Surface Elements). Surfel is a method for representing 3D shapes using points and their local surface information (normals, area, etc.) instead of polygon meshes. It is particularly effective for rendering and surface-based feature extraction.

[0126] In the present invention, the following processing is specifically performed by utilizing Surfel.

[0127] Continuously acquired 3D point cloud data is incrementally integrated into a multi-resolution grid map.

[0128] Even if the measurement points are sparse, we statistically assign surfaces using a Gaussian mixture model (GMM) to estimate the surface structure probabilistically.

[0129] 3) Mathematical feature-based matching algorithm A matching algorithm based on mathematical features is a method of representing 3D point cloud data not in terms of geometric feature points but in terms of distributional and statistical properties in space, and using this to estimate changes in position and orientation.

[0130] This invention employs the NDT (normal distribution transformation) algorithm. When processing 3D point cloud data, NDT divides the space into a fixed grid (cells) and approximates the point cloud distribution within each cell with a multidimensional normal distribution (Gaussian distribution). This allows complex point cloud data to be represented as a statistical model, enabling alignment (matching) between scans (i.e., between consecutive 3D point cloud data in a time series) using a probabilistic method.

[0131] In the present invention, in addition to the conventional NDT algorithm, the following advanced method is adopted to achieve both high matching accuracy and calculation efficiency.

[0132] 3D NDT This is an NDT algorithm that targets the entire three-dimensional space, and is suitable not only for aligning three-dimensional point clouds, but also for modeling the entire environment (map generation).

[0133] Occupancy NDT (ONDT) By probabilistically representing whether each cell is occupied or not, more precise map construction and location estimation are achieved.

[0134] ·RGC-NDT(Regional Growth Clustering - NDT) A clustering algorithm is used to identify features in the environment as local structures (for example, a series of walls or a group of pillars).

[0135] (SLAM algorithm) Deep learning may be used to perform data-driven learning, resulting in more accurate models than manual design. The combination of SLAM and deep learning improves the efficiency, accuracy, and robustness of SLAM algorithms by applying deep learning to some or more steps of the traditional SLAM framework.

[0136] Semantic SLAM uses deep learning to extract semantic information and build a semantic map, improving a robot's ability to understand itself.

[0137] As an inter-frame estimation method using deep learning, either of the following (A) or (B) may be used.

[0138] (1) Point cloud registration method GMM (Gaussian Mixed Model), 3DFeat-Net, FCGF (Fully Convolutional Geometric Features), Siamese-PointNet (Siamese Point Network), UGMMReg (Unified Gaussian Mixture Model Registration)

[0139] (2) End-to-end deep learning point cloud registration method DeepGMR (Deep Gaussian Mixture Registration), 3DRegNet (3D Registration Network), MLP-GCN (MultiLayer Perceptron-Graph Convolutional Networks), IPCR (Iterative Point Cloud Registration)

[0140] (Backend) The backend uses a 2D / 3D pose graph. Specifically, the backend performs the following processing, for example:

[0141] Use the Navigation Toolbox to generate 2D / 3D pose graphs.

[0142] The pose graph is optimized based on node (point) and edge (line / edge) constraints.

[0143] Use Computer Vision Toolbox to perform bundle adjustment.

[0144] Use the SLAM map generation tool to generate an occupancy grid map.

[0145] Find and fix loop closures and export the map as an occupancy grid for use in path planning.

[0146] Use Navigation Toolbox to implement path planning algorithms and perform path planning and control.

[0147] Send control commands to follow the planned path and avoid obstacles.

[0148] Use Parallel Computing Toolbox to run computationally intensive processes in parallel and speed them up.

[0149] (SLAM algorithm based on filtering theory) In the backend, a SLAM algorithm based on filtering theory may be used.

[0150] SLAM algorithms based on filtering theory apply the principles of Bayesian inference, which is a method of probabilistically estimating the current state (posterior distribution) by combining past information (prior distribution) with new observation data (likelihood).

[0151] SLAM algorithms based on filtering theory include Kalman filter-based SLAM and particle filter-based SLAM.

[0152] In relation to Kalman filter-based SLAM, the Kalman filter is an algorithm specialized for estimating the state of a linear Gaussian system, whereas the actual movement of a transport vehicle 9 and the observations of a 3D LIDAR sensor contain many nonlinear elements. For this reason, an extended Kalman filter (EKF) is used. In the extended Kalman filter, nonlinear movement models and observation models are linearly approximated using Taylor expansion or the like, and state estimation is performed based on this.

[0153] However, the extended Kalman filter has limitations in handling nonlinearity, and there are issues with accuracy and stability. Therefore, this invention introduces an unscented Kalman filter (UKF). The unscented Kalman filter handles nonlinear transformations using representative sample points called sigma points, eliminating the need for linearization and enabling more accurate representation of the state.

[0154] Particle filters are an estimation method that can handle complex systems with nonlinear and non-Gaussian distributions. This method uses multiple "particles (hypotheses)" to represent the posterior distribution of the state. Each particle represents the state of the transport vehicle 9 and has its own weight. Accurate estimation is possible by updating the particle weights based on observations, retaining important particles and eliminating unnecessary ones.

[0155] In the present invention, with regard to SLAM, a particle filter is used to estimate the position of the delivery vehicle 9, and a trackless Kalman filter is used to generate and update the map. As a result, in the present invention, the position of the delivery vehicle 9 and the generation and update of the map can be performed accurately and quickly, and the calculation cost can be significantly reduced.

[0156] (SLAM algorithm based on nonlinear optimization) Algorithms based on graph optimization are one of the most widely used approaches in current SLAM. In this method, the trajectory of a delivery vehicle9 and landmarks in the environment are represented as nodes (points), and the relative positional relationships between nodes are represented as edges (lines / edges), and the graph structure consisting of these is optimized.

[0157] The major advantage of graph optimization frameworks is that they can reduce the amount of calculation while maintaining accuracy. Rather than optimizing all data at once, efficient calculations are possible by selectively retaining and updating only the nodes with the greatest impact (in other words, the highest amount of information). This makes it possible to generate highly accurate maps in a realistic processing time even in large-scale environments.

[0158] Graph optimization-based SLAM algorithms fall into four categories:

[0159] Least Squares Method A linear or non-linear least squares method is used to adjust the node positions so that the sum of the errors is minimized.

[0160] ·Relaxation method This is a type of least squares method in which the position is adjusted to "relax" the tension along the edge.

[0161] Stochastic Gradient Descent This method randomly selects local errors rather than the overall error and updates them sequentially.

[0162] Manifold optimization This is a method for formulating the optimization target on a manifold in order to handle non-Euclidean spaces such as rotation. It is effective in estimating position and orientation (direction), especially in situations where accurate handling of rotation representation is required.

[0163] Furthermore, the data from multiple 3D LIDAR sensors are integrated to construct local submaps, which are then combined by a submap matching algorithm to achieve global optimization. This hierarchical approach significantly improves computational efficiency and scalability.

[0164] As a result, SLAM systems using graph optimization can build 2D and 3D maps with high accuracy and efficiency, and can be highly robust even in dynamic and complex environments.

[0165] (Loop Closure) In the present invention, the loop closure detection algorithm is improved.

[0166] Conventional loop closure detection methods mainly use handcrafted features to determine the similarity with previously visited locations. However, this method has issues such as low feature extraction accuracy and vulnerability to environmental changes and sensor noise. Furthermore, it also has issues such as high computational costs due to point-by-point comparisons of large amounts of data.

[0167] On the other hand, loop closure detection using deep learning can automatically learn and extract features with high discriminatory power while being robust to unwanted variables such as noise, lighting, and viewpoint changes, thanks to abundant training data and appropriate network design. Any of the following deep learning-based loop closure detection methods may be used:

[0168] 1. PointNetVLAD: Point Cloud-Based End-to-End Feature Extraction PointNetVLAD is an end-to-end global descriptor extraction method for 3D point clouds. It is designed in two stages: PointNet extracts local geometric features and NetVLAD aggregates them into global features.

[0169] This allows us to generate high-dimensional and highly discriminative descriptors directly from 3D point cloud data and apply them to loop closure detection.

[0170] 2. DH3D (Deep Hierarchical 3D) DH3D is a method that integrates local features learned hierarchically from 3D point clouds into a global descriptor using a dedicated feature aggregation mechanism. By simultaneously capturing detailed geometric information at the local level and spatial context at the global level, it enables more robust loop closure detection.

[0171] 3. MinkLoc3D: Support for sparse point clouds MinkLoc3D is a lightweight and highly accurate global descriptor extraction method that utilizes Minkowski encoding, which can process point clouds as sparse tensors. Its strength lies in its robustness against different sensor densities and scanning conditions.

[0172] 4.PPT-Net(Pyramid Point Transformer Network) PPT-Net is a point cloud processing network that incorporates a transformer architecture and extracts local features at multiple scales using a pyramidal structure. It excels in aggregating spatial contextual information and can generate more accurate global descriptors.

[0173] 5.FastLCD (Fast Loop Closure Detection) FastLCD is a lightweight framework that extracts multimodal features (geometry, structure, etc.) directly from 3D point clouds and efficiently detects loop closures. Its rotation-invariant feature representation makes it highly robust against pose changes.

[0174] 6. NDT-Transformer: Fusion with normal distribution transformation NDT-Transformer uses normal distribution transform (NDT) as a preprocessing method to learn global descriptors through a lightweight transformer module. It is scalable to large-scale point clouds and can detect loop closures with high accuracy.

[0175] 7. Overlap-Transformer: Depth Image-Based Descriptor Generation The Overlap-Transformer uses a lightweight transformer model to learn the similarity of overlapping regions from depth images and generate a global descriptor, which is particularly effective in detecting loops in the presence of partial overlaps.

[0176] 8. Image-based Loop Closure Detection In addition to traditional point cloud-based methods, image feature descriptors (e.g., JT128) may also be used, which are very fast and accurate and can also help with long-term, stable self-localization.

[0177] 9. Semantics-based Loop Closure Detection Semantic information-based methods utilize object categories, semantic labels, and their spatial arrangement to accurately evaluate the possibility of loop closure. A typical example is the SSC method below.

[0178] · SSC (Semantic Scan Context) We assign semantic labels to point cloud scans and integrate them with scan context to build a global descriptor. A three-stage hierarchical search enables more efficient loop closure detection.

[0179] (Action and effect) According to the transport vehicle guidance method, transport vehicle guidance device 1, and transport vehicle guidance program of the embodiment, the travel of the transport vehicle 9 is controlled within the warehouse 100 using 3D point cloud data of the ceiling and 3D point cloud data of the front. Therefore, positioning navigation can be realized by constructing and using 3D point cloud data / ceiling map including the ceiling, whose environmental characteristics are unlikely to change and are relatively stable (note that there is no need to install additional markers on the ceiling). This makes it possible to appropriately control the guided travel of the self-propelled transport vehicle 9 even in situations where environmental characteristics are changing, and ultimately makes it possible to improve the stability of the transport vehicle 9's operations and increase its reliability.

[0180] The above describes the embodiments of the present invention, but the specific configuration of the present invention is not limited to the above embodiments, and the present invention also includes forms in which modifications and changes are made to the above embodiments within the scope of the gist of the present invention. [Industrial Applicability]

[0181] The vehicle guidance method, guidance device, and guidance program of the present invention can appropriately control the guidance of a self-propelled material handling and transport vehicle even in situations where environmental characteristics change, and are therefore useful, for example, in the field of technology for automatically driving material handling and transport vehicles within a facility. [Explanation of symbols]

[0182] 1. Guidance device for transport vehicles 2 Ceiling detector 3 Forward detection unit 5 Automatic control unit 51 Volatile memory unit 52 Non-volatile memory unit 53 Guidance Program 6 Control Unit 61 Travel control unit 9 Transport vehicles 100 warehouse 101 Entrance / exit 103 Cargo storage area 104 Wall 105 pillars 200 transport vehicles 201 Cargo bed 300 luggage 301 Palette 400 Loading Yard

Claims

1. A method for controlling travel of a transport vehicle in a structure having a ceiling and in which items stored therein are subject to change, using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of a front area, From the three-dimensional point cloud data of the ceiling, a three-dimensional point cloud corresponding to a vertical or approximately vertical surface (referred to as a "ceiling-continuous vertical surface") that is a surface of a portion of a structure that extends downward and is continuous with the ceiling, is extracted; and extracting, from the three-dimensional point cloud data of the front, a three-dimensional point cloud corresponding to a surface of a portion of a structure that is continuous with a floor of the structure and extends upward, the surface being vertical or approximately vertical (referred to as a "floor-continuous vertical surface"); If the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous floor vertical plane match or are approximately matched, it is determined that the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the three-dimensional point cloud corresponding to the continuous floor vertical plane are point clouds relating to the same structure and are three-dimensional point clouds that are in a corresponding relationship in the two three-dimensional point cloud data; synthesizing the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area based on the combination of three-dimensional point clouds that correspond to each other in the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area; and estimating a self-position based on the three-dimensional point cloud data of the ceiling and avoiding obstacles based on the three-dimensional point cloud data of the front. A method for guiding a transport vehicle.

2. In the vertical direction, there is a blank area where no three-dimensional point cloud exists between the lower end of the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling and the upper end of the three-dimensional point cloud corresponding to the continuous vertical surface of the floor.

2. The method for guiding a transport vehicle according to claim 1.

3. When extracting the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling in response to input of the three-dimensional point cloud data of the ceiling, using as training data the three-dimensional point clouds corresponding to the surfaces of the walls and columns that are structures that extend downward in continuity with the ceiling, and When extracting the three-dimensional point cloud corresponding to the continuous floor vertical plane, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous floor vertical plane in the front three-dimensional point cloud data in response to input of the front three-dimensional point cloud data, using as training data three-dimensional point clouds corresponding to the surfaces of walls and columns that are structures extending upward in continuity with the floor.

2. The method for guiding a transport vehicle according to claim 1.

4. The structure is a warehouse and the transport vehicle is a forklift.

4. The method for guiding a transport vehicle according to claim 1.

5. a means for controlling travel of a transport vehicle in a structure having a ceiling and in which objects stored therein are moved, using the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of a front area; From the three-dimensional point cloud data of the ceiling, a three-dimensional point cloud corresponding to a vertical or approximately vertical surface (referred to as a "ceiling-continuous vertical surface") that is a surface of a portion of a structure that extends downward and is continuous with the ceiling, is extracted; and extracting, from the three-dimensional point cloud data of the front, a three-dimensional point cloud corresponding to a surface of a portion of a structure that is continuous with a floor of the structure and extends upward, the surface being vertical or approximately vertical (referred to as a "floor-continuous vertical surface"); If the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous floor vertical plane match or are approximately matched, it is determined that the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the three-dimensional point cloud corresponding to the continuous floor vertical plane are point clouds relating to the same structure and are three-dimensional point clouds that are in a corresponding relationship in the two three-dimensional point cloud data; synthesizing the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area based on the combination of three-dimensional point clouds that correspond to each other in the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area; the means estimates a self-position based on the three-dimensional point cloud data of the ceiling, and avoids obstacles based on the three-dimensional point cloud data of the front. A transport vehicle guidance device characterized by:

6. In the vertical direction, there is a blank area where no three-dimensional point cloud exists between the lower end of the three-dimensional point cloud corresponding to the continuous vertical plane of the ceiling and the upper end of the three-dimensional point cloud corresponding to the continuous vertical plane of the floor.

6. The transport vehicle guidance device according to claim 5.

7. When extracting the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling in response to input of the three-dimensional point cloud data of the ceiling, using as training data the three-dimensional point clouds corresponding to the surfaces of the walls and columns that are structures that extend downward in continuity with the ceiling, and When extracting the three-dimensional point cloud corresponding to the continuous floor vertical plane, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous floor vertical plane in the front three-dimensional point cloud data in response to input of the front three-dimensional point cloud data, using as training data three-dimensional point clouds corresponding to the surfaces of walls and columns that are structures extending upward in continuity with the floor.

6. The transport vehicle guidance device according to claim 5.

8. a program that causes an information processing device to execute a process of controlling the travel of a transport vehicle in a structure that has a ceiling and in which items stored therein are movable, using three-dimensional point cloud data of the ceiling and three-dimensional point cloud data of a front area, From the three-dimensional point cloud data of the ceiling, a three-dimensional point cloud corresponding to a vertical or approximately vertical surface (referred to as a "ceiling-continuous vertical surface") that is a surface of a portion of a structure that extends downward and is continuous with the ceiling, is extracted; and extracting, from the three-dimensional point cloud data of the front, a three-dimensional point cloud corresponding to a surface of a portion of a structure that is continuous with a floor of the structure and extends upward, the surface being vertical or approximately vertical (referred to as a "floor-continuous vertical surface"); If the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the values ​​of the planar position coordinates of the three-dimensional point cloud corresponding to the continuous floor vertical plane match or are approximately matched, it is determined that the three-dimensional point cloud corresponding to the continuous ceiling vertical plane and the three-dimensional point cloud corresponding to the continuous floor vertical plane are point clouds relating to the same structure and are three-dimensional point clouds that are in a corresponding relationship in the two three-dimensional point cloud data; synthesizing the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area based on the combination of three-dimensional point clouds that correspond to each other in the three-dimensional point cloud data of the ceiling and the three-dimensional point cloud data of the front area; and estimating a self-position based on the three-dimensional point cloud data of the ceiling and avoiding obstacles based on the three-dimensional point cloud data of the front. A transport vehicle guidance program comprising:

9. In the vertical direction, there is a blank area where no three-dimensional point cloud exists between the lower end of the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling and the upper end of the three-dimensional point cloud corresponding to the continuous vertical surface of the floor.

9. The transport vehicle guidance program according to claim 8.

10. When extracting the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous vertical surface of the ceiling in response to input of the three-dimensional point cloud data of the ceiling, using as training data the three-dimensional point clouds corresponding to the surfaces of the walls and columns that are structures that extend downward in continuity with the ceiling, and When extracting the three-dimensional point cloud corresponding to the continuous floor vertical plane, a trained model is used that has been trained in advance to extract the three-dimensional point cloud corresponding to the continuous floor vertical plane in the front three-dimensional point cloud data in response to input of the front three-dimensional point cloud data, using as training data three-dimensional point clouds corresponding to the surfaces of walls and columns that are structures extending upward in continuity with the floor.

9. The transport vehicle guidance program according to claim 8.

Images