Cargo Handling System

Abstract

To detect a position of a palette.SOLUTION: A cargo handling vehicle is equipped with a cargo handling device and an external sensor. The external sensor detects a position of an object using coordinates of a three-dimensional coordinate system. The control device detects positions of palettes and the number of the palettes, from point cloud data that is an assembly of points representing the position of the object. The control device determines the palette on which a cargo is placed.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to a material handling system. [Background technology]

[0002] Patent Document 1 discloses a cargo handling vehicle that automatically retrieves cargo under the control of a control device. The cargo handling vehicle disclosed in Patent Document 1 moves to a cargo retrieval position while estimating its own position. Once at the cargo retrieval position, the cargo handling vehicle retrieves the cargo. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-160885 Summary of the Invention [Problem to be solved by the invention]

[0004] When a cargo handling vehicle picks up a load, it needs to move close to the pallet, and in order to move the cargo handling vehicle close to the pallet, it is necessary to detect the position of the pallet. [Means for solving the problem]

[0005] A loading and unloading system that solves the above problem is a loading and unloading system that includes an external sensor that detects the position of an object using coordinates in a three-dimensional coordinate system, a loading and unloading vehicle equipped with a loading and unloading device, and a control device, wherein the control device detects the position of a pallet from point cloud data, which is a collection of points that represent the position of the object, and determines the pallet to be unloaded.

[0006] The points in the point cloud data represent the positions of objects, so the control device can detect the position of the pallet from the point cloud data. In the above-mentioned cargo handling system, the control device may create a side view of the pallet from the point cloud data and detect the position and number of the pallet by extracting the pallet from the side view using image recognition.

[0007] In the above-mentioned cargo handling system, the control device may extract the driver's seat of the transport vehicle carrying the pallet from the side view by image recognition, and determine that the pallet farthest from the driver's seat is the pallet to be unloaded. [Effects of the Invention]

[0008] According to the present invention, the position of the pallet can be detected. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a schematic diagram of an area in which a loading vehicle operates; [Figure 2] FIG. 2 is a perspective view of a transport vehicle and a loading vehicle. [Figure 3] FIG. 1 is a schematic configuration diagram of a cargo handling vehicle. [Figure 4] 10 is a flowchart showing palette detection control. [Figure 5] FIG. 1 is a schematic diagram of a point cloud map. [Figure 6] FIG. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the cargo handling system will be described below. As shown in FIG. 1, a stopping position PS1 is set in the area A1. A transport vehicle 10 stops at the stopping position PS1. The area of ​​the stopping position PS1 is larger than the area of ​​the transport vehicle 10 when viewed from above when stopped at the stopping position PS1. No structures such as pillars are present at the stopping position PS1. The area A1 may be, for example, the entire area or a part of a place such as a factory, port, airport, commercial facility, or public facility. A cargo handling vehicle 20 is operated in the area A1. The cargo handling vehicle 20 performs loading and unloading. Loading is the work of loading a load C1 placed on a pallet PA1 onto the transport vehicle 10. Unloading is the work of removing the pallet PA1 loaded on the transport vehicle 10 and the load C1 placed on the pallet PA1 from the transport vehicle 10. In the following description, the front, back, left, right, top, and bottom of the transport vehicle 10 refer to the front, back, left, right, top, and bottom when the transport vehicle 10 is used as the reference. The left, right, top and bottom of the cargo handling vehicle 20 refer to the left, right, top and bottom when the cargo handling vehicle 20 is used as the reference.

[0011] 2, the transport vehicle 10 is a wing truck. The transport vehicle 10 may be any type of truck, such as a flatbed truck. The transport vehicle 10 includes a driver's seat 11, a front panel 12, a rear door 13, a loading platform 14, wing side panels 16, and a tailgate 17.

[0012] The driver's seat 11 is a position where the driver of the transport vehicle 10 sits. The front panel 12 is located further rearward of the driver's seat 11 on the transport vehicle 10. The front panel 12 is located adjacent to the driver's seat 11. The rear door 13 is located further rearward of the front panel 12 on the transport vehicle 10. The front panel 12 and the rear door 13 are located at a distance from each other in the fore-and-aft direction of the transport vehicle 10. The loading platform 14 extends in the fore-and-aft direction of the transport vehicle 10 between the front panel 12 and the rear door 13. The loading platform 14 has a loading surface 15. The loading surface 15 is the upper surface of the loading platform 14. A load C1 placed on a pallet PA1 is loaded on the loading surface 15. The wing side panel 16 is located between the front panel 12 and the rear door 13. The wing side panel 16 is located so as to be rotatable in the up-and-down direction of the transport vehicle 10 around the center position of the transport vehicle 10 in the vehicle width direction. The wing side panels 16 are provided one on each side in the vehicle width direction of the transport vehicle 10. The tailgates 17 are provided so as to extend in the front-to-rear direction of the transport vehicle 10. The tailgates 17 are provided along the edges of the loading platform 14, which are the edges that extend in the front-to-rear direction of the transport vehicle 10. The tailgates 17 are provided one on each side in the vehicle width direction of the transport vehicle 10.

[0013] <Loading vehicle> The cargo handling vehicle 20 includes a vehicle body 21, drive wheels 22, steering wheels 23, and a cargo handling device 24. The cargo handling device 24 is provided at the front of the vehicle body 21. The cargo handling device 24 includes a mast 25, a lift cylinder 28, a lift bracket 29, and a fork 30.

[0014] The mast 25 includes an outer mast 26 and an inner mast 27. The inner mast 27 is provided so as to be able to move up and down relative to the outer mast 26. The forks 30 are fixed to lift brackets 29. The lift brackets 29 and the forks 30 move up and down together with the inner mast 27. The lift cylinders 28 raise and lower the inner mast 27. The lift cylinders 28 are hydraulic cylinders.

[0015] 3, the cargo handling vehicle 20 includes an external sensor 51, a control device 52, an auxiliary storage device 55, a vehicle control device 56, a travel actuator 59, a cargo handling actuator 60, and a camera 61. The cargo handling vehicle 20 is a cargo handling system.

[0016] The external sensor 51 detects the position of an object using coordinates in a three-dimensional coordinate system. The external sensor 51 is provided on the upper part of the cargo handling vehicle 20. For example, the external sensor 51 is provided on a head guard of the cargo handling vehicle 20.

[0017] Examples of the external sensor 51 include a millimeter-wave radar, a stereo camera, a ToF (Time of Flight) camera, and a LIDAR (Laser Imaging Detection and Ranging). In this embodiment, a LIDAR is used as the external sensor 51. The external sensor 51 irradiates the surroundings with a laser and receives light reflected from the point where the laser hits, thereby deriving the distance to the point. The point where the laser hits represents a part of the surface of the object. The position of the point can be expressed by coordinates in a polar coordinate system. The coordinates of the point in the polar coordinate system are converted into coordinates in a Cartesian coordinate system. The conversion from the polar coordinate system to the Cartesian coordinate system may be performed by the external sensor 51 or by the control device 52. In this embodiment, it is assumed that the conversion from the polar coordinate system to the Cartesian coordinate system is performed by the external sensor 51. The external sensor 51 derives the coordinates of the point in the sensor coordinate system. The sensor coordinate system is a three-axis Cartesian coordinate system with the external sensor 51 as the origin. The external sensor 51 outputs the coordinates of a plurality of points obtained by irradiating the laser to the control device 52 as point cloud data.

[0018] The control device 52 includes a processor 53 and a storage unit 54. The storage unit 54 includes a random access memory (RAM) and a read-only memory (ROM). The storage unit 54 stores program code or instructions configured to cause the processor 53 to execute processes. The storage unit 54, i.e., a computer-readable medium, includes any available medium accessible by a general-purpose or special-purpose computer. The control device 52 may be configured with a hardware circuit such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The control device 52, which is a processing circuit, may include one or more processors operating according to a computer program, one or more hardware circuits such as an ASIC or FPGA, or a combination thereof.

[0019] The auxiliary storage device 55 stores information that can be read by the control device 52. Examples of the auxiliary storage device 55 include a hard disk drive and a solid state drive. The auxiliary storage device 55 stores an environmental map D1. The auxiliary storage device 55 stores a first image recognition model D2. The auxiliary storage device 55 stores a second image recognition model D3.

[0020] The environmental map D1 is information about the physical structure of area A1, such as the shapes of objects present in area A1 and the size of area A1. In this embodiment, the environmental map D1 is data that represents the structure of area A1 using coordinates in a map coordinate system. The map coordinate system is a three-axis Cartesian coordinate system. The map coordinate system is a coordinate system that has an arbitrary point in area A1 as its origin. In the map coordinate system, the horizontal direction is defined by the X-axis and Y-axis, which are perpendicular to each other. The XY plane defined by the X-axis and Y-axis can be said to represent the horizontal plane. In the map coordinate system, the up-down direction is defined by the Z-axis, which is perpendicular to the X-axis and Y-axis. Coordinates in the map coordinate system will be referred to as map coordinates where appropriate. The map coordinate system is a three-dimensional coordinate system that represents three-dimensional positions.

[0021] The control device 52 estimates the self-position of the cargo handling vehicle 20. The control device 52 outputs the self-position of the cargo handling vehicle 20 to the vehicle control device 56. The self-position is the position of the cargo handling vehicle 20 on the environmental map D1. The self-position is a coordinate indicating a point of the cargo handling vehicle 20 in the map coordinate system. The point of the cargo handling vehicle 20 is arbitrary, but can be, for example, the center position of the cargo handling vehicle 20 in the horizontal direction.

[0022] The self-location is estimated by comparing the detection results of the external sensor 51 with the environmental map D1. The control device 52 extracts landmarks from the environmental map D1 that have the same shape as the landmarks obtained from the point cloud data. The control device 52 recognizes the positions of the landmarks from the environmental map D1. The positional relationship between the landmark positions and the cargo handling vehicle 20 can be determined from the detection results of the external sensor 51. Therefore, the control device 52 can estimate the self-location by recognizing the positions of the landmarks. A landmark is an object that has characteristics that can be identified by the external sensor 51. A landmark is a physical structure whose position is unlikely to change. Examples of landmarks include walls and pillars. The self-location may be estimated by combining the self-location estimation using the external sensor 51 with dead reckoning using an internal sensor. The self-location may be estimated by combining the self-location estimation using the external sensor 51 with satellite signals transmitted from GNSS (Global Navigation Satellite System) satellites.

[0023] The vehicle control device 56 has, for example, the same hardware configuration as the control device 52. The vehicle control device 56 has a processor 57 and a storage unit 58. The travel actuator 59 is an actuator that causes the cargo handling vehicle 20 to travel. The travel actuator 59 includes, for example, a motor that rotates the drive wheels 22 and a steering mechanism. The vehicle control device 56 controls the travel actuator 59 while determining its own position to cause the cargo handling vehicle 20 to travel.

[0024] The cargo handling actuator 60 is an actuator that causes the cargo handling vehicle 20 to handle cargo. The cargo handling actuator 60 includes, for example, a motor that drives a pump that supplies hydraulic oil to hydraulic equipment, and a control valve that controls the supply of hydraulic oil. The hydraulic equipment includes the lift cylinder 28. The vehicle control device 56 controls the cargo handling actuator 60 to raise and lower the forks 30.

[0025] The cargo handling vehicle 20 travels automatically by controlling a travel actuator 59 by a vehicle control device 56. The cargo handling vehicle 20 automatically handles cargo by controlling a cargo handling actuator 60 by a vehicle control device 56. The cargo handling vehicle 20 is an automatically operated forklift.

[0026] The camera 61 is disposed so as to capture an image in front of the cargo handling vehicle 20. The camera 61 is a monocular camera. The camera 61 is disposed, for example, between the two forks 30 in the vehicle width direction of the cargo handling vehicle 20. The camera 61 is disposed so as to rise and fall together with the forks 30. For example, the camera 61 is attached to the lift bracket 29.

[0027] The pallet detection control performed by the control device 52 will now be described. The pallet detection control is performed to move the cargo handling vehicle 20 closer to the pallet PA1 when the cargo handling vehicle 20 picks up the cargo. The cargo handling vehicle 20 moves closer to the transport vehicle 10, for example, based on a command from a higher-level control device. Then, when the distance between the cargo handling vehicle 20 and the transport vehicle 10 becomes less than a predetermined distance, the pallet detection control is started. The predetermined distance is, for example, a distance that can be detected by the transport vehicle 10 using the external sensor 51.

[0028] <Pallet detection control> As shown in FIGS. 4 and 5, in step S1, the control device 52 creates a point cloud map PM1. The point cloud map PM1 is created by overlaying point cloud data obtained from the detection results of the external sensor 51. The control device 52 acquires point cloud data from the external sensor 51 multiple times while the loading vehicle 20 is moving. The control device 52 converts the coordinates of each point P1 in the point cloud data from the coordinates of the sensor coordinate system into map coordinates based on its own position. The control device 52 can recognize the origin of the sensor coordinate system in the map coordinate system based on its own position estimated by the control device 52. The control device 52 can recognize the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system based on the origin of the sensor coordinate system in the map coordinate system and the deviation between the coordinate axes of the map coordinate system and the coordinate axes of the sensor coordinate system. The control device 52 creates the point cloud map PM1 by overlaying each point P1 converted into map coordinates each time point cloud data is acquired. The point cloud map PM1 is a collection of point cloud data. Compared to the points P1 of the point cloud data, the points P1 of the point cloud map PM1 are denser.

[0029] FIG. 5 shows the point cloud map PM1 obtained by the processing of step S1. Each point P1 included in the point cloud map PM1 represents the map coordinates of an object. For ease of explanation, the points P1 included in the point cloud map PM1 will be classified into a first point P11, a second point P12, a third point P13, and a fourth point P14. The first point P11 is a point P1 obtained by irradiating the gate 17 with a laser. The second point P12 is a point P1 obtained by irradiating the driver's seat 11 with a laser. The third point P13 is a point P1 obtained by irradiating the pallet PA1 and the load C1 loaded on the loading surface 15 with a laser. The fourth point P14 is a point P1 that does not fall into any of the first point P11, the second point P12, and the third point P13.

[0030] As shown in Figures 4 and 6, in step S2, the control device 52 creates a side view IM1 from the point cloud map PM1. The side view IM1 is a view in which point P1 is projected in the vehicle width direction of the transfer vehicle 10. This makes it possible to make the point cloud map PM1 two-dimensional, so that the side view IM1 can be treated as image data. The side view IM1 can also be said to be a side view of the pallet PA1. An example of processing performed by the control device 52 when creating the side view IM1 will be described.

[0031] The control device 52 calculates the normal vector of each point P1. A normal vector is a vector oriented in a direction perpendicular to a plane surrounded by multiple points P1. Methods for deriving a normal vector include a method of determining a curved surface from each point P1 and deriving the normal vector of each point P1 from the curved surface, and a method using the cross product of vectors. For example, when the control device 52 determines the normal vector of one point P1, it determines the cross product of vectors directed from this point P1 to each of two points P1 located within a predetermined range. This cross product is the normal vector.

[0032] The control device 52 extracts a point P1 where the normal vector is oriented horizontally. For example, the control device 52 determines whether the angle of the normal vector with respect to the XY plane of the map coordinate system is within a predetermined range. If the normal vector is oriented horizontally, the normal vector is parallel to the XY plane of the map coordinate system. Taking into consideration the inclination of the transfer vehicle 10 and measurement errors, the predetermined range is set so that the point P1 where the normal vector can be considered to be oriented horizontally can be extracted.

[0033] The control device 52 derives a plane equation from point P1, whose normal vector is oriented horizontally. The plane equation can be derived using, for example, a robust estimation method such as RANSAC (Random Sample Consensus) or the least squares method. The plane represented by the plane equation is a surface that extends in the vertical direction. In the case of the transport vehicle 10 of this embodiment, the plane can be obtained from the first point P11 defined by the gate 17. Furthermore, when loading an item, since the transport vehicle 10 is loaded with a pallet PA1 and an item C1, the plane can be obtained from the third point P13 defined by the pallet PA1 and the item C1.

[0034] The control device 52 creates a side view IM1 by projecting point P1 in a direction perpendicular to the plane represented by the plane equation. The control device 52 creates the side view IM1 by projecting point P1 within a predetermined range horizontally from the plane represented by the plane equation. As shown in FIG. 1 , the transport vehicle 10 may be loaded with pallets PA1 arranged in the width direction of the transport vehicle 10. The predetermined range PT1 is set so as not to include pallets PA1 arranged in the width direction of the transport vehicle 10 that are far from the cargo handling vehicle 20. The predetermined range PT1 is set so as to include the surfaces of pallets PA1 arranged in the width direction of the transport vehicle 10 that are close to the cargo handling vehicle 20, facing the cargo handling vehicle 20. When the cargo handling vehicle 20 retrieves cargo, it retrieves the pallets PA1 arranged in the width direction of the transport vehicle 10 that are close to the cargo handling vehicle 20. By setting the predetermined range PT1 as described above, a side view IM1 can be obtained that excludes pallets PA1 that are not to be retrieved by the cargo handling vehicle 20.

[0035] As shown in Figures 4 and 6, next, in step S3, the control device 52 detects the position of pallet PA1 and the number of pallets PA1. The position of pallet PA1 is the position of pallet PA1 that is the object of loading by the cargo handling vehicle 20. The number of pallets PA1 is the number of pallets PA1 that are the object of loading by the cargo handling vehicle 20. The position of pallet PA1 is the coordinate of pallet PA1 in the map coordinate system. The position of pallet PA1 can be obtained by converting the coordinate of pallet PA1 in the side view IM1 into coordinates in the map coordinate system. This will be explained in detail below.

[0036] The control device 52 extracts the pallet PA1 from the side view IM1 to determine the position of the pallet PA1 in the side view IM1. The position of the pallet PA1 in the side view IM1 can be represented by an image coordinate system. The image coordinate system is a coordinate system in which the horizontal direction of the side view IM1 is the X axis and the vertical direction is the Y axis. The position of the pallet PA1 is determined using image recognition. In this embodiment, a case will be described in which image recognition is performed using the first image recognition model D2, but image recognition may also be performed by pattern matching.

[0037] The first image recognition model D2 is a trained model generated by machine learning. The first image recognition model D2 uses an algorithm capable of determining the class of an object on a region-by-region basis. The class is set to "palette." Examples of machine learning algorithms include SSD (Single Shot Multibox Detector), R-CNN (Regional Convolutional Neural Network), fast R-CNN, faster R-CNN, and YOLO (You Only Look Once). The first image recognition model D2 is generated by supervised learning or semi-supervised learning using training data. The training data includes image data containing an object corresponding to the class, the position of the object in the image data, and a label. The training data can be generated, for example, by enclosing the object in the image data in a frame and labeling the image data. In this embodiment, the training data may be obtained by enclosing the palette PA1 of the image data in a frame and labeling the image data with "palette." The image data used as training data may be obtained from the point cloud map PM1, as with the side view IM1, or may be obtained by capturing an image using an imaging device.

[0038] The first image recognition model D2 identifies the area containing pallet PA1 from the input side view IM1. The area containing pallet PA1 is represented by bounding boxes B1 and B2. In the example shown in FIG. 6, two bounding boxes B1 and B2 are extracted from the side view IM1. The bounding boxes B1 and B2 surround pallet PA1, which is represented by the third point P13. The number of bounding boxes B1 and B2 represents the number of pallets PA1. The positions of the bounding boxes B1 and B2 in the side view IM1 represent the position of pallet PA1 in the image coordinate system. Because the side view IM1 is created by projecting point P1 in the map coordinate system, coordinates in the map coordinate system are associated with point P1 constituting the side view IM1. The control device 52 can detect the position of pallet PA1 in the map coordinate system from the position of pallet PA1 in the side view IM1. Because the side view IM1 is created from point cloud data, it can be said that the control device 52 detects the position of pallet PA1 and the number of pallets PA1 from the point cloud data.

[0039] Next, in step S4, the control device 52 determines the position of the driver's seat 11 in the side view IM1. The determination of the position of the driver's seat 11 in the side view IM1 is performed using image recognition. In this embodiment, a case where image recognition is performed using the second image recognition model D3 will be described, but image recognition may also be performed by pattern matching.

[0040] The second image recognition model D3 is a trained model generated by machine learning. The second image recognition model D3 is generated by the same method as the first image recognition model D2. In the second image recognition model D3, "driver's seat" is set as a class. As a result, the second image recognition model D3 identifies an area including the driver's seat 11 from the input side view IM1. In the example shown in FIG. 6, a bounding box B3 is extracted from the side view IM1. In the example shown in FIG. 6, the driver's seat 11 represented by the second point P12 is surrounded by the bounding box B3. The bounding box B3 represents the position of the driver's seat 11 in the side view IM1.

[0041] Next, in step S5, the control device 52 determines the pallet PA1 to be picked up. When there are multiple pallets PA1, it can be said that the control device 52 determines the pallet PA1 to be picked up first. When the cargo handling vehicle 20 picks up cargo, it picks up the pallets PA1 loaded on the loading platform 14 of the transport vehicle 10 sequentially from the rear to the front of the transport vehicle 10. The control device 52 determines that the pallet PA1 farthest from the driver's seat 11 is the pallet PA1 to be picked up first. In this embodiment, the pallet PA1 represented by the bounding box B1 is the pallet PA1 to be picked up first. The pallet PA1 represented by the bounding box B2 is the pallet PA1 to be picked up second.

[0042] Next, in step S6, the control device 52 transmits the coordinates of the pallet PA1 in the map coordinate system, i.e., the position of the pallet PA1, to the vehicle control device 56. The control device 52 may transmit only the position of the pallet PA1 that is to be picked up first to the vehicle control device 56. The control device 52 may also transmit the positions of all detected pallets PA1 to the vehicle control device 56.

[0043] After completing the process of step S6, the control device 52 ends the pallet detection control. The vehicle control device 56 controls the travel actuator 59 in accordance with the position of the pallet PA1 transmitted from the control device 52. The vehicle control device 56 generates a route to the pallet PA1 from which the load is to be first removed. At this time, the vehicle control device 56 generates a route to an approach start position that is a predetermined distance before the pallet PA1. The predetermined distance is set to a distance that allows the camera 61 to capture an image of the pallet PA1. The vehicle control device 56 controls the travel actuator 59 so that the cargo handling vehicle 20 moves along the route. When the cargo handling vehicle 20 reaches the approach start position, the vehicle control device 56 moves the cargo handling vehicle 20 closer to the pallet PA1 while checking the position of the pallet PA1 using image data obtained by the camera 61. When the cargo handling vehicle 20 reaches a position where the pallet PA1 will be removed, the vehicle control device 56 controls the cargo handling actuator 60 to remove the load. The vehicle control device 56 checks the position of the pallet PA1 using image data obtained by the camera 61 and adjusts the position of the fork 30 to match the pallet PA1. This allows the cargo handling vehicle 20 to remove the load.

[0044] [Effects of this embodiment] (1) The control device 52 can detect the position of the pallet PA1 from the point cloud data, thereby determining the pallet PA1 to be picked up.

[0045] (2) The control device 52 can detect the position of the pallet PA1 from the point cloud data. This allows the vehicle control device 56 to generate a route toward the pallet PA1. The camera 61 is used to move the cargo handling vehicle 20 from the approach start position to the position where the pallet PA1 is unloaded. Because the detection range of the camera 61 is limited, it is necessary to move the cargo handling vehicle 20 to the approach start position with high accuracy. By detecting the position of the pallet PA1 from the point cloud data and generating a route, the cargo handling vehicle 20 can be moved to the approach start position with high accuracy.

[0046] It is also possible to set a route in advance and move the loading vehicle 20 to a position where the pallet PA1 is to be unloaded. However, in this case, it is necessary to set a route for each size of the transport vehicle 10, size of the pallet PA1, and type of pallet PA1, which would require a huge amount of work. In contrast, by detecting the position of the pallet PA1 from point cloud data and generating a route, it is possible to generate a route according to the position of the pallet PA1. This eliminates the need to set a route in advance.

[0047] (3) The control device 52 detects the position of pallet PA1 and the number of pallets PA1 from the side view IM1 through image recognition. By converting the point cloud data into the side view IM1, the side view IM1 can be treated as image data. This allows the control device 52 to detect the position of pallet PA1 in the side view IM1 and the number of pallets PA1 through image recognition. Then, from the position of pallet PA1 in the side view IM1, the position of pallet PA1 in the map coordinate system can be detected.

[0048] (4) The control device 52 extracts the driver's seat 11 from the side view IM1 by image recognition. When the cargo handling vehicle 20 picks up cargo, it sequentially picks up the cargo C1 loaded on the loading platform 14 of the transport vehicle 10 from the rear to the front of the transport vehicle 10. Because the driver's seat 11 of the transport vehicle 10 is located at the front of the transport vehicle 10, by extracting the driver's seat 11 of the transport vehicle 10, it can determine that the pallet PA1 farthest from the driver's seat 11 is the pallet PA1 to be picked up first.

[0049] (5) The position and number of pallets PA1 can be detected using the external sensor 51 used to estimate the self-position. This reduces the number of parts compared to when a dedicated sensor is used to detect the position and number of pallets PA1.

[0050] [Example of change] The embodiment can be modified as follows: The embodiment and the following modifications can be combined with each other to the extent that they are not technically inconsistent.

[0051] The control device 52 may detect the positions and number of pallets PA1 from the point cloud map PM1. In this case, for example, the positions and number of pallets PA1 are detected from the point cloud map PM1 using a model that has learned the features of the three-dimensional data representing the pallets PA1.

[0052] In step S3, the control device 52 only needs to detect the position of the pallet PA1, and does not need to detect the number of pallets PA1. The control device 52 does not have to create the point cloud map PM1. In this case, the control device 52 converts the point cloud data acquired from the external sensor 51 into map coordinates and performs the processes from step S2 onwards. That is, the control device 52 may detect the positions and number of pallets PA1 from a single point cloud data, without creating the point cloud map PM1 which is a collection of multiple point cloud data.

[0053] The vehicle control device 56 may generate a route to a position where the cargo handling vehicle 20 picks up the pallet PA1. In this case, image data obtained by capturing an image with the camera 61 is used to adjust the position of the fork 30.

[0054] If there is a space adjacent to the pallet PA1 in the horizontal direction, the control device 52 may determine that the pallet PA1 is the pallet PA1 to be loaded first. The pallet PA1 to be loaded first does not have any pallets PA1 adjacent to it in the rear direction of the transport vehicle 10. Therefore, the pallet PA1 to be loaded first has a space adjacent to it in the horizontal direction that is equal to or greater than the threshold. By detecting this space, the pallet PA1 to be loaded first can be determined. If there is cushioning material between adjacent pallets PA1, the threshold is set taking into account the thickness of the cushioning material. If there is no cushioning material between adjacent pallets PA1, the threshold may be set based on the width of the pallet PA1. For example, half the width of the pallet PA1 may be set as the threshold. The control device 52 may detect the space from the side view IM1 or from the point cloud map PM1. Because point P1 does not exist in the space, the control device 52 can detect the space based on the presence or absence of point P1.

[0055] The second image recognition model D3 may be an algorithm capable of determining the class of an object for each side view IM1. In this case, "right-facing" and "left-facing" may be set as classes. An example of the machine learning algorithm is a convolution neural network (CNN). As training data, data in which image data showing the transfer vehicle 10 facing right is labeled as "right-facing" and data in which image data showing the transfer vehicle 10 facing left is labeled as "left-facing" may be used.

[0056] The control device 52 inputs the side view IM1 into the second image recognition model D3, thereby determining whether the transport vehicle 10 shown in the side view IM1 is facing right or left. When the transport vehicle 10 is facing right, the control device 52 determines that the pallet PA1 located at the leftmost position in the image coordinate system is the pallet PA1 to be loaded. When the transport vehicle 10 is facing left, the control device 52 determines that the pallet PA1 located at the rightmost position in the image coordinate system is the pallet PA1 to be loaded.

[0057] When the parking direction of the transport vehicle 10 is constant, the pallet PA1 to be picked up may be determined based on the map coordinates of the pallet PA1. When the parking direction of the transport vehicle 10 is constant, the pallet PA1 loaded further rearward on the transport vehicle 10 will have a larger or smaller horizontal coordinate in the map coordinate system. Whether the pallet PA1 loaded further rearward on the transport vehicle 10 has a larger or smaller horizontal coordinate in the map coordinate system depends on the positional relationship between the origin of the map coordinate system and the transport vehicle 10. The control device 52 can determine the pallet PA1 to be picked up based on the magnitude of the map coordinates. For example, when the pallet PA1 loaded further rearward on the transport vehicle 10 has a larger X coordinate in the map coordinate system, the control device 52 can determine that the pallet PA1 with the largest X coordinate in the map coordinate system is the pallet PA1 to be picked up.

[0058] Part of the pallet detection control process may be performed by the vehicle control device 56. In this case, the vehicle control device 56 can also be considered a control device. A part of the processing of the pallet detection control may be performed by a host control device that issues commands to the cargo handling vehicle 20. In this case, the host control device can also be considered a control device. All of the processing of the pallet detection control may be performed by a host control device that issues commands to the cargo handling vehicle 20. In this case, the host control device is the control device. In this way, when at least a part of the processing of the pallet detection control is performed by the host control device, the control device 52 transmits data required for the processing to the host control device via wireless communication. For example, when all of the processing of the pallet detection control is performed by the host control device, the control device 52 transmits point cloud data to the host control device. The host control device transmits the results obtained by performing the pallet detection control to the control device 52 via wireless communication. The host control device is installed in, for example, area A1. In this case, the cargo handling system is made up of the cargo handling vehicle 20 and the host control device.

[0059] The cargo handling vehicle 20 may pick up the pallet PA1 placed on a shelf or on the ground. The transfer vehicle 10 may be any vehicle capable of performing transfer, such as an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot).

[0060] The cargo handling vehicle 20 may be provided with a laser range finder instead of the camera 61 as a sensor for adjusting the position of the fork 30 . The cargo handling device 24 may be equipped with, for example, a robot arm. [Explanation of symbols]

[0061] P1...point, PA1...pallet, 10...transport vehicle, 11...driver's seat, 20...cargo vehicle which is a cargo handling system, 24...cargo handling device, 51...external sensor, 52...control device.

Claims

1. a cargo handling vehicle equipped with an external sensor that detects the position of an object in coordinates of a three-dimensional coordinate system and a cargo handling device; A cargo handling system comprising: The control device extracting the points whose normal vectors are oriented horizontally from point cloud data, which is a set of points representing the positions of the objects; Deriving a plane equation from the point where the normal vector is oriented horizontally; creating a side view by projecting the points of the point cloud data within a predetermined range from the plane represented by the plane equation in a direction perpendicular to the plane; extracting pallets from the side view by image recognition to detect the positions of the pallets and the number of the pallets; A material handling system that determines the pallet to be picked up.

2. The control device extracting a driver's seat of the transport vehicle carrying the pallet from the side view by image recognition; The cargo handling system according to claim 1 , wherein the pallet farthest from the driver's seat is determined to be the pallet to be picked up.

Images