Handling system and handling vehicle

Abstract

To provide a cargo handling system and a cargo handling vehicle capable of reducing the risk of cargo collapse in a cargo handling object in which pallets and cargo are alternately stacked.SOLUTION: A cargo handling system 10 includes a cargo handling vehicle 11 for transporting a cargo handling object WA including a first pallet P1, a first cargo L1 loaded on the first pallet, a second pallet P2 loaded on the first cargo, and a second cargo L2 loaded on the second pallet. The cargo handling system has an imaging device 31 that images the cargo handling object, and a control unit 51 that stores a machine-learned model that outputs a protrusion value indicating a protrusion amount of the second pallet relative to the first pallet by inputting image data captured by the imaging device, and a protrusion threshold that is a threshold value for the protrusion value. The control unit inputs image data to the model, and when the protrusion value output by the model is greater than the protrusion threshold, it notifies that the protrusion value is larger than the protrusion threshold.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a cargo handling system and a cargo handling vehicle.

Background Art

[0002] Patent Document 1 describes a cargo handling system including a cargo handling vehicle that transports a cargo handling target including a cargo and a pallet on which the cargo is loaded. This cargo handling system determines the risk of collapse based on the bias of the load of the cargo handling target applied to the cargo handling vehicle when the cargo handling vehicle lifts the cargo handling target.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] If the cargo is loaded so as to protrude from the pallet, collapse is likely to occur. Therefore, usually, if the cargo is properly loaded on the pallet, collapse is less likely to occur. However, for a cargo handling target in which the pallet and the cargo are alternately loaded, collapse may be likely to occur even if the cargo is properly loaded on the pallet. For example, if the upper pallet protrudes from the lower pallet, collapse is likely to occur.

Means for Solving the Problems

[0005] The cargo handling system that solves the above problems includes a cargo handling vehicle that transports a cargo handling target including a first pallet, a first load stacked on the first pallet, a second pallet stacked on the first load, and a second load stacked on the second pallet. The cargo handling system includes an imaging device that images the cargo handling target, a machine learning model that outputs an overhang value indicating the overhang amount of the second pallet with respect to the first pallet when image data captured by the imaging device is input, and a control device that stores an overhang threshold that is a threshold value of the overhang value. The control device inputs the image data to the model, and when the overhang value output by the model is greater than the overhang threshold, notifies that the overhang value is greater than the overhang threshold.

[0006] According to the above configuration, the cargo handling system can obtain an overhang value by inputting image data to the model. The larger the overhang amount of the second pallet with respect to the first pallet, that is, the larger the overhang value, the more likely it is that the load will collapse. The cargo handling system can determine the possibility of load collapse by comparing the overhang value with the overhang threshold. When the overhang value is greater than the overhang threshold, that is, when the load is likely to collapse, the cargo handling system notifies that the overhang value is greater than the overhang threshold. Thereby, for example, the operator of the cargo handling vehicle, the administrator of the cargo handling system, etc. can grasp that the load is likely to collapse. For example, by the operator or the administrator stopping the transportation of the cargo handling target by the cargo handling vehicle or adjusting the position of the second pallet with respect to the first pallet, the possibility of load collapse is reduced.

[0007] In the above cargo handling system, the cargo handling vehicle is a forklift that operates autonomously, and includes a detection unit that detects the position of the first pallet and the position of the second pallet. When the overhang value is greater than the overhang threshold, the control device may move the second pallet to the cargo handling vehicle so that the overhang amount of the second pallet with respect to the first pallet becomes smaller.

[0008] According to the above configuration, when the overhang value is greater than the overhang threshold value, by moving the second pallet based on the detection result of the detection unit, the overhang amount of the second pallet with respect to the first pallet is reduced. Thereby, the possibility of load collapse is reduced.

[0009] In the above-mentioned handling system, the detection unit includes a laser sensor that emits a laser to the first pallet and the second pallet and receives the reflected laser. The laser sensor obtains point cloud data indicating the reflection points of the laser by receiving the laser reflected by the first pallet and the second pallet. The control device may detect the position of the first pallet and the position of the second pallet from the point cloud data. According to the above configuration, the control device can re-stack the second pallet on the handling vehicle with respect to the first pallet based on the point cloud data.

[0010] In the above-mentioned handling system, when the image data is input, the model outputs the overhang value and a classification value indicating whether any of the overhang amounts of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large. When the classification value indicates that any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large, the control device may notify that any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large.

[0011] If any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large, there is a risk of load collapse. In a handling vehicle, while the pallet and the load can be moved together by lifting the pallet, it is difficult to move the load alone. Therefore, if any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large, it is necessary to manually re-stack the load.

[0012] According to the above configuration, when the classification value indicates that either the overhang amount of the first package with respect to the first pallet or the overhang amount of the second package with respect to the first pallet is large, it notifies that either the overhang amount of the first package with respect to the first pallet is large or the overhang amount of the second package with respect to the first pallet is large. Thereby, for example, an operator of a cargo handling vehicle, an administrator of a cargo handling system, etc. can grasp that the first package or the second package protrudes greatly with respect to the first pallet. When the operator or administrator who has received this notification, for example, reloads the first package or the second package, the risk of cargo collapse is reduced.

[0013] The cargo handling vehicle that solves the above problems is a cargo handling vehicle that transports a cargo handling target including a first pallet, a first package loaded on the first pallet, a second pallet loaded on the first package, and a second package loaded on the second pallet, wherein the cargo handling vehicle includes an imaging device that images the cargo handling target, and a machine learning model that outputs an overhang value indicating the overhang amount of the second pallet with respect to the first pallet when image data captured by the imaging device is input, and a control device that stores an overhang threshold that is a threshold value of the overhang value. The control device inputs the image data into the model, and when the overhang value output by the model is larger than the overhang threshold, it notifies that the overhang value is larger than the overhang threshold. According to the above cargo handling vehicle, the same effect as the above-described cargo handling system can be obtained.

Effect of the Invention

[0014] According to the present invention, the risk of cargo collapse is reduced in a cargo handling target in which pallets and packages are alternately loaded.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0016] <Configuration of the Handling System> An example of a handling system including a handling vehicle will be described. As shown in FIG. 1, the handling system 10 includes a handling vehicle 11. The handling vehicle 11 is, for example, a forklift. The handling vehicle 11 constitutes the handling system 10.

[0017] The handling vehicle 11 is, for example, an autonomous forklift that operates autonomously. The handling vehicle 11 may be, for example, a manned forklift operated by a boarding operator. The handling vehicle 11 may be, for example, an unmanned forklift remotely operated by an operator. The handling vehicle 11 may be configured to be switchable between an autonomous operation that operates autonomously and a manual operation that operates by the operation of an operator. The handling vehicle 11 may be switched to a manual operation, for example, when it cannot cope with an autonomous operation.

[0018] The handling vehicle 11 transports, that is, conveys, the handling target WA. The handling vehicle 11 transports the handling target WA by an autonomous operation or a manual operation. The handling target WA includes, for example, a pallet and a load. The handling vehicle 11 lifts the pallet and transports the pallet and the load stacked on the pallet.

[0019] The handling vehicle 11 unloads the handling target WA or loads the handling target WA onto, for example, a truck, a shelf, or the like. The handling vehicle 11 may transport, for example, a handling target WA in which a plurality of pallets and a plurality of packages are stacked respectively. In this handling target WA, for example, a plurality of pallets and packages are alternately stacked respectively. The handling target WA includes, for example, a first pallet P1, a first package L1, a second pallet P2, and a second package L2.

[0020] The first pallet P1 and the second pallet P2 are, for example, plate-shaped flat pallets. One or more insertion holes Pa are respectively opened in the first pallet P1 and the second pallet P2. The first pallet P1 is the pallet located at the lower stage in the handling target WA. Therefore, the first pallet P1 contacts, for example, the floor surface. The second pallet P2 is the pallet located above the first pallet P1. The second pallet P2 is the pallet located at the upper stage in the handling target WA. The second pallet P2 is loaded on, for example, the first package L1. The first package L1 is loaded on the first pallet P1. The second package L2 is loaded on the second pallet P2. Therefore, in the handling target WA, the second package L2, the second pallet P2, the first package L1, and the first pallet P1 are arranged vertically in this order.

[0021] As shown in FIGS. 2 and 3, in the handling target WA, the second pallet P2 may protrude with respect to the first pallet P1. In the examples shown in FIGS. 2 and 3, in the handling target WA on the right side, the second pallet P2 is correctly stacked with respect to the first pallet P1. In the handling target WA on the left side, the second pallet P2 is not correctly stacked with respect to the first pallet P1.

[0022] When viewing the cargo handling target WA where the second pallet P2 is not properly stacked on the first pallet P1 from above, the second pallet P2 is positioned so that a part of it protrudes with respect to the first pallet P1. That is, when viewing the cargo handling target WA where the second pallet P2 is not properly stacked on the first pallet P1 from above, the second pallet P2 protrudes horizontally with respect to the first pallet P1. For example, when viewing this cargo handling target WA from the front of the cargo handling vehicle 11, the second pallet P2 protrudes in the front-rear direction, left-right direction, or both with respect to the first pallet P1.

[0023] If the second pallet P2 protrudes with respect to the first pallet P1, there is a risk of cargo collapse. For example, when the cargo handling vehicle 11 lifts the cargo handling target WA, there is a risk of cargo collapse due to its vibration. For example, when the cargo handling vehicle 11 lowers the cargo handling target WA, there is a risk of cargo collapse due to its vibration. Cargo collapse refers to, for example, the first cargo L1, the second pallet P2, the second cargo L2, etc. falling from the first pallet P1. If cargo collapse occurs, there is a risk of damage to the cargo.

[0024] If the second pallet P2 protrudes with respect to the first pallet P1, the second pallet P2 is likely to interfere with surrounding objects. There is a risk of cargo collapse due to the protruding second pallet P2 coming into contact with surrounding objects with respect to the first pallet P1. For example, when the cargo handling vehicle 11 places the cargo handling target WA next to another cargo handling target WA, there is a risk of cargo collapse due to the protruding second pallet P2 with respect to the first pallet P1 interfering with another cargo handling target WA.

[0025] When the second pallet P2 protruding from the first pallet P1 contacts another cargo handling target WA, not only is there a risk of cargo collapse, but there is also a risk that the cargo of another cargo handling target WA may be damaged. In the cargo handling vehicle 11, a plurality of cargo handling targets WA may be arranged by abutting the first pallet P1 against another first pallet P1. Therefore, if the second pallet P2 protrudes with respect to the first pallet P1, there is a high risk that the second pallet P2 will collide with the cargo of another cargo handling target WA. Therefore, the cargo handling system 10 determines whether or not cargo collapse is likely to occur in the cargo handling target WA.

[0026] As shown in FIG. 4, in the cargo handling target WA, the cargo may protrude with respect to the pallet. That is, the first cargo L1, the second cargo L2, or both may protrude with respect to the first pallet P1. Even in this case, as in the case where the second pallet P2 protrudes with respect to the first pallet P1, there is a risk of cargo collapse.

[0027] As shown in FIG. 1, the cargo handling vehicle 11 includes a vehicle body 12. The cargo handling vehicle 11 includes a front wheel 13 and a rear wheel 14. The front wheel 13 and the rear wheel 14 are attached to the vehicle body 12. For example, the front wheel 13 is a drive wheel and the rear wheel 14 is a steering wheel.

[0028] The cargo handling vehicle 11 includes a mast 15. The mast 15 is attached in front of the vehicle body 12. The mast 15 has an outer mast 16 and an inner mast 17. The outer mast 16 extends vertically. The inner mast 17 is attached to the outer mast 16. The inner mast 17 is configured to move up and down with respect to the outer mast 16.

[0029] The cargo handling vehicle 11 includes a lift bracket 18. The lift bracket 18 is attached to the inner mast 17. Therefore, the lift bracket 18 moves up and down with respect to the outer mast 16 together with the inner mast 17.

[0030] The material handling vehicle 11 is equipped with a shifter 19. The shifter 19 is attached to the lift bracket 18. The shifter 19 is configured to move left and right with respect to the lift bracket 18.

[0031] The material handling vehicle 11 is equipped with forks 20. The forks 20 are attached to the shifter 19. The forks 20 can be moved up and down by the lift bracket 18. The forks 20 can be moved left and right by the shifter 19.

[0032] The material handling vehicle 11 transports the material handling target WA by inserting the forks 20 into the pallet. Specifically, the material handling vehicle 11 transports the material handling target WA by inserting the forks 20 into the insertion hole Pa. The material handling vehicle 11 can lift or lower the material handling target WA by moving the forks 20 up and down. The material handling vehicle 11 can move the material handling target WA horizontally left and right by moving the forks 20 left and right.

[0033] The material handling vehicle 11 is equipped with a detection unit 21. The detection unit 21 is attached to the mast 15, for example. The detection unit 21 is a device that detects the position of the pallet. The detection unit 21 detects the position of the pallet by, for example, LiDAR (Light Detection and Ranging). The detection unit 21 includes, for example, a laser sensor 22. The laser sensor 22 includes a light emitting element that emits a laser and a light receiving element that receives the reflected laser.

[0034] The detection unit 21 irradiates a laser in front of the cargo handling vehicle 11. The detection unit 21 acquires point cloud data by receiving the reflected laser. The point cloud data is data indicating a collection of laser reflection points. The point cloud data includes two-dimensional data indicating the positions of the reflection points. The point cloud data includes data indicating the distance from the detection unit 21 to the laser reflection point. The distance from the detection unit 21 to the laser reflection point is measured by counting the time from when the laser is irradiated until the reflected laser returns. Therefore, the detection unit 21 detects the distance to an object located in front of the cargo handling vehicle 11.

[0035] The detection unit 21 detects the position of the pallet by irradiating a laser to the cargo handling target WA. When the cargo handling vehicle 11 starts transporting the cargo handling target WA, the cargo handling vehicle 11 faces the cargo handling target WA. That is, when the cargo handling vehicle 11 starts transporting the cargo handling target WA, the cargo handling target WA is located in front of the cargo handling vehicle 11. Therefore, when the cargo handling vehicle 11 starts transporting the cargo handling target WA, the detection unit 21 irradiates a laser to the cargo handling target WA.

[0036] The detection unit 21 acquires point cloud data indicating a collection of reflection points on the first pallet P1, the first load L1, the second pallet P2, and the second load L2 by irradiating a laser to the cargo handling target WA. That is, the point cloud data includes data indicating the positions of the first pallet P1 and the second pallet P2. Therefore, the detection unit 21 detects the positions of the first pallet P1 and the second pallet P2 by acquiring the point cloud data.

[0037] The cargo handling system 10 includes one or more imaging devices 31. The imaging device 31 is a device that images the cargo handling target WA. The imaging device 31 acquires image data in which the cargo handling target WA is captured by imaging the cargo handling target WA. The imaging device 31 is, for example, a camera.

[0038] In this example, the imaging device 31 is mounted on the handling vehicle 11. The imaging device 31 is attached to, for example, the mast 15. In this example, the imaging device 31 is a part of the configuration of the handling vehicle 11. The imaging device 31 may be located outside the handling vehicle 11. The imaging device 31 may be attached, for example, to the workplace where the handling vehicle 11 works. The imaging device 31 may be attached, for example, to the ceiling, wall, etc. of the warehouse where the handling target WA is stored.

[0039] In this example, the imaging device 31 images the front of the handling vehicle 11. When the handling vehicle 11 starts transporting the handling target WA, it faces the handling target WA. Therefore, when the handling vehicle 11 starts transporting the handling target WA, the imaging device 31 images the handling target WA. When the imaging device 31 is located outside the handling vehicle 11, it may be located at a position where the handling target WA is imaged.

[0040] The handling system 10 may include a plurality of imaging devices 31. For example, the plurality of imaging devices 31 may be mounted on the handling vehicle 11 or may be located outside the handling vehicle 11. For example, among the plurality of imaging devices 31, one may be mounted on the handling vehicle 11 and one may be located outside the handling vehicle 11. By the handling system 10 including a plurality of imaging devices 31, the plurality of imaging devices 31 can image the handling target WA from different angles.

[0041] As shown in FIG. 5, the handling system 10 includes a notification device 41. The notification device 41 is a device that notifies the operator of the handling vehicle 11, the administrator of the handling system 10, etc. of information regarding the handling system 10. The notification device 41 notifies, for example, that an abnormality has occurred in the handling vehicle 11.

[0042] The notification device 41 may be, for example, a speaker that notifies information regarding the handling system 10 by sound. The notification device 41 may be, for example, a lamp that notifies information regarding the handling system 10 by lighting. The notification device 41 may be, for example, a display that notifies information regarding the handling system 10 by displaying characters and images.

[0043] In this example, the notification device 41 is located outside the cargo handling vehicle 11. The notification device 41 is attached, for example, to a workplace where the cargo handling vehicle 11 is used, such as a warehouse. The notification device 41 is located, for example, in a control room where an administrator of the cargo handling system 10 is present.

[0044] The notification device 41 may be mounted on the cargo handling vehicle 11. In this case, the notification device 41 is a part of the configuration of the cargo handling vehicle 11. The notification device 41 may be composed of a portable information terminal owned by the operator of the cargo handling vehicle 11 or an operator of the cargo handling system 10.

[0045] The cargo handling system 10 includes a control device 51. The control device 51 is a device that controls the cargo handling system 10. In this example, the control device 51 is mounted on the cargo handling vehicle 11. Therefore, in this example, the control device 51 is a part of the configuration of the cargo handling vehicle 11. Thus, in this example, the control device 51 controls the cargo handling vehicle 11. The control device 51 may be located outside the cargo handling vehicle 11. In this case, the control device 51 is configured to be able to communicate with the cargo handling vehicle 11 from outside the cargo handling vehicle 11.

[0046] The control device 51 is connected to the imaging device 31 wirelessly or by wire so as to be able to communicate therewith. The control device 51 acquires image data from the imaging device 31. The control device 51 is connected to the notification device 41 wirelessly or by wire so as to be able to communicate therewith. The control device 51 notifies the notification device 41 of information, thereby causing the notification device 41 to notify the information.

[0047] The control device 51 acquires point cloud data from the detection unit 21. The control device 51 detects the positions of the first pallet P1 and the second pallet P2 from the reflection points of the first pallet P1 and the reflection points of the second pallet P2 included in the point cloud data. The control device 51 detects the positions of the first pallet P1 and the second pallet P2, for example, by calculating a plane equation indicating the front surfaces of the first pallet P1 and the second pallet P2 or extracting a straight line indicating the edges of the first pallet P1 and the second pallet P2 based on the point cloud data.

[0048] The control device 51 may be composed of one or more processors that execute various processes according to a computer program. The control device 51 may be composed of one or more dedicated hardware circuits such as application-specific integrated circuits that execute at least some of the various processes. The control device 51 may be composed of a circuit including a combination of a processor and a hardware circuit. The processor includes a CPU and a memory such as a RAM and a ROM. The memory stores program code or instructions configured to cause the CPU to execute a process. The memory, that is, the computer-readable medium, includes any medium that can be accessed by a general-purpose or dedicated computer.

[0049] The control device 51 includes, for example, a CPU 52 and a memory 53. The CPU 52 controls the cargo handling system 10, for example, by executing a program stored in the memory 53.

[0050] The memory 53 stores a learned machine model 54. When image data is input, the model 54 outputs a protrusion value indicating the amount of protrusion of the second palette P2 with respect to the first palette P1. The model 54 is defined by mapping data indicating the relationship between the image data and the protrusion value. The amount of protrusion of the second palette P2 with respect to the first palette P1 is the dimension of the portion protruding from the first palette P1 in the second palette P2. That is, the larger the deviation of the second palette P2 with respect to the first palette P1, the larger the amount of protrusion. Since the cargo handling target WA is shown in the image data, information indicating the amount of protrusion of the second palette P2 with respect to the first palette P1 is included.

[0051] The overhang value output by the model 54 indicates, for example, the overhang amounts on the left and right of the second pallet P2 with respect to the first pallet P1. The overhang value output by the model 54 may indicate the overhang amounts in front of and behind the second pallet P2 with respect to the first pallet P1. The model 54 may output an overhang value indicating the larger one of the overhang amounts on the left and right of the second pallet P2 with respect to the first pallet P1 and the overhang amounts in front of and behind the second pallet P2 with respect to the first pallet P1. The model 54 may output both an overhang value indicating the overhang amount on the left and right of the second pallet P2 with respect to the first pallet P1 and an overhang value indicating the overhang amount in front of and behind the second pallet P2 with respect to the first pallet P1.

[0052] The model 54 is obtained by having a machine learning program learn the correspondence between the image data and the overhang value. The learning method of the machine learning program is, for example, supervised learning by a neural network. The learning method of the machine learning program is, for example, deep learning. The machine learning program generates a function that maps the image data and the corresponding overhang value, that is, mapping data, by learning the correspondence between the image data and the overhang value.

[0053] The machine learning program learns the correspondence between the image data and the overhang value when teacher data is input. The teacher data is data with a label attached to the image data. Specifically, the teacher data is data in which an overhang value is associated with the image data. The teacher data is, for example, data in which an overhang value indicating the overhang amount of the second pallet P2 with respect to the first pallet P1 is associated with the image data showing the handling target WA in which the second pallet P2 protrudes with respect to the first pallet P1.

[0054] In the teacher data, the overhang value associated with the image data is, for example, a measured value. In the teacher data, for example, the value of the dimension actually measured by a measuring instrument such as a ruler or a tape measure is associated with the image data as the overhang value.

[0055] The teacher data may include image data in which the handling target WA where the second pallet P2 does not protrude with respect to the first pallet P1 is imaged. In this case, this image data is labeled with a label having a protrusion value of 0. Thereby, the model 54 with improved output accuracy can be obtained.

[0056] The model 54 may be configured to output multiple outputs when image data is input. That is, when image data is input, the model 54 may output another value in addition to the protrusion value of the second pallet P2 with respect to the first pallet P1. For example, when image data is input, the model 54 may output a classification value indicating whether the protrusion amount of the first load L1 with respect to the first pallet P1 and the protrusion amount of the second load L2 with respect to the first pallet P1 are large. Since the handling target WA is imaged in the image data, information indicating the protrusion amount of the first load L1 with respect to the first pallet P1 and the protrusion amount of the second load L2 with respect to the first pallet P1 is included.

[0057] The classification value is a value that serves as an index indicating whether the image data input to the model 54 belongs to a predetermined class. The classification value is represented by, for example, a value between 0 and 1. For example, the classification value indicates the probability that the image data input to the model 54 belongs to a class in which either the protrusion amount of the first load L1 with respect to the first pallet P1 or the protrusion amount of the second load L2 with respect to the first pallet P1 is large.

[0058] The model 54 classifies the input image data based on the protrusion amount of the first load L1 with respect to the first pallet P1 and the protrusion amount of the second load L2 with respect to the first pallet P1. That is, the model 54 performs binary classification on the input image data.

[0059] When the amount by which the first package L1 protrudes from the first pallet P1 is large, the image data input to the model 54 is determined to correspond to the class. When the amount by which the second package L2 protrudes from the first pallet P1 is large, the image data input to the model 54 is determined to correspond to the class. When neither the amount by which the first package L1 protrudes from the first pallet P1 nor the amount by which the second package L2 protrudes from the first pallet P1 is large, the image data input to the model 54 is determined not to correspond to the class.

[0060] When either the amount by which the first package L1 protrudes from the first pallet P1 or the amount by which the second package L2 protrudes from the first pallet P1 is large, there is a risk of package collapse. Therefore, it can be said that the classification value indicates whether the image data input to the model 54 corresponds to a class in which there is a risk of package collapse.

[0061] The model 54 is obtained by having a machine learning program learn the correspondence between image data and classification values. The learning method of the machine learning program is, for example, supervised learning by a neural network. The learning method of the machine learning program is, for example, deep learning. For example, by defining a branch in a neural network, a model 54 having a classification output and a regression output can be obtained. The machine learning program generates a function that maps image data and the corresponding classification value, that is, mapping data, by learning the correspondence between image data and classification values.

[0062] The machine learning program learns the correspondence between image data and classification values when teacher data is input. The teacher data is, for example, data in which a classification value is associated with the image data in addition to a protrusion value indicating the amount by which the second pallet P2 protrudes from the first pallet P1.

[0063] In the teacher data, the classification values associated with the image data are assigned, for example, by the operator of the cargo handling vehicle 11, the administrator of the cargo handling system 10, etc. The operator or the administrator assigns a classification value to the image data in which the cargo handling target WA is reflected, for example, by visually observing the cargo handling target WA, from the degree of protrusion of the first load L1 and the second load L2 with respect to the first pallet P1. For the image data in which the cargo handling target WA determined by the operator or the administrator to have a risk of cargo collapse appears, for example, "1" is assigned as the classification value. For the image data in which the cargo handling target WA determined by the operator or the administrator not to have a risk of cargo collapse appears, for example, "0" is assigned as the classification value.

[0064] The memory 53 stores the protrusion threshold value 55. The protrusion threshold value 55 is a threshold value to be compared with the protrusion value output by the model 54. When the protrusion value is larger than the protrusion threshold value 55, the control device 51 determines that there is a risk of cargo collapse based on the protrusion amount of the second pallet P2 with respect to the first pallet P1. That is, the protrusion threshold value 55 is an index indicating whether or not cargo collapse occurs based on the protrusion amount of the second pallet P2 with respect to the first pallet P1.

[0065] The memory 53 stores the classification threshold value 56. The classification threshold value 56 is a threshold value to be compared with the classification value output by the model 54. When the classification value is larger than the classification threshold value 56, the control device 51 determines that the image data input to the model 54 corresponds to the class. That is, when the classification value is larger than the classification threshold value 56, the control device 51 determines that there is a risk of cargo collapse based on the protrusion amounts of the first load L1 and the second load L2 with respect to the first pallet P1. The classification threshold value 56 is an index indicating whether or not cargo collapse occurs based on the protrusion amounts of the first load L1 and the second load L2 with respect to the first pallet P1.

[0066] <Determination Process> Next, the determination process executed by the control device 51 will be described. The determination process is a process for determining whether or not cargo collapse occurs. The determination process is executed, for example, immediately before the cargo handling vehicle 11 starts transporting the cargo handling target WA.

[0067] As shown in FIG. 6, in step S11, the control device 51 acquires image data from the imaging device 31. The control device 51 may perform known image processing such as shading correction and gamma correction on the acquired image data.

[0068] In step S12, the control device 51 inputs the acquired image data into the model 54. In step S13, the control device 51 acquires an overhang value from the model 54. The overhang value is estimated from the image data by the model 54. Specifically, the overhang value is a value estimated by the model 54 from the image data for the handling target WA located in front of the handling vehicle 11, representing the overhang amount of the second pallet P2 with respect to the first pallet P1.

[0069] In step S14, the control device 51 acquires a classification value from the model 54. The classification value is estimated from the image data by the model 54. Specifically, the classification value is a value estimated by the model 54 from the image data for the handling target WA located in front of the handling vehicle 11, representing an index of whether or not load collapse occurs based on the overhang amounts of the first load L1 and the second load L2 with respect to the first pallet P1.

[0070] In step S15, the control device 51 determines whether or not the overhang value is greater than the overhang threshold 55. At this time, the control device 51 determines whether or not there is a risk of load collapse based on the overhang amount of the second pallet P2 with respect to the first pallet P1. If the overhang value is greater than the overhang threshold 55, the control device 51 determines that there is a risk of load collapse based on the overhang amount of the second pallet P2 with respect to the first pallet P1. If the overhang value is greater than the overhang threshold 55, the process proceeds to step S16. If the overhang value is less than or equal to the overhang threshold 55, the control device 51 determines that no load collapse will occur based on the overhang amount of the second pallet P2 with respect to the first pallet P1. If the overhang value is less than or equal to the overhang threshold 55, the control device 51 proceeds to step S18.

[0071] In step S16, the control device 51 notifies that the protrusion value is greater than the protrusion threshold value 55. For example, the control device 51 notifies the notification device 41 that the protrusion value is greater than the protrusion threshold value 55, thereby causing the notification device 41 to notify that the protrusion value is greater than the protrusion threshold value 55. Thereby, the operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10 grasps that the protrusion value is greater than the protrusion threshold value 55. That is, the operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10 grasps that there is a risk of cargo collapse. The operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10, for example, reloads the second pallet P2 onto the first pallet P1. Thereby, the risk of cargo collapse is reduced.

[0072] In step S17, the control device 51 may cause the cargo handling vehicle 11 to reload the second pallet P2. In this case, the cargo handling vehicle 11 autonomously reloads the second pallet P2. Specifically, the control device 51 moves the second pallet P2 relative to the first pallet P1 on the cargo handling vehicle 11 to reduce the amount by which the second pallet P2 protrudes from the first pallet P1. For example, the control device 51 reloads the second pallet P2 onto the cargo handling vehicle 11 relative to the first pallet P1 so that the second pallet P2 does not protrude from the first pallet P1.

[0073] In step S17, the control device 51 grasps the positions of the first pallet P1 and the second pallet P2 from the detection unit 21, for example. The control device 51 grasps the positions of the first pallet P1 and the second pallet P2 from the point cloud data, for example. Based on the detection result of the detection unit 21, the control device 51 moves the second pallet P2 relative to the first pallet P1. The control device 51 moves the second pallet P2 by causing the handling vehicle 11 to lift the second pallet P2. That is, the control device 51 reloads the second pallet P2 onto the handling vehicle 11 with respect to the first pallet P1. Thereby, the risk of load collapse is reduced. The convenience is improved by the handling vehicle 11 autonomously reloading the second pallet P2. When the control device 51 finishes the process of step S17, it ends the determination process.

[0074] In step S18, the control device 51 determines whether the classification value is greater than the classification threshold value 56. At this time, the control device 51 determines whether there is a risk of load collapse based on the overhang amounts of the first load L1 and the second load L2 with respect to the first pallet P1. If the classification value is greater than the classification threshold value 56, the control device 51 determines that there is a risk of load collapse based on the overhang amounts of the first load L1 and the second load L2 with respect to the first pallet P1. If the classification value is greater than the classification threshold value 56, the control device 51 proceeds to step S19. If the classification value is less than or equal to the classification threshold value 56, the control device 51 determines that no load collapse will occur based on the overhang amounts of the first load L1 and the second load L2 with respect to the first pallet P1. If the classification value is less than or equal to the classification threshold value 56, the control device 51 ends the determination process.

[0075] The control device 51 notifies in step S19 that the classification value is greater than the classification threshold 56. For example, the control device 51 notifies the notification device 41 that the classification value is greater than the classification threshold 56, thereby causing the notification device 41 to notify that the classification value is greater than the classification threshold 56. As a result, the operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10 grasps that the classification value is greater than the classification threshold 56. That is, the operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10 grasps that there is a risk of cargo collapse. The operator of the cargo handling vehicle 11 or the administrator of the cargo handling system 10, for example, rearranges the first cargo L1 or the second cargo L2 on the first pallet P1. Thereby, the risk of cargo collapse is reduced.

[0076] In the cargo handling vehicle 11, while the cargo can be moved together with the pallet, it is difficult to move the cargo alone. That is, in the cargo handling vehicle 11, it is difficult to move the first cargo L1 alone or the second cargo L2 alone with respect to the first pallet P1. Therefore, when the overhang amount of the first cargo L1 or the second cargo L2 with respect to the first pallet P1 is large, it is necessary for the worker of the cargo handling vehicle 11, the administrator of the cargo handling system 10, etc. to rearrange the cargo.

[0077] <Effect of the cargo handling system> Next, the effect of the cargo handling system 10 will be described. (1) The control device 51 inputs the image data into the model 54. When the overhang value output by the model 54 is greater than the overhang threshold 55, the control device 51 notifies that the overhang value is greater than the overhang threshold 55.

[0078] According to the above configuration, the handling system 10 can obtain the overhang value by inputting the image data into the model 54. The larger the overhang amount of the second pallet P2 with respect to the first pallet P1, that is, the larger the overhang value, the more likely it is that the load will collapse. The handling system 10 can determine the possibility of load collapse by comparing the overhang value with the overhang threshold value 55. When the overhang value is larger than the overhang threshold value 55, that is, when the load is likely to collapse, the handling system 10 notifies that the overhang value is larger than the overhang threshold value 55. Thereby, for example, the operator of the handling vehicle 11, the administrator of the handling system 10, etc. can grasp that the load is likely to collapse. For example, by the operator or the administrator stopping the handling of the handling target WA by the handling vehicle 11 or adjusting the position of the second pallet P2 with respect to the first pallet P1, the possibility of load collapse is reduced.

[0079] (2) When the overhang value is larger than the overhang threshold value 55, the control device 51 moves the second pallet P2 to the handling vehicle 11 so that the overhang amount of the second pallet P2 with respect to the first pallet P1 becomes smaller.

[0080] According to the above configuration, when the overhang value is larger than the overhang threshold value 55, by moving the second pallet P2 based on the detection result of the detection unit 21, the overhang amount of the second pallet P2 with respect to the first pallet P1 is reduced. Thereby, the possibility of load collapse is reduced.

[0081] (3) The control device 51 detects the positions of the first pallet P1 and the second pallet P2 from the point cloud data. According to the above configuration, based on the point cloud data, the control device 51 can reload the second pallet P2 onto the handling vehicle 11 with respect to the first pallet P1.

[0082] When the classification value indicates that either the overhang amount of the first load L1 with respect to the first pallet P1 or the overhang amount of the second load L2 with respect to the first pallet P1 is large, the control device 51 notifies that either the overhang amount of the first load L1 with respect to the first pallet P1 or the overhang amount of the second load L2 with respect to the first pallet P1 is large.

[0083] When either the overhang amount of the first load L1 with respect to the first pallet P1 or the overhang amount of the second load L2 with respect to the first pallet P1 is large, there is a risk of load collapse. In the handling vehicle 11, while the pallet and the loads can be moved together by lifting the pallet, it is difficult to move the loads individually. Therefore, when either the overhang amount of the first load L1 with respect to the first pallet P1 or the overhang amount of the second load L2 with respect to the first pallet P1 is large, it is necessary to manually reload the loads.

[0084] According to the above configuration, when the classification value indicates that either the overhang amount of the first load L1 with respect to the first pallet P1 or the overhang amount of the second load L2 with respect to the first pallet P1 is large, it notifies that either the overhang amount of the first load L1 with respect to the first pallet P1 is large or the overhang amount of the second load L2 with respect to the first pallet P1 is large. Thereby, for example, the operator of the handling vehicle 11, the administrator of the handling system 10, etc. can grasp that the first load L1 or the second load L2 protrudes significantly with respect to the first pallet P1. When the operator or administrator who has received this notification, for example, reloads the first load L1 or the second load L2, the risk of load collapse is reduced.

[0085] <Modification example of the handling system> The above embodiment can be implemented with the following modifications. The above embodiment and the following modification examples can be implemented in combination with each other within a technically non - conflicting range.

[0086] ○ As shown in FIG. 7, the imaging device 31 may be provided outside the handling vehicle 11. The control device 51 may be provided outside the handling vehicle 11. ○ The imaging device 31 may also serve as the detection unit 21. For example, by being configured as a TOF camera, the imaging device 31 can measure the distance to the handling target WA. In this case, the imaging device 31 can acquire, for example, image data including depth information for each pixel. The control device 51 can determine the positions of the first pallet P1 and the second pallet P2 from the image data.

[0087] ○ The control device 51 may separately store a machine - learned model that outputs an over - flow value when image data is input and a machine - learned model that outputs a classification value when image data is input. In this case, the learning methods of the machine - learning programs for obtaining the two models may be different methods.

[0088] ○ The learning method of the machine - learning program for obtaining the model 54 may be a method other than a neural network.

Explanation of Reference Numerals

[0089] 10…Handling system, 11…Handling vehicle, 31…Imaging device, 51…Control device, 54…Model, 55…Overflow threshold, L1…First load, L2…Second load, P1…First pallet, P2…Second pallet, WA…Handling target.

Claims

1. A cargo handling system including a cargo handling vehicle for transporting a cargo handling target including a first pallet, a first load stacked on the first pallet, a second pallet stacked on the first load, and a second load stacked on the second pallet, wherein the cargo handling system includes an imaging device for imaging the cargo handling target, a machine learning model that outputs an overhang value indicating an overhang amount of the second pallet with respect to the first pallet when image data captured by the imaging device is input, and a control device that stores an overhang threshold that is a threshold value of the overhang value, wherein the control device inputs the image data into the model, and when the overhang value output by the model is greater than the overhang threshold, notifies that the overhang value is greater than the overhang threshold.

2. The cargo handling vehicle is an autonomously operating forklift, and includes a detection unit that detects positions of the first pallet and the second pallet, wherein the control device moves the second pallet to the cargo handling vehicle so that the overhang amount of the second pallet with respect to the first pallet becomes smaller when the overhang value is greater than the overhang threshold. The cargo handling system according to claim 1.

3. The detection unit includes a laser sensor that emits a laser to the first pallet and the second pallet and receives the reflected laser, and the laser sensor acquires point cloud data indicating a reflection point of the laser by receiving the laser reflected by the first pallet and the second pallet, wherein the control device detects positions of the first pallet and the second pallet from the point cloud data. The cargo handling system according to claim 2.

4. When the image data is input, the model outputs the overhang value and a classification value indicating whether any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large, wherein the control device notifies that any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large when the classification value indicates that any of the overhang amount of the first load with respect to the first pallet and the overhang amount of the second load with respect to the first pallet is large. The cargo handling system according to any one of claims 1 to 3.

5. A cargo handling vehicle for transporting a cargo handling target including a first pallet, a first load stacked on the first pallet, a second pallet stacked on the first load, and a second load stacked on the second pallet, wherein the cargo handling vehicle includes an imaging device for imaging the cargo handling target, a machine learning model that outputs an overhang value indicating the overhang amount of the second pallet with respect to the first pallet when image data captured by the imaging device is input, and a control device that stores an overhang threshold that is a threshold value of the overhang value, wherein the control device inputs the image data into the model, and when the overhang value output by the model is greater than the overhang threshold, notifies that the overhang value is greater than the overhang threshold.

Images