Packing style state detection system

Abstract

To provide a packing style state detection system capable of improving the detection accuracy of a packing style state of a pallet or luggage.SOLUTION: A packing style state detection system 10 comprises: a camera 11 that images a pallet 4 and luggage 5 to acquire image data of the pallet 4 and the luggage 5; a laser sensor 12 that measures distances to the pallet 4 and the luggage 5 to acquire point group data of the pallet 4 and the luggage 5; a region detection unit 41 that detects regions of the pallet 4 and the luggage 5 in the image data; a point group processing unit 42 and an image-point group matching unit 43 that extract a point group of the regions of the pallet 4 and the luggage 5 on the basis of the point group data, and calculate distances from a forklift 2 to the pallet 4 and the luggage 5 on the basis of the point group of the regions of the pallet 4 and the luggage 5; and a front rear partition unit 44 that partitions a plurality of rows of pallets 4 in a positional relationship in a depth direction perpendicular to a lateral direction on the basis of the distances from the forklift 2 to the pallet 4 and the luggage 5.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a package state detection system. [Background technology]

[0002] For example, Patent Document 1 describes a technology in which an autonomous forklift equipped with sensors and cameras unloads goods loaded on a truck and loads them onto an autonomous unmanned guided vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-62964 Summary of the Invention [Problem to be solved by the invention]

[0004] In actual logistics situations, there are situations where pallets and packages are located behind (at the back) the pallets and packages being handled. In such cases, when detecting the loading status (packing status) of the pallets and packages based on camera image data, the areas of all pallets and packages within the image data are detected. Furthermore, because image data is two-dimensional information, it is not possible to determine whether the pallets and packages are in front or behind the camera. Therefore, when detecting abnormalities in the packaging status based on such image data, it may be erroneously detected that the pallets and packages located at the back are interfering with the pallets and packages located at the front, even though the packaging status is normal.

[0005] An object of the present invention is to provide a packaging state detection system that can improve the accuracy of detecting the packaging state of pallets and packages. [Means for solving the problem]

[0006] (1) One aspect of the present invention is a packaging condition detection system that detects the packaging condition of pallets and cargo placed on the pallets when a forklift loads cargo onto at least one level of pallets arranged in multiple rows in the horizontal direction, and includes an image acquisition unit that images the pallets and cargo and acquires image data of the pallets and cargo, a point cloud acquisition unit that measures the distance to the pallets and cargo and acquires point cloud data of the pallets and cargo, an area detection unit that detects the areas of the pallets and cargo in the image data acquired by the image acquisition unit, a distance calculation unit that extracts a point cloud of the areas of the pallets and cargo detected by the area detection unit based on the point cloud data acquired by the point cloud acquisition unit and calculates the distance from the forklift to the pallets and cargo based on the point cloud of the areas of the pallets and cargo, and a separation unit that separates the positional relationship in the depth direction perpendicular to the horizontal direction of each of the pallets arranged in multiple rows based on the distance from the forklift to the pallets and cargo calculated by the distance calculation unit.

[0007] In this type of packaging condition detection system, images of the pallets and packages are captured, image data of the pallets and packages is acquired, and distances to the pallets and packages are measured to acquire point cloud data of the pallets and packages. The areas of the pallets and packages are then detected in the image data. A point cloud of the areas of the pallets and packages in the image data is then extracted based on the point cloud data of the pallets and packages, and the distance from the forklift to the pallets and packages is calculated based on the point cloud. The positional relationship of each pallet arranged in multiple rows in the depth direction, perpendicular to the horizontal direction, is then determined based on the distance from the forklift to the pallets and packages. Therefore, for each pallet arranged in multiple rows, it is possible to determine whether it is located on the near side (front side) or the far side (rear side). This improves the accuracy of detecting the packaging condition of the pallets and packages by excluding pallets and packages located at the far side when detecting the packaging condition of the pallets and packages.

[0008] (2) In (1) above, the packaging condition detection system may further include an interference determination unit that determines, based on the areas of the pallets and cargo detected by the area detection unit, whether cargo in a row adjacent to the pallet to be held is interfering with a pallet or cargo in the same row as the pallet to be held by the forklift, and the separation unit may recognize, among the pallets arranged in multiple rows, a pallet that is at a distance from the forklift that is equal to or greater than a predetermined value as a rear pallet that is shifted toward the rear in the depth direction, and exclude the rear pallet and the cargo placed on the rear pallet from the packaging condition detection targets for detecting the packaging condition, and the interference determination unit may determine, based on the areas of the pallets and cargo detected by the area detection unit that have not been excluded from the packaging condition detection targets by the separation unit, whether cargo in a row adjacent to the pallet to be held is interfering with a pallet or cargo in the same row as the pallet to be held.

[0009] In this configuration, among the pallets arranged in multiple rows, a pallet that is a distance from the forklift equal to or greater than a specified value is identified as the rear pallet, and the rear pallet and the cargo placed on the rear pallet are excluded from the package state detection target, thereby excluding the pallets and cargo located at the rear. Then, based on the area of ​​the pallets and cargo that are not excluded from the package state detection target, it is determined whether cargo located in a row adjacent to the pallet to be held is interfering with a pallet or cargo located in the same row as the pallet to be held. This prevents the erroneous determination that a cargo located at the rear is interfering with a pallet or cargo located in the front.

[0010] (3) In (2) above, when the interference determination unit determines that a package in a row adjacent to the pallet to be held is interfering with a pallet or package in the same row as the pallet to be held, the interference determination unit may virtually shrink the package in the adjacent row horizontally in the image data so that the interfering end of the package in the adjacent row moves a specified amount horizontally to the opposite side from the interfering side, and in that state may again determine whether the package in the adjacent row is interfering with the pallet or package in the same row as the pallet to be held.

[0011] In this configuration, the image data shows the packages in the adjacent rows virtually shrinking horizontally so that the edge of the package in the row adjacent to the pallet to be held that is interfering with the pallet is moved a specified distance horizontally toward the opposite side. Even in this state, if a package in an adjacent row is again determined to be interfering with a pallet or package in the same row as the pallet to be held, the package in the adjacent row is officially determined to be interfering. This prevents a situation in which the sides of adjacent packages are merely touching each other in the horizontal direction from being erroneously determined to be interfering with a pallet or package in the same row as the pallet to be held.

[0012] (4) In (3) above, when the interference determination unit determines that a pallet or cargo in the same row as the pallet to be held is being interfered with by a cargo in an adjacent row to the pallet to be held, the interference determination unit may virtually shrink the cargo in the adjacent row horizontally so that the interfering end of the cargo in the adjacent row in the image data moves horizontally to the opposite side from the interfering side by a number of pixels corresponding to the distance from the forklift to the cargo in the adjacent row calculated by the distance calculation unit.

[0013] The number of pixels corresponding to the specified amount of virtual horizontal shrinkage of packages in adjacent rows varies depending on the distance from the forklift to the pallet and package. Therefore, by moving the interfering edge of the package in the adjacent row in the image data horizontally to the opposite side of the interfering side by a number of pixels corresponding to the distance from the forklift to the package in the adjacent row, the amount of shrinkage of the package in the adjacent row becomes constant regardless of the distance from the forklift to the package in the adjacent row. This further reduces the erroneous determination that a package in an adjacent row is interfering with a pallet or package in the same row as the pallet being held.

[0014] (5) In (1) above, the packaging condition detection system may further include an overhang determination unit that determines whether a cargo or other pallet placed on a pallet to be held by a forklift protrudes laterally by more than a specified amount from the pallet to be held, based on the areas of the pallet and cargo detected by the area detection unit; the separation unit may recognize a pallet among multiple rows that is at a distance from the forklift that is more than a predetermined specified value as a rear pallet that is shifted toward the rear in the depth direction, and exclude the rear pallet and the cargo placed on the rear pallet from the packaging condition detection target for detecting the packaging condition; and the overhang determination unit may determine whether a cargo or other pallet placed on a pallet to be held protrudes laterally by more than a specified amount from the pallet to be held, based on the areas of the pallet and cargo detected by the area detection unit that are not excluded from the packaging condition detection target by the separation unit.

[0015] In this configuration, among the pallets arranged in multiple rows, a pallet that is a distance from the forklift equal to or greater than a specified value is identified as the rear pallet, and the rear pallet and the cargo placed on the rear pallet are excluded from the packaging condition detection target, thereby excluding the pallets and cargo located at the rear. Then, based on the area of ​​the pallets and cargo that are not excluded from the packaging condition detection target, it is determined whether the cargo placed on the pallet to be held or another pallet protrudes laterally from the pallet to be held by more than a specified amount. This prevents the pallets and cargo located at the rear from being erroneously determined to protrude laterally from the pallet to be held by more than a specified amount.

[0016] (6) In (5) above, the protrusion determination unit may determine whether the cargo or other pallet placed on the pallet to be held protrudes laterally beyond the pallet to be held by more than a specified amount based on the area of ​​the pallet and cargo that has not been excluded from the package state detection target by the separation unit and the distance from the forklift to the cargo or other pallet calculated by the distance calculation unit.

[0017] In this configuration, the distance from the forklift to the cargo or other pallets placed on the pallet to be held is used to determine whether the cargo or other pallets placed on the pallet to be held protrude laterally from the pallet to be held by more than a specified amount. Therefore, even if the other pallet placed on the pallet to be held protrudes forward more than the pallet to be held, for example, it is possible to accurately determine whether the cargo or other pallets placed on the pallet to be held protrude laterally from the pallet to be held by more than a specified amount.

[0018] (7) In any of (1) to (6) above, the distance calculation unit may extract a point cloud of the front area of ​​the pallet and luggage from the point cloud of the area of ​​the pallet and luggage detected by the area detection unit, identify the center of the front of the pallet and luggage based on the point cloud of the front area of ​​the pallet and luggage, and calculate the distance from the forklift to the center of the front of the pallet and luggage.

[0019] This configuration prevents shadows from the side areas of the pallet and the package from affecting the detection of the packaging state of the pallet and the package in the point cloud data, thereby further improving the detection accuracy of the packaging state of the pallet and the package. [Effects of the Invention]

[0020] According to the present invention, it is possible to improve the accuracy of detecting the packaging state of a pallet and packages. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram showing the configuration of a cargo handling control system including a package state detection system according to an embodiment of the present invention. [Figure 2] 2 is a side view of a forklift truck on which the cargo handling control system shown in FIG. 1 is mounted. [Figure 3] FIG. 1 is a side view showing a state in which a pallet on which cargo is placed is loaded onto the bed of a truck. [Figure 4] 10A and 10B are diagrams showing a state in which a pallet and a load interfere with an adjacent pallet or load, and a state in which a load placed on a lower pallet or an upper pallet protrudes from the lower pallet. [Figure 5] 2 is a block diagram showing the functions of an area detection unit shown in FIG. 1. FIG. [Figure 6] FIG. 6 is a block diagram showing a function for creating the learned data shown in FIG. 5. [Figure 7] 2 is a flowchart showing a procedure of point cloud processing executed by the point cloud processing unit shown in FIG. 1. [Figure 8]2 is a flowchart showing the procedure of a calculation process executed by an image-point cloud matching unit shown in FIG. 1. [Figure 9] 2 is a flowchart showing the procedure of a separation process executed by a front-rear separation unit shown in FIG. 1. [Figure 10] 10 is a flowchart showing the procedure of a luggage interference determination process executed by a luggage interference determination unit shown in FIG. [Figure 11] 11 is a diagram showing how the luggage interference determination unit shown in FIG. 10 determines whether or not there is luggage interference. FIG. [Figure 12] 11 is a flowchart showing details of step S143 shown in FIG. 10. [Figure 13] 13 is a diagram showing how a baggage interference determination region is set in step S143 shown in FIG. 12. FIG. [Figure 14] 10A and 10B are diagrams showing how a pallet or a piece of luggage interfering with a set of luggage to be handled is shrunk laterally. [Figure 15] 2 is a flowchart showing the procedure of a protrusion determination process executed by a protrusion determination unit shown in FIG. 1; [Figure 16] 16 is a diagram showing how the protrusion determination unit shown in FIG. 15 determines whether or not a package or pallet is protruding. FIG. [Figure 17] FIG. 10 is a diagram showing how the amount of overhang of an upper pallet relative to a lower pallet is calculated. [Figure 18] FIG. 10 is a diagram showing how, when a pallet and cargo are present behind the pallet and cargo to be handled, it is erroneously determined that the pallet and cargo present behind interfere with the pallet and cargo to be handled. [Figure 19] 10 is a diagram showing a state in which, when an upper pallet protrudes further forward than a lower pallet, the upper pallet is erroneously determined to protrude beyond a specified amount relative to the lower pallet. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0023] Fig. 1 is a block diagram showing the configuration of a cargo handling control system equipped with a package state detection system according to one embodiment of the present invention. In Fig. 1, the cargo handling control system 1 is mounted on a forklift 2 as shown in Fig. 2. The forklift 2 is, for example, a counter-load forklift.

[0024] The forklift 2 has a body 31, a pair of left and right front wheels 32 which are drive wheels arranged at the front of the body 31, a pair of left and right rear wheels 33 which are steering wheels arranged at the rear of the body 31, a mast 34 attached to the front end of the body 31, a pair of left and right forks 36 attached to the mast 34 via lift brackets 35 so that they can be raised and lowered, a lift cylinder 37 which raises and lowers the forks 36 via the lift brackets 35, and a tilt cylinder 38 which tilts the mast 34.

[0025] The cargo handling control system 1 is a system that controls cargo handling when cargo handling is automatically performed by a forklift 2. Here, as shown in FIG. 3, the cargo handling control system 1 controls the forklift 2 when holding a pallet 4 loaded on a loading platform 3a of a truck 3 with the forks 36 of the forklift 2, that is, when carrying out what is called cargo pick-up.

[0026] The pallet 4 is, for example, a flat pallet made of plastic or wood. The pallet 4 has a square or nearly square shape in plan view. Cargo 5 is placed on the pallet 4. The pallet 4 has two fork holes 6 into which forks 36 are inserted. The fork holes 6 extend from the front face (front surface) 4a of the pallet 4 toward the rear.

[0027] Pallets 4 are loaded in multiple rows along the front-to-rear direction of the truck 3 on the loading platform 3a of the truck 3. The pallets 4 are arranged with their front faces 4a facing the side of the truck 3. Therefore, the forklift 2 picks up cargo from the side of the truck 3. At this time, the multiple rows of pallets 4 are arranged side by side in the horizontal direction (left-right direction) as viewed from the forklift 2. The forklift 2 loads and unloads pallets 4 in one level or multiple levels (upper and lower levels) on which cargo 5 is placed (see Figure 4(c) etc.). For example, when the forklift 2 picks up cargo from a two-level pallet 4, the forks 36 are inserted into the fork holes 6 of the lower level pallet 4.

[0028] 3, when the pallet 4 and the cargo 5 are properly packed, the cargo can be smoothly picked up by the forklift 2. The packing state refers to the way the pallet 4 and the cargo 5 are placed.

[0029] However, as shown in Figure 4, if the packaging state of the pallets 4 and the cargo 5 is abnormal, it may be impossible to load the cargo using the forklift 2. For example, as shown in Figure 4(a), if laterally adjacent pallets 4 interfere with each other in the width direction of the truck 3 (the direction in which the pallets 4 are pulled out from the loading platform 3a to the side of the truck 3), the cargo may collapse when being loaded.

[0030] Furthermore, as shown in Figure 4(b), if a load 5 placed on a pallet 4 interferes with a load 5 placed on a laterally adjacent pallet 4, or if a load 5 placed on a pallet 4 interferes with a laterally adjacent pallet 4, the load may collapse during loading. Here, interference refers to a state in which a load 5 placed on a pallet 4 rests on a laterally adjacent pallet 4 or load 5.

[0031] Furthermore, as shown in Figure 4(c), due to vibrations during transport by truck 3, cargo 5 may protrude significantly laterally from pallets 4, or an upper pallet 4 may protrude significantly laterally from a lower pallet 4. In this case, too, there is a possibility that cargo may collapse when unloading.

[0032] To solve such problems, the cargo handling control system 1 detects whether the packing state of the pallet 4 and the cargo 5 is abnormal, and if the packing state of the pallet 4 and the cargo 5 is abnormal, performs appropriate control according to the packing state.

[0033] Therefore, the cargo handling control system 1 is equipped with a packaging state detection system 10 of this embodiment that detects whether the packaging state of the pallets 4 and cargo 5 is normal or abnormal. The packaging state detection system 10 is a system that detects the packaging state of the pallets 4 and cargo 5 when the forklift 2 handles cargo on at least one layer of pallets 4 arranged in multiple rows in the horizontal direction.

[0034] The cargo handling control system 1 includes a camera 11, a laser sensor 12, a drive unit 13, and a controller 14. The camera 11, the laser sensor 12, the drive unit 13, and the controller 14 are mounted on the forklift 2.

[0035] The camera 11 is an image acquisition unit that captures images of the pallet 4 and the luggage 5 and acquires image data of the pallet 4 and the luggage 5. The camera 11 captures images of a range including the front of the pallet 4 and the luggage 5.

[0036] The laser sensor 12 is a point cloud acquisition unit that measures the distance to the pallet 4 and the luggage 5 and acquires point cloud data of the pallet 4 and the luggage 5. The laser sensor 12 acquires point cloud data of the pallet 4 and the luggage 5 by emitting a laser toward the pallet 4 and the luggage 5 and receiving the reflected laser light. The laser sensor 12 emits a laser toward an area that includes the front of the pallet 4 and the luggage 5. The point cloud is a collection of reflected points of the laser. The point cloud data of the pallet 4 and the luggage 5 includes the distance to the pallet 4 and the luggage 5. A LiDAR, a laser range finder, or the like is used as the laser sensor 12.

[0037] The drive unit 13 has the above-mentioned lift cylinder 37 and tilt cylinder 38, a travel motor (not shown) that rotates the front wheels 32 of the forklift 2, and a steering motor (not shown) that steers the rear wheels 33 of the forklift 2.

[0038] The controller 14 is configured with a CPU, RAM, ROM, an input / output interface, etc. The controller 14 has an area detection unit 41, a point cloud processing unit 42, an image-point cloud matching unit 43, a front / rear separation unit 44, a luggage interference determination unit 45, an overhang determination unit 46, and a cargo handling control unit 47.

[0039] The camera 11, laser sensor 12, area detection unit 41, point cloud processing unit 42, image-point cloud matching unit 43, front and rear separation unit 44, luggage interference determination unit 45 and overhang determination unit 46 constitute the above-mentioned packaging state detection system 10.

[0040] The area detection unit 41 detects the areas of the pallet 4 and the luggage 5 in the image data acquired by the camera 11. The area detection unit 41 detects the areas of the pallet 4 and the luggage 5 in pixel units of the image data.

[0041] The area detection unit 41 detects the areas of the pallet 4 and the luggage 5 in the image data acquired by the camera 11. The area detection unit 41 extracts the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 (see FIG. 3) in the image data. The front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 are surfaces that face the forklift 2 that picks up the luggage. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 correspond to the front areas of the pallet 4 and the luggage 5.

[0042] 5, the area detection unit 41 is configured with a function having a package shape recognition model 20 and learned data 21. The area detection unit 41 reads the learned data 21 into the package shape recognition model 20, thereby extracting the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the image data acquired by the camera 11.

[0043] The package recognition model 20 is an instance segmentation model built using deep learning. Instance segmentation is a technology that estimates the position of individual objects on a pixel-by-pixel basis, and estimates the object position by labeling the target object area in detail and performing learning. The package recognition model 20 is built using data in which the front areas of the pallet 4 and luggage 5 are annotated (added) to images of the pallet 4 and luggage 5.

[0044] The package recognition model 20 has a feature extraction unit 22 that uses trained data 21 to extract features of the package image acquired by the camera 11, and an area recognition unit 23 that uses the trained data 21 and the features of the package image to recognize the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the image data.

[0045] The trained data 21 consists of two types of data: a package image of the pallet 4 and the luggage 5, and a data file containing information about the pallet 4 and the luggage 5 in the package image. The information about the pallet 4 and the luggage 5 in the package image includes pixel numbers of the package image. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the package image are represented by pixel numbers of the package image.

[0046] For example, if the number of types of luggage 5 is two, namely, cardboard boxes and parts boxes, and the number of types of pallets 4 is two, namely, plastic pallets and post pallets, learned data 21 is prepared in which labels are assigned to the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 for a total of four types of packaging images in which the pallets 4 and luggage 5 are photographed.

[0047] The trained data 21 is generated by pre-training a mathematical formula-driven database, as shown in Fig. 6. The mathematical formula-driven database is an image generated from a mathematical formula, and is data that does not require labeling of the image. For example, a dataset (FractalDB) composed of fractal geometric images is used as the mathematical formula-driven database.

[0048] Specifically, as shown in Fig. 6(a), first, pre-training of a formula-driven database 25 is performed using an image classification model 24 to generate pre-trained data 26. Formula-driven supervised learning is used for the pre-training of the formula-driven database 25. The image classification model 24 has a feature extraction unit 27 that extracts features of the formula-driven database 25, and an image classification unit 28 that solves an image classification problem using the features of the formula-driven database 25.

[0049] 6(b), transfer learning is performed using pre-trained data 26 to construct the above-mentioned packaging shape recognition model 20, thereby creating trained data 21. At this time, pre-trained data 26, which has been pre-trained using formula-driven database 25, is read into feature extraction unit 22, and data in which the areas of front surface 4a of pallet 4 and front surface 5a of package 5 are assigned to package shape image 29 in which pallet 4 and package 5 are photographed is subjected to transfer learning, thereby acquiring trained data 21.

[0050] The area detection unit 41 outputs the front area data of the pallet 4 and the luggage 5 in the image data to the image-point cloud matching unit 43, the luggage interference determination unit 45, and the overhang determination unit 46. The front area data of the pallet 4 and the luggage 5 includes the position of the center of gravity of the front surface 4a of the pallet 4 and the position of the center of gravity of the front surface 5a of the luggage 5.

[0051] 1, the point cloud processing unit 42 inputs the point cloud data of the pallet 4 and the luggage 5 acquired by the laser sensor 12 and performs predetermined point cloud processing on the point cloud data. The point cloud data acquired by the laser sensor 12 is represented in the sensor coordinate system.

[0052] Fig. 7 is a flowchart showing the procedure of point cloud processing executed by the point cloud processing unit 42. In Fig. 7, the point cloud processing unit 42 first acquires point cloud data of the laser sensor 12 (step S111).

[0053] Next, the point cloud processing unit 42 converts the point cloud data in the sensor coordinate system into point cloud data in the camera coordinate system using the camera-sensor external parameters acquired in advance (step S112). This makes it possible to calculate the distance from the camera 11 using the point cloud coordinates.

[0054] Next, the point cloud processing unit 42 uses the camera parameters acquired in advance to project the point cloud data in the camera coordinate system onto an image plane to generate a distance image (step S113). The distance image is an image in which distance is represented by color. The point cloud processing unit 42 then outputs the point cloud data in the camera coordinate system and the distance image to the image-point cloud matching unit 43 (step S114).

[0055] Returning to Figure 1, the image-point cloud matching unit 43 matches the image data with the point cloud data based on the front area of ​​the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41 and the point cloud data and distance image in the camera coordinate system obtained by the point cloud processing unit 42, and calculates the distance from the forklift 2 to the pallet 4 and luggage 5.

[0056] Fig. 8 is a flowchart showing the procedure of the calculation process executed by the image-point cloud matching unit 43. In Fig. 8, the image-point cloud matching unit 43 first acquires front area data of the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41, and point cloud data and a range image in the camera coordinate system obtained by the point cloud processing unit 42 (step S121).

[0057] Next, the image-point cloud matching unit 43 extracts a point cloud on the distance image corresponding to the front area of ​​the pallet 4 and luggage 5 in the image data, taking advantage of the correspondence between the pixel coordinates of the areas of the pallet 4 and luggage 5 in the image data and the pixel coordinates of the distance image (step S122).

[0058] Next, the image-point cloud matching unit 43 takes advantage of the correspondence between the point cloud on the distance image and the point cloud in the camera coordinate system to extract a point cloud of the area of ​​the pallet 4 and luggage 5 in the camera coordinate system from the point cloud on the distance image extracted in step S122 (step S123).

[0059] Next, the image-point cloud matching unit 43 extracts a point cloud of the front area of ​​the pallet 4 and luggage 5 in the camera coordinate system using a point cloud processing algorithm such as RANSAC (Random Sample Consensus) that extracts a plane from a point cloud (step S124).

[0060] Next, the image-point cloud matching unit 43 identifies the center coordinates of the pallet 4 and the luggage 5 from the point cloud coordinates of the front areas of the pallet 4 and the luggage 5 in the camera coordinate system (step S125). Then, the image-point cloud matching unit 43 calculates the distance from the camera 11 to the center coordinates of the pallet 4 and the luggage 5 as the distance from the forklift 2 to the front of the pallet 4 and the luggage 5 (step S126). The camera 11 is the origin of the camera coordinate system.

[0061] Next, the image-point cloud matching unit 43 outputs point cloud data of the front area of ​​the pallet 4 and luggage 5 in the camera coordinate system and data on the distance from the forklift 2 to the front of the pallet 4 and luggage 5 to the front / rear separation unit 44 (step S127).

[0062] Returning to Figure 1, the point cloud processing unit 42 and the image-point cloud matching unit 43 constitute a distance calculation unit that extracts a point cloud of the area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41 based on the point cloud data acquired by the laser sensor 12, and calculates the distance from the forklift 2 to the pallet 4 and luggage 5 based on the point cloud of the area of ​​the pallet 4 and luggage 5.

[0063] The front-rear separating unit 44 separates the positional relationship in the depth direction (front-rear direction) of each of the pallets 4 arranged in multiple rows based on the distance from the forklift 2 to the pallets 4 and the cargo 5 calculated by the image-point cloud matching unit 43. The depth direction is the direction perpendicular to the lateral direction.

[0064] The front-to-rear separating unit 44 recognizes, among the pallets 4 arranged in multiple rows, the pallets 4 whose distance from the forklift 2 is greater than a predetermined specified value as the rear pallet 4Z (see Figure 18) that is shifted to the rear in the depth direction (rear in the front-to-rear direction), and excludes the rear pallet 4Z and the rear cargo 5Z (see Figure 18) that is the cargo 5 placed on the rear pallet 4Z from the package state detection targets for detecting the package state.

[0065] Fig. 9 is a flowchart showing the procedure of the separation process executed by the front / rear separation unit 44. In Fig. 9, the front / rear separation unit 44 first acquires point cloud data of the front area of ​​the pallet 4 and the cargo 5, and distance data from the forklift 2 to the front of the pallet 4 and the cargo 5 (step S131).

[0066] Next, the front / rear separating unit 44 determines whether the distance from the forklift 2 to the front of the pallet 4 and the package 5 is equal to or greater than a predetermined specified value (step S132).

[0067] When the front / rear separating unit 44 determines that the distance from the forklift 2 to the front of the pallet 4 and the cargo 5 is equal to or greater than a specified value, it recognizes the pallet 4 and the cargo 5 as the rear pallet 4Z and the rear cargo 5Z (step S133).Then, the front / rear separating unit 44 excludes the rear pallet 4Z and the rear cargo 5Z from the package state detection targets (step S134).

[0068] When the front / rear separating unit 44 determines that the distance from the forklift 2 to the front of the pallet 4 and the cargo 5 is less than the specified value, it does not execute steps S133 and S134. Therefore, the pallet 4 and the cargo 5 whose distance from the forklift 2 is shorter than the specified value are not excluded from the packing state detection targets.

[0069] Next, the front / rear separating unit 44 outputs distance data from the forklift 2 to the front of the pallet 4 and the luggage 5 for which the packaging state is to be detected, to the luggage interference determining unit 45 and the protrusion determining unit 46 (step S135).

[0070] Returning to Figure 1, the luggage interference determination unit 45 is an interference determination unit that determines whether a luggage 5 in a row adjacent to the pallet 4 to be held by the forklift 2 is interfering with a pallet 4 or luggage 5 in the same row as the pallet 4 to be held by the forklift 2, based on the front area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41.

[0071] Here, the pallet 4 to be held is referred to as holding pallet 4A (see FIG. 11, etc.). Holding pallet 4A is the bottommost pallet 4. Pallets 4 existing in the same row as holding pallet 4A include holding pallet 4A itself.

[0072] The luggage interference determination unit 45 determines whether luggage 5 in a row adjacent to the holding pallet 4A is interfering with a pallet 4 or luggage 5 in the same row as the holding pallet 4A, based on the front area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41 that has not been excluded from the package state detection target by the front / rear separation unit 44.

[0073] When the cargo interference determination unit 45 determines that a cargo 5 in a row adjacent to the holding pallet 4A is interfering with a pallet 4 or cargo 5 in the same row as the holding pallet 4A, it virtually shrinks the interfering cargo 5K horizontally in the image data so that the interfering end of the cargo 5 in the adjacent row (referred to as interfering cargo 5K) moves a specified amount horizontally to the opposite side from the interfering side, and in that state it again determines whether the interfering cargo 5K is interfering with the pallet 4 or cargo 5 in the same row as the holding pallet 4A.

[0074] When the cargo interference determination unit 45 determines that a cargo 5 in a row adjacent to the holding pallet 4A is interfering with a pallet 4 or cargo 5 in the same row as the holding pallet 4A, it virtually shrinks the interfering cargo 5S horizontally so that in the image data, the interfering end of the interfering cargo 5K moves horizontally to the opposite side from the interfering side by a number of pixels corresponding to the distance from the forklift 2 to the interfering cargo 5K calculated by the image-point cloud matching unit 43.

[0075] FIG. 10 is a flowchart showing the procedure of the luggage interference determination process executed by the luggage interference determination unit 45.

[0076] 10, the luggage interference determination unit 45 first acquires front area data of the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41 (step S141). FIG. 11(a) shows an example of the front area of ​​the pallet 4 and luggage 5 in the image data D obtained by the area detection unit 41. In FIG. 11(a), the X direction indicates the left-right direction (horizontal direction) of the pallet 4 and luggage 5, and the Y direction indicates the up-down direction (height direction) of the pallet 4 and luggage 5.

[0077] Next, the luggage interference determination unit 45 creates multiple luggage sets S in the front area of ​​the pallets 4 and luggage 5 in the image data (step S142). As shown in FIG. 11(b), the luggage set S represents a rectangular area including the pallets 4 and luggage 5 that are adjacent to each other in the vertical direction in the image data D, from the center of gravity position G of the front area of ​​the pallets 4 and luggage 5. In the image data D shown in FIG. 11(b), two luggage sets Sa and Sb that are adjacent in the horizontal direction are created. The luggage sets Sa and Sb include two tiers of pallets 4 and luggage 5, one above the other.

[0078] Next, the luggage interference determination unit 45 sets a luggage interference determination region R for the cargo handling target in the image data (step S143).

[0079] Fig. 12 is a flowchart showing the details of step S143. In Fig. 12, the luggage interference determination unit 45 extends two imaginary lines L1 in the vertical direction from the left and right ends of the lowermost pallet 4, which is the holding pallet 4A of the luggage set S to be handled, in the image data D toward the top end E of the image, as shown in Fig. 13(a) (step S161).

[0080] Then, the luggage interference determination unit 45 determines whether at least one of two imaginary lines L1 extending vertically from both the left and right ends of the bottommost pallet 4 toward the top end E of the image hits a luggage 5 in the luggage set S to be loaded or another pallet 4 (step S162).

[0081] In image data D shown in Figure 13(a), a virtual line L1 extending upward from the left end of the lower pallet 4 reaches the top end E of the image without hitting the luggage 5 in the luggage set S to be loaded or the upper pallet 4, but a virtual line L1 extending upward from the right end of the lower pallet 4 hits the upper pallet 4 in the luggage set S to be loaded.

[0082] When the luggage interference determination unit 45 determines that the imaginary line L1 extending in the vertical direction from both the left and right ends of the bottommost pallet 4 hits a luggage 5 or another pallet 4 in the luggage set S to be handled, it extends the imaginary line L1 horizontally along the bottom edge of the luggage 5 or another pallet 4 to the end of either the left or right end of the luggage 5 or another pallet 4 that is closer to the tangent point with the imaginary line L1 (step S163). Next, the luggage interference determination unit 45 extends the imaginary line L1 in the vertical direction from the end of either the left or right end of the luggage 5 or another pallet 4 that is closer to the tangent point with the imaginary line L1 toward the top E of the image (step S164).

[0083] In image data D shown in Figure 13(a), as shown in Figure 13(b), a virtual line L1 is extended horizontally along the bottom edge of the upper pallet from the point of contact between the virtual line L1 and the upper pallet 4 to pixel P at the right edge of the upper pallet 4, and then a virtual line L1 is extended upward from the right edge of the upper pallet 4 towards the top edge E of the image. Note that in Figure 13, for convenience, only one pixel P is shown at each of the center of the height direction at the left and right edges of the pallet 4 and the luggage 5, but in reality, pixels P exist from the bottom to the top of the left and right edges of the pallet 4 and the luggage 5.

[0084] After executing step S164, or when it is determined in step S162 that the virtual line L1 does not hit any of the packages 5 in the package set S to be handled or any other pallet 4, the package interference determination unit 45 determines whether or not both of the two virtual lines L1 hit the top end E of the image (step S165). If the package interference determination unit 45 does not satisfy the condition that both of the two virtual lines L1 hit the top end E of the image, it executes the above-described step S162 again.

[0085] When the condition that both of the two virtual lines L1 coincide with the top edge E of the image is met, the luggage interference determination unit 45 sets the area surrounded by the two virtual lines L1, the horizontal line L2 along the bottom edge of the lowest pallet 4, and the horizontal line L3 along the top edge E of the image as the luggage interference determination area R for the object to be loaded, as shown in Figure 13(c) (step S166).

[0086] 10, the luggage interference determination unit 45 determines whether luggage 5 of the luggage set S other than the loading target is included in the luggage interference determination area R of the loading target set in step S143 (step S144). In the image data D shown in FIG. 11(c), luggage 5 of the luggage set S other than the loading target is included in the luggage interference determination area R of the loading target.

[0087] When the luggage interference determination unit 45 determines that luggage 5 of a luggage set S other than the luggage to be handled is included within the luggage interference determination area R of the luggage to be handled, it determines that luggage 5 of the luggage set S other than the luggage to be handled is interfering with the pallet 4 or luggage 5 of the luggage set S of the luggage to be handled (step S145).

[0088] In the image data D shown in Figure 11(c), it is determined that the luggage 5 of the luggage set S other than the luggage to be handled is interfering with the upper pallet 4 of the luggage set S to be handled. Therefore, the luggage 5 of the luggage set S other than the luggage to be handled is an interfering luggage 5K that is interfering with the upper pallet 4 of the luggage set S to be handled.

[0089] Next, the luggage interference determination unit 45 acquires distance data calculated by the front / rear separating unit 44 from the forklift 2 to the front of the pallet 4 and luggage 5 for which the packaging state is to be detected (step S146).

[0090] Then, the luggage interference determination unit 45 determines the amount of reduction of the interfering luggage 5K based on the distance from the forklift 2 to the interfering luggage 5K (step S147). Here, the amount of reduction of the interfering luggage 5K is the number of pixels corresponding to a predetermined specified amount. The number of pixels corresponding to the specified amount differs depending on the distance from the forklift 2 to the interfering luggage 5K.

[0091] Next, the luggage interference determination unit 45 virtually reduces the interfering luggage 5K in the horizontal direction by the reduction amount so that the interfering end of the interfering luggage 5K moves laterally to the opposite side of the luggage set S to be handled (step S148). As a result, the horizontal dimension of the interfering luggage 5K is reduced by the reduction amount.

[0092] In image data D shown in Figure 11(c), the left end of a piece of luggage 5 in the set S of luggage other than the one to be handled is interfering with the upper pallet 4 of the set S of luggage to be handled. Therefore, as shown in Figure 14, the piece of luggage 5 in the set S of luggage other than the one to be handled (interfering piece of luggage 5K) is reduced to the right by a reduction amount C.

[0093] After executing step S148, the luggage interference determination unit 45 determines whether the reduced interfering luggage 5K is included in the luggage interference determination area R of the loading object (step S149). In the image data D shown in Fig. 11(d), the reduced interfering luggage 5K is included in the luggage interference determination area R of the loading object.

[0094] When the luggage interference determination unit 45 determines that the reduced interfering luggage 5K is included in the luggage interference determination area R of the luggage to be handled, it finally determines that a luggage 5 of a luggage set S other than the luggage to be handled is interfering with a pallet 4 or luggage 5 of the luggage set S of the luggage to be handled (step S150). In the image data D shown in Figure 11(d), it is finally determined that a luggage 5 of a luggage set S other than the luggage to be handled is interfering with the upper pallet 4 of the luggage set S of the luggage to be handled.

[0095] Then, the cargo interference determination unit 45 outputs an interference abnormality control signal to the cargo handling control unit 47 (step S151).

[0096] When the luggage interference determination unit 45 determines in step S144 that no luggage 5 of the luggage set S other than the luggage to be handled is included in the luggage interference determination area R of the luggage to be handled, or when it determines in step S149 that no interfering luggage 5K after reduction is included in the luggage interference determination area R of the luggage to be handled, it finally determines that no luggage 5 of the luggage set S other than the luggage to be handled is interfering with the pallet 4 or luggage 5 of the luggage set S of the luggage to be handled (step S152). Then, the luggage interference determination unit 45 outputs a normal control signal to the luggage handling control unit 47 (step S153).

[0097] Returning to Figure 1, the protrusion determination unit 46 determines, based on the front area of ​​the pallet 4 and cargo 5 detected by the area detection unit 41, whether the cargo 5 or other pallets 4 placed on the pallet 4 (holding pallet 4A) to be held by the forklift 2 protrudes laterally from the holding pallet 4A by more than a specified amount.

[0098] The protrusion determination unit 46 determines whether the luggage 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount based on the front area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41 that has not been excluded from the package state detection target by the front / rear separation unit 44.

[0099] The protrusion determination unit 46 determines whether the luggage 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount based on the front area of ​​the pallet 4 and luggage 5 that have not been excluded from the package state detection targets by the front / rear separation unit 44 and the distance from the forklift 2 to the luggage 5 or other pallets 4 placed on the holding pallet 4A calculated by the image-point cloud matching unit 43.

[0100] FIG. 15 is a flowchart showing the procedure of the protrusion determination process executed by the protrusion determination unit 46.

[0101] 15, the protrusion determination unit 46 first acquires front area data of the pallet 4 and the luggage 5 in the image data obtained by the area detection unit 41 (step S171). FIG. 16(a) shows an example of the front area of ​​the pallet 4 and the luggage 5 in the image data D obtained by the area detection unit 41. In FIG. 16(a), the X direction and Y direction are the same as those in FIG. 11(a).

[0102] Next, the overhang determination unit 46 creates multiple package sets S in the front areas of the pallets 4 and packages 5 in the image data (step S172). As shown in FIG. 16(b), the package sets S represent rectangular areas including the pallets 4 and packages 5 that are adjacent to each other in the vertical direction in the image data D, starting from the center of gravity G of the front areas of the pallets 4 and packages 5. In the image data D shown in FIG. 16(b), two package sets Sa and Sb that are adjacent to each other in the horizontal direction are created. The package sets Sa and Sb include two tiers of pallets 4 and packages 5, one above the other.

[0103] Next, as shown in Figure 16(c), the overhang determination unit 46 extends two virtual lines L1 in the vertical direction from the left and right ends of the lowest pallet 4, which is the holding pallet 4A of the cargo set S to be loaded, in the image data D toward the top end E of the image (step S173).

[0104] Then, as shown in Figure 16(c), the overhang determination unit 46 sets the area in the image data D surrounded by two virtual lines L1, a horizontal line L2 along the bottom edge of the lowest pallet 4, and a horizontal line L3 along the top edge E of the image as the overhang determination area Q of the loading object (step S174).

[0105] Next, the protrusion determination unit 46 determines whether or not there is any luggage 5 or pallet 4 in the luggage set S to be handled that protrudes laterally from the protrusion determination area Q (step S175). In the image data D shown in Figure 16(c), the right end of the upper pallet 4 protrudes laterally from the protrusion determination area Q.

[0106] When the overhang determination unit 46 determines that there is luggage 5 or a pallet 4 that is protruding laterally from the overhang determination area Q, it acquires distance data calculated by the front / rear separation unit 44 from the forklift 2 to the front of the pallet 4 and luggage 5 whose packaging state is to be detected (step S176).

[0107] Then, the overhang determination unit 46 calculates the amount of overhang of the luggage 5 or pallet 4 from the overhang determination area Q based on the distance from the forklift 2 to the luggage 5 or pallet 4 that overhangs laterally from the overhang determination area Q (step S177). As shown in FIG. 17, the length d per pixel varies depending on the distance from the forklift 2 to the luggage 5 or pallet 4, so the amount of overhang f of the luggage 5 or pallet 4 from the overhang determination area Q is calculated from the distance from the forklift 2 to the luggage 5 or pallet 4.

[0108] Next, the overhang determination unit 46 determines whether the overhang amount f of the luggage 5 or pallet 4 from the overhang determination area Q in the luggage set S to be handled is equal to or greater than a predetermined threshold (step S178). The threshold is set to a value that does not affect the loading operation of the pallet 4 and luggage 5 even if the luggage 5 or pallet 4 overhangs the overhang determination area Q in the lateral direction, for example, a value that does not cause the luggage 5 to collapse.

[0109] When the protrusion determination unit 46 determines that the protrusion amount f of the cargo 5 or pallet 4 with respect to the protrusion determination area Q is equal to or greater than the threshold value, it determines that the cargo 5 or other pallet 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by a specified amount or more (step S179).Then, the protrusion determination unit 46 outputs a protrusion abnormality control signal to the cargo handling control unit 47 (step S180).

[0110] When the overhang determination unit 46 determines in step S175 that no luggage 5 or pallet 4 is present that protrudes laterally from the overhang determination area Q, or when the overhang determination unit 46 determines in step S178 that the overhang amount f of the luggage 5 and pallet 4 from the overhang determination area Q is not equal to or greater than the threshold, the overhang determination unit 46 determines that the luggage 5 or other pallets 4 placed on the holding pallet 4A do not protrude laterally from the holding pallet 4A by more than a specified amount (step S181).The overhang determination unit 46 then outputs a normal control signal to the cargo handling control unit 47 (step S182).

[0111] Returning to Fig. 1, the cargo handling control unit 47 controls the drive unit 13 in accordance with the determination results by the luggage interference determination unit 45 and the overhang determination unit 46. When the cargo handling control unit 47 receives a normal control signal from both the luggage interference determination unit 45 and the overhang determination unit 46, it controls the drive unit 13 to perform normal cargo handling control. When the cargo handling control unit 47 receives an interference abnormality control signal from the luggage interference determination unit 45 or an overhang abnormality control signal from the overhang determination unit 46, it controls the drive unit 13 to perform abnormal cargo handling control.

[0112] In the above, when the forklift 2 is used to pick up goods, the forklift 2 travels toward the truck 3. Then, when the forklift 2 reaches the side of the truck 3, the forklift 2 stops temporarily. In this state, the pallets 4 and goods 5 loaded on the truck 3 are imaged by the camera 11, and image data D of the pallets 4 and goods 5 is acquired.

[0113] Then, the front areas of the pallet 4 and luggage 5 in the image data D are extracted. Then, based on the front areas of the pallet 4 and luggage 5 in the image data D, it is determined whether luggage 5 in an adjacent luggage set S interferes with luggage 5 or pallet 4 in the luggage set S to be handled. Also, based on the front areas of the pallet 4 and luggage 5 in the image data D, it is determined whether luggage 5 or other pallets 4 placed on the holding pallet 4A protrude laterally beyond the holding pallet 4A by more than a specified amount.

[0114] Then, when it is determined that the luggage 5 and pallet 4 of the luggage set S to be handled are not interfering with the luggage 5 of the adjacent luggage set S, and when it is determined that the luggage 5 and other pallets 4 placed on the holding pallet 4A do not protrude laterally beyond the holding pallet 4A by more than a specified amount, the drive unit 13 is controlled to unload the luggage set S to be handled.

[0115] In an actual logistics site, as shown in FIG. 18(a), there may be a pallet 4 and a package 5 behind the pallet 4 and package 5 to be handled. In this case, when the camera 11 captures images of both the front and rear pallets 4 and packages 5, image data D showing the front and rear pallets 4 and packages 5 is acquired. However, because the image data D is only two-dimensional information, it is not possible to determine whether the pallet 4 and package 5 are in front of or behind the camera 11.

[0116] When such image data D is used to detect abnormalities in the packaging state, as shown in Figure 18(b), the cargo interference detection area R for the cargo to be handled includes pallets 4 and cargo 5 other than the cargo to be handled, even though the packaging state is normal. As a result, it is erroneously determined that the pallets 4 and cargo 5 located at the rear (back side) are interfering with the pallets 4 and cargo 5 located at the front (near side).

[0117] 19(a), in a situation where the upper pallet 4 protrudes further forward than the lower pallet 4, the distance from the camera 11 to the upper pallet 4 is shorter than the distance from the camera 11 to the lower pallet 4. Therefore, even though the upper and lower pallets 4 are the same size, the front surface 4a of the upper pallet 4 appears larger than the front surface 4a of the lower pallet 4 in the image data D, as shown in FIG. 19(b). For this reason, the protrusion amount f of the upper pallet 4 relative to the lower pallet 4 may be erroneously determined to be equal to or greater than the threshold.

[0118] To address this issue, in this embodiment, images of the pallets 4 and packages 5 are captured, image data of the pallets 4 and packages 5 is acquired, and the distances to the pallets 4 and packages 5 are measured to acquire point cloud data of the pallets 4 and packages 5. The areas of the pallets 4 and packages 5 are then detected in the image data. A point cloud of the areas of the pallets 4 and packages 5 in the image data is then extracted based on the point cloud data of the pallets 4 and packages 5, and the distance from the forklift 2 to the pallets 4 and packages 5 is calculated based on the point cloud. The positional relationship in the depth direction, perpendicular to the lateral direction, of each pallet 4 arranged in multiple rows is then determined based on the distance from the forklift 2 to the pallets 4 and packages 5. Therefore, it becomes possible to determine whether each pallet 4 arranged in multiple rows is located on the near side (front side) or the far side (rear side). This improves the accuracy of detecting the packaging state of the pallets 4 and packages 5 by excluding the pallets 4 and packages 5 located on the far side when detecting the packaging state of the pallets 4 and packages 5.

[0119] Furthermore, in this embodiment, among the pallets 4 arranged in multiple rows, a pallet 4 that is a distance from the forklift 2 equal to or greater than a specified value is recognized as a rear pallet 4Z, and the rear pallet 4Z and the cargo 5 (rear cargo 5Z) placed on the rear pallet 4Z are excluded from the package state detection targets, thereby excluding the pallets 4 and cargo 5 located at the rear. Then, based on the areas of the pallets 4 and cargo 5 that are not excluded from the package state detection targets, it is determined whether cargo 5 located in a row adjacent to the holding pallet 4A is interfering with pallets 4 or cargo 5 located in the same row as the pallet 4 to be held (holding pallet 4A). This prevents the cargo 5 located at the rear from being erroneously determined to be interfering with the pallet 4 or cargo 5 located at the front.

[0120] Furthermore, in this embodiment, the interfering luggage 5K in the image data D is virtually reduced in the horizontal direction so that the edge of the interfering side of the luggage 5 (interfering luggage 5K) in the row adjacent to the holding pallet 4A moves a specified amount horizontally toward the opposite side from the interfering side. Even in this state, when it is again determined that the interfering luggage 5K is interfering with the pallet 4 or luggage 5 in the same row as the holding pallet 4A, the luggage 5 in the row adjacent to the pallet 4 or luggage 5 in the same row as the holding pallet 4A is officially determined to be interfering. Therefore, a state in which the sides of laterally adjacent luggage 5 are merely in contact with each other is prevented from being erroneously determined as the luggage 5 in the row adjacent to the pallet 4 or luggage 5 in the same row as the holding pallet 4A being interfering.

[0121] Furthermore, in this embodiment, the number of pixels corresponding to the specified amount by which the luggage 5 (interfering luggage 5K) in the adjacent row is virtually reduced in the horizontal direction varies depending on the distance from the forklift 2 to the pallet 4 and luggage 5. Therefore, by moving the interfering end of the interfering luggage 5K in the image data D in the horizontal direction to the opposite side from the interfering side by a number of pixels corresponding to the distance from the forklift 2 to the interfering luggage 5K, the amount of reduction of the interfering luggage 5K becomes constant regardless of the distance from the forklift 2 to the interfering luggage 5K. This further reduces the erroneous determination that a luggage 5 in an adjacent row is interfering with a pallet 4 or luggage 5 in the same row as the holding pallet 4A.

[0122] Furthermore, in this embodiment, among the pallets 4 arranged in multiple rows, the pallets 4 that are at a distance from the forklift 2 equal to or greater than a specified value are identified as the rear pallets 4Z, and the rear pallets 4Z and the packages 5 placed on the rear pallets 4Z (rear packages 5Z) are excluded from the package state detection targets, thereby excluding the pallets 4 and packages 5 located at the rear. Then, based on the areas of the pallets 4 and packages 5 that are not excluded from the package state detection targets, it is determined whether the packages 5 placed on the holding pallet 4A or other pallets 4 protrude laterally from the holding pallet 4A by more than a specified amount. This prevents the pallets 4 and packages 5 located at the rear from being erroneously determined to laterally protrude by more than a specified amount from the holding pallet 4A.

[0123] Furthermore, in this embodiment, the distance from the forklift 2 to the cargo 5 or other pallets 4 placed on the holding pallet 4A is used to determine whether the cargo 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount. Therefore, even if the other pallet 4 placed on the holding pallet 4A protrudes forward more than the holding pallet 4A, for example, it is possible to accurately determine whether the cargo 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount.

[0124] Furthermore, in this embodiment, the centers of the front faces of the pallet 4 and the cargo 5 are identified based on the point clouds of the front areas of the pallet 4 and the cargo 5, and the distance from the forklift 2 to the centers of the front faces of the pallet 4 and the cargo 5 is calculated. This prevents shadows of the side areas of the pallet 4 and the cargo 5 in the point cloud data from affecting the detection of the packaging state of the pallet 4 and the cargo 5. This further improves the detection accuracy of the packaging state of the pallet 4 and the cargo 5.

[0125] It should be noted that the present invention is not limited to the above-described embodiment. For example, in the above-described embodiment, the regions of the pallet 4 and the luggage 5 in the image data are detected using instance segmentation, but the present invention is not particularly limited to this method, and the regions of the pallet 4 and the luggage 5 in the image data may be detected using bounding boxes such as object detection.

[0126] Furthermore, in the above embodiment, the front areas of the pallet 4 and the luggage 5 are extracted from the image data D, and the packing state of the pallet 4 and the luggage 5 is detected based on the front areas of the pallet 4 and the luggage 5, but this is not particularly limited to such an embodiment. For example, areas including the front, side, and bottom surfaces of the pallet 4 and the luggage 5 may be extracted from the image data D, and the packing state of the pallet 4 and the luggage 5 may be detected based on the areas of the pallet 4 and the luggage 5.

[0127] Furthermore, in the above embodiment, the camera 11 and the laser sensor 12 are mounted on the forklift 2, but the present invention is not limited to this particular form, and the camera 11 and the laser sensor 12 may be installed on-site.

[0128] Furthermore, in the above embodiment, the laser sensor 12 measures the distance to the pallet 4 and the luggage 5, and point cloud data of the pallet 4 and the luggage 5 is acquired, but this is not limited to a particular form, and instead of the laser sensor 12, a sensor capable of measuring the distance to an object and acquiring point cloud data, such as an RGBD camera, may be used.

[0129] Furthermore, in the above embodiment, when the forklift 2 handles the pallet 4 loaded on the loading platform 3a of the truck 3, the state of the packaging of the pallet 4 and the cargo 5 is detected. However, the present invention is not limited to the loading platform 3a of the truck 3, and can also be applied to cases where the pallet 4 is loaded on the floor of a factory, for example. [Explanation of symbols]

[0130] 2...forklift, 4...pallet, 4A...holding pallet (pallet to be held), 4Z...rear pallet, 4a...front (front), 5...luggage, 5a...front (front), 10...packing condition detection system, 11...camera (image acquisition unit), 12...laser sensor (point cloud acquisition unit), 41...area detection unit, 42...point cloud processing unit (distance calculation unit), 43...image-point cloud matching unit (distance calculation unit), 44...front and rear separation unit (separation unit), 45...luggage interference determination unit (interference determination unit), 46...overhang determination unit, D...image data.

Claims

1. A package state detection system that detects the package state of at least one tier of pallets arranged in multiple rows in a horizontal direction when a forklift is used to handle the pallets, and the package state of goods placed on the pallets, an image acquisition unit that captures images of the pallet and the luggage and acquires image data of the pallet and the luggage; a point cloud acquisition unit that measures distances to the pallet and the package and acquires point cloud data of the pallet and the package; an area detection unit that detects areas of the pallet and the luggage in the image data acquired by the image acquisition unit; a distance calculation unit that extracts a point cloud of the area of ​​the pallet and the luggage detected by the area detection unit based on the point cloud data acquired by the point cloud acquisition unit, and calculates a distance from the forklift to the pallet and the luggage based on the point cloud of the area of ​​the pallet and the luggage; and a dividing unit that divides the positional relationship of each pallet arranged in the multiple rows in the depth direction perpendicular to the lateral direction based on the distance from the forklift to the pallet and the cargo calculated by the distance calculation unit.

2. an interference determination unit that determines whether a pallet or a load in a row adjacent to the pallet to be held by the forklift is interfering with a pallet or load in the same row as the pallet to be held by the forklift, based on the areas of the pallet and the load detected by the area detection unit; The dividing unit identifies, among the pallets arranged in the plurality of rows, a pallet whose distance from the forklift is equal to or greater than a predetermined specified value as a rear pallet that is shifted toward the rear side in the depth direction, and excludes the rear pallet and the cargo placed on the rear pallet from objects to be subjected to the detection of the packaging state, 2. The packaging state detection system according to claim 1, wherein the interference determination unit determines whether a pallet or a piece of luggage in a row adjacent to the pallet to be held is interfering with a pallet or a piece of luggage in the same row as the pallet to be held, based on the area of ​​the pallet and the luggage detected by the area detection unit that has not been excluded from the packaging state detection target by the separation unit.

3. 3. The packaging state detection system according to claim 2, wherein when the interference determination unit determines that a package in a row adjacent to the pallet to be held is interfering with a pallet or package in the same row as the pallet to be held, the interference determination unit virtually shrinks the package in the adjacent row in the horizontal direction in the image data so that the end of the package on the interfering side in the adjacent row moves a specified amount along the horizontal direction to the opposite side from the interfering side, and in that state again determines whether the package in the adjacent row is interfering with the pallet or package in the same row as the pallet to be held.

4. 4. The packaging state detection system of claim 3, wherein when the interference determination unit determines that a package in a row adjacent to the pallet to be held is interfering with a pallet or package in the same row as the pallet to be held, the interference determination unit virtually shrinks the package in the adjacent row in the horizontal direction so that, in the image data, the edge of the package in the adjacent row that is interfering moves along the horizontal direction to the opposite side from the interfering side by a number of pixels corresponding to the distance from the forklift to the package in the adjacent row calculated by the distance calculation unit.

5. The forklift truck further includes a protrusion determination unit that determines whether a load or another pallet placed on a pallet to be held by the forklift protrudes from the pallet to be held by the forklift truck by a specified amount or more in the lateral direction, based on the areas of the pallet and the load detected by the area detection unit, The dividing unit identifies, among the pallets arranged in the plurality of rows, a pallet whose distance from the forklift is equal to or greater than a predetermined specified value as a rear pallet that is shifted toward the rear side in the depth direction, and excludes the rear pallet and the cargo placed on the rear pallet from objects to be subjected to the detection of the packaging state, 2. The packaging condition detection system according to claim 1, wherein the protrusion determination unit determines whether a cargo or another pallet placed on the pallet to be held protrudes laterally beyond the pallet to be held by more than the specified amount, based on the area of ​​the pallet and the cargo detected by the area detection unit that has not been excluded from the packaging condition detection target by the separation unit.

6. 6. The packaging condition detection system of claim 5, wherein the protrusion determination unit determines whether the luggage or other pallet placed on the pallet to be held protrudes laterally from the pallet to be held by more than the specified amount based on the area of ​​the pallet and luggage that has not been excluded from the packaging condition detection target by the separation unit and the distance from the forklift to the luggage or other pallet calculated by the distance calculation unit.

7. 2. The packaging state detection system according to claim 1, wherein the distance calculation unit extracts a point cloud of a front area of ​​the pallet and the luggage from the point cloud of the areas of the pallet and the luggage detected by the area detection unit, identifies a center of the front of the pallet and the luggage based on the point cloud of the front area of ​​the pallet and the luggage, and calculates a distance from the forklift to the center of the front of the pallet and the luggage.

Images