Packing state detection system

Abstract

To provide a packing state detection system capable of accurately detecting whether a packing state of a pallet and a load is abnormal.SOLUTION: A packing state detection system 10 comprises: a camera 11 that images pallets 4 and loads 5; a load set setting unit 45a that sets a plurality of rows of load sets S indicating the range of the pallets 4 and the loads 5 stacked in the vertical direction of images in regions of the pallets 4 and the loads 5 in image data; an interference region setting unit 45c that sets, as a load interference determination region R, a track range when the pallets 4 present in the load set S to be loaded are virtually moved upward; and an interference determination unit 45d that determines whether the loads 5 present in the load set S adjacent to the load set S to be loaded are included in the load interference determination region R to determine whether the loads 5 present in the adjacent load set S interfere with the pallets 4 present in the load set S to be loaded.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to a package state detection system. [Background technology]

[0002] For example, Patent Document 1 describes a technology in which an autonomous forklift equipped with sensors and cameras unloads goods loaded on a truck and loads them onto an autonomous unmanned guided vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-62964 Summary of the Invention [Problem to be solved by the invention]

[0004] In actual logistics situations, if the loading condition (packing condition) of the pallet and the cargo is abnormal, it may not be possible to unload the cargo using normal cargo handling control. Therefore, it is necessary to accurately detect whether the packing condition of the pallet and the cargo is abnormal, and if the packing condition is abnormal, to take appropriate measures according to the packing condition.

[0005] An object of the present invention is to provide a packaging state detection system that can accurately detect whether the packaging state of a pallet and cargo is abnormal. [Means for solving the problem]

[0006] (1) One aspect of the present invention is a packaging condition detection system that detects the packaging condition of a pallet and cargo placed on the pallet when loading and unloading at least one level of pallets arranged in multiple rows horizontally using a forklift, and includes an image acquisition unit that images the pallet and cargo and acquires image data of the pallet and cargo, an area detection unit that detects the area of ​​the pallet and cargo in the image data acquired by the image acquisition unit, a cargo set setting unit that sets multiple rows of cargo sets representing the range in which the pallet and cargo are stacked in the vertical direction of the image in the area of ​​the pallet and cargo in the image data detected by the area detection unit, an interference area setting unit that sets the trajectory range when a pallet present in a cargo set to be unloaded, among the multiple rows of cargo sets set by the cargo set setting unit, is virtually moved toward the top of the image, as a cargo interference determination area, and an interference determination unit that determines whether cargo present in a cargo set adjacent to the cargo set to be unloaded is interfering with a pallet present in the cargo set to be unloaded by determining whether the cargo interference determination area set by the interference area setting unit includes cargo present in a cargo set adjacent to the cargo set to be unloaded.

[0007] In such a packaging state detection system, image data of the pallet and the luggage is acquired by capturing an image of the pallet and the luggage. Then, in the area of ​​the pallet and luggage in the image data, multiple rows of luggage sets are set, representing the range in which the pallet and luggage are stacked in the vertical direction of the image. Then, a trajectory range when a pallet in one of the rows of luggage sets, which is the target of loading, is virtually moved upward in the image, is set as a luggage interference determination area. Then, by determining whether luggage in a luggage set adjacent to the target of loading is included in the luggage interference determination area, it is determined whether a pallet in the target of loading is interfering with a luggage in the target of loading. Therefore, it is possible to accurately determine whether a luggage adjacent in the horizontal direction is interfering with the target of loading. This allows for accurate detection of whether the packaging state of the pallet and the luggage is abnormal.

[0008] (2) In (1) above, the interference area setting unit sets the trajectory range when the pallet and luggage present in the luggage set to be loaded are virtually moved upward in the image as the luggage interference determination area, and the interference determination unit may determine whether luggage present in a luggage set adjacent to the luggage set to be loaded is interfering with the pallet or luggage present in the luggage set to be loaded by determining whether luggage present in a luggage set adjacent to the luggage set to be loaded is included in the luggage interference determination area.

[0009] With this configuration, by determining whether a package in an adjacent package set to the package set to be handled is included in the package interference detection area, it is possible to determine whether a package in an adjacent package set to the package set to be handled is interfering with a pallet or package in the package set to be handled. This allows for accurate determination of whether a package adjacent to the pallet or package to be handled is interfering with a package in the horizontal direction. This allows for even more accurate detection of whether the packaging state of the pallet and package is abnormal.

[0010] (3) In (1) or (2) above, the luggage interference detection area may be the trajectory range when a pallet present in the luggage set to be loaded is virtually moved to a position above the pallet in the image.

[0011] In this configuration, the cargo interference detection area is set to a range that includes a position above the pallet in the cargo set to be handled on the image, thereby making it possible to more accurately determine whether or not a cargo adjacent to the pallet to be handled is interfering with the cargo.

[0012] (4) In any of (1) to (3) above, the packaging condition detection system further includes a point cloud acquisition unit that measures the distance to the pallet and the luggage and acquires point cloud data of the pallet and the luggage; a distance calculation unit that extracts a point cloud of the area of ​​the pallet and luggage detected by the area detection unit based on the point cloud data acquired by the point cloud acquisition unit and calculates the distance from the forklift to the pallet and the luggage based on the point cloud of the area of ​​the pallet and luggage; and a virtual movement amount determination unit that determines a virtual movement amount of the pallet present in the luggage set to be handled based on the distance from the forklift to the pallet calculated by the distance calculation unit, and the interference area setting unit may set the trajectory range when the pallet present in the luggage set to be handled is virtually moved upward in the image by the virtual movement amount determined by the virtual movement amount determination unit as the luggage interference detection area.

[0013] When a pallet in a set of packages to be handled is virtually moved upward in the image by a predetermined amount, the number of pixels corresponding to the predetermined amount varies depending on the distance from the forklift to the pallet. Therefore, by determining the virtual movement amount of the pallet in the set of packages to be handled based on the distance from the forklift to the pallet, and setting the trajectory range when the pallet in the set of packages to be handled is virtually moved upward in the image by the virtual movement amount as the package interference detection area, the virtual movement amount of the pallet in the set of packages to be handled is constant regardless of the distance from the forklift to the pallet. Therefore, it is possible to more accurately determine whether a package adjacent to the pallet to be handled is interfering with the pallet in the horizontal direction.

[0014] (5) In any of (1) to (4) above, the area detection unit may detect the front area of ​​the pallet and luggage in the image data acquired by the image acquisition unit, and the luggage set setting unit may set multiple rows of luggage sets in the front area of ​​the pallet and luggage in the image data detected by the area detection unit.

[0015] This configuration prevents the shadows of the side areas of the pallet and the cargo from affecting the cargo interference determination in the image data, thereby enabling more accurate determination of whether or not a cargo adjacent in the horizontal direction is interfering with the pallet to be handled. [Effects of the Invention]

[0016] According to the present invention, it is possible to accurately detect whether the packaging state of the pallet and the package is abnormal. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram showing the configuration of a cargo handling control system including a package state detection system according to an embodiment of the present invention. [Figure 2] 2 is a side view of a forklift truck on which the cargo handling control system shown in FIG. 1 is mounted. [Figure 3] FIG. 1 is a side view showing a state in which a pallet on which cargo is placed is loaded onto the bed of a truck. [Figure 4] 10A and 10B are diagrams showing a state in which a pallet and a load interfere with an adjacent pallet or load, and a state in which a load placed on a lower pallet or an upper pallet protrudes from the lower pallet. [Figure 5] 2 is a block diagram showing the functions of an area detection unit shown in FIG. 1. FIG. [Figure 6] FIG. 6 is a block diagram showing a function for creating the learned data shown in FIG. 5. [Figure 7] 2 is a flowchart showing a procedure of point cloud processing executed by the point cloud processing unit shown in FIG. 1. [Figure 8] 2 is a flowchart showing the procedure of a calculation process executed by an image-point cloud matching unit shown in FIG. 1. [Figure 9] 2 is a flowchart showing the procedure of a separation process executed by a front-rear separation unit shown in FIG. 1. [Figure 10] 2 is a functional block diagram of a luggage interference determination unit shown in FIG. 1. FIG. [Figure 11]11 is a flowchart showing the procedure of a luggage interference determination process executed by a luggage interference determination unit shown in FIG. [Figure 12] 12 is a diagram showing how the luggage interference determination unit shown in FIG. 11 determines whether or not there is luggage interference. FIG. [Figure 13] 2 is a flowchart showing the procedure of a protrusion determination process executed by a protrusion determination unit shown in FIG. 1; [Figure 14] 14 is a diagram showing how the protrusion determination unit shown in FIG. 13 determines whether or not a package or pallet is protruding. FIG. [Figure 15] FIG. 10 is a diagram showing how the amount of overhang of an upper pallet relative to a lower pallet is calculated. [Figure 16] FIG. 10 is a diagram showing how, when a pallet and cargo are present behind the pallet and cargo to be handled, it is erroneously determined that the pallet and cargo present behind interfere with the pallet and cargo to be handled. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0019] Fig. 1 is a block diagram showing the configuration of a cargo handling control system equipped with a package state detection system according to one embodiment of the present invention. In Fig. 1, the cargo handling control system 1 is mounted on a forklift 2 as shown in Fig. 2. The forklift 2 is, for example, a counter-load forklift.

[0020] The forklift 2 has a body 31, a pair of left and right front wheels 32 which are drive wheels arranged at the front of the body 31, a pair of left and right rear wheels 33 which are steering wheels arranged at the rear of the body 31, a mast 34 attached to the front end of the body 31, a pair of left and right forks 36 attached to the mast 34 via lift brackets 35 so that they can be raised and lowered, a lift cylinder 37 which raises and lowers the forks 36 via the lift brackets 35, and a tilt cylinder 38 which tilts the mast 34.

[0021] The cargo handling control system 1 is a system that controls cargo handling when cargo handling is automatically performed by a forklift 2. Here, as shown in FIG. 3, the cargo handling control system 1 controls the forklift 2 when holding a pallet 4 loaded on a loading platform 3a of a truck 3 with the forks 36 of the forklift 2, that is, when carrying out what is called cargo pick-up.

[0022] The pallet 4 is, for example, a flat pallet made of plastic or wood. The pallet 4 has a square or nearly square shape in a plan view. Cargo 5 is placed on the pallet 4. The pallet 4 has two fork holes 6 into which forks 36 are inserted. The fork holes 6 extend from the front face (front surface) 4a of the pallet 4 toward the rear. The pallet 4 may also be a basket pallet or the like.

[0023] Pallets 4 are loaded in multiple rows along the front-to-rear direction of the truck 3 on the loading platform 3a of the truck 3. The pallets 4 are arranged with their front faces 4a facing the side of the truck 3. Therefore, the forklift 2 picks up cargo from the side of the truck 3. At this time, the multiple rows of pallets 4 are arranged side by side in the horizontal direction (left-right direction) as viewed from the forklift 2. The forklift 2 loads and unloads pallets 4 in one level or multiple levels (upper and lower levels) on which cargo 5 is placed (see Figure 4(c) etc.). For example, when the forklift 2 picks up cargo from a two-level pallet 4, the forks 36 are inserted into the fork holes 6 of the lower level pallet 4.

[0024] 3, when the pallet 4 and the cargo 5 are properly packed, the cargo can be smoothly picked up by the forklift 2. The packing state refers to the way the pallet 4 and the cargo 5 are placed.

[0025] However, as shown in Figure 4, if the packaging state of the pallets 4 and the cargo 5 is abnormal, it may be impossible to load the cargo using the forklift 2. For example, as shown in Figure 4(a), if laterally adjacent pallets 4 interfere with each other in the width direction of the truck 3 (the direction in which the pallets 4 are pulled out from the loading platform 3a to the side of the truck 3), the cargo may collapse when being loaded.

[0026] Furthermore, as shown in Figure 4(b), if a load 5 placed on a pallet 4 interferes with a load 5 placed on a laterally adjacent pallet 4, or if a load 5 placed on a pallet 4 interferes with a laterally adjacent pallet 4, the load may collapse during loading. Here, interference refers to a state in which a load 5 placed on a pallet 4 rests on a laterally adjacent pallet 4 or load 5.

[0027] Furthermore, as shown in Figure 4(c), due to vibrations during transport by truck 3, cargo 5 may protrude significantly laterally from pallets 4, or an upper pallet 4 may protrude significantly laterally from a lower pallet 4. In this case, too, there is a possibility that cargo may collapse when unloading.

[0028] To solve such problems, the cargo handling control system 1 detects whether the packing state of the pallet 4 and the cargo 5 is abnormal, and if the packing state of the pallet 4 and the cargo 5 is abnormal, performs appropriate control according to the packing state.

[0029] Therefore, the cargo handling control system 1 is equipped with a packaging state detection system 10 of this embodiment that detects whether the packaging state of the pallets 4 and cargo 5 is normal or abnormal. The packaging state detection system 10 is a system that detects the packaging state of the pallets 4 and cargo 5 when the forklift 2 handles cargo on at least one layer of pallets 4 arranged in multiple rows in the horizontal direction.

[0030] The cargo handling control system 1 includes a camera 11, a laser sensor 12, a drive unit 13, and a controller 14. The camera 11, the laser sensor 12, the drive unit 13, and the controller 14 are mounted on the forklift 2.

[0031] The camera 11 is an image acquisition unit that captures images of the pallet 4 and the luggage 5 and acquires image data of the pallet 4 and the luggage 5. The camera 11 captures images of a range including the front of the pallet 4 and the luggage 5.

[0032] The laser sensor 12 is a point cloud acquisition unit that measures the distance to the pallet 4 and the luggage 5 and acquires point cloud data of the pallet 4 and the luggage 5. The laser sensor 12 acquires point cloud data of the pallet 4 and the luggage 5 by emitting a laser toward the pallet 4 and the luggage 5 and receiving the reflected laser light. The laser sensor 12 emits a laser toward an area that includes the front of the pallet 4 and the luggage 5. The point cloud is a collection of reflected points of the laser. The point cloud data of the pallet 4 and the luggage 5 includes the distance to the pallet 4 and the luggage 5. A LiDAR, a laser range finder, or the like is used as the laser sensor 12.

[0033] The drive unit 13 has the above-mentioned lift cylinder 37 and tilt cylinder 38, a travel motor (not shown) that rotates the front wheels 32 of the forklift 2, and a steering motor (not shown) that steers the rear wheels 33 of the forklift 2.

[0034] The controller 14 is configured with a CPU, RAM, ROM, an input / output interface, etc. The controller 14 has an area detection unit 41, a point cloud processing unit 42, an image-point cloud matching unit 43, a front / rear separation unit 44, a luggage interference determination unit 45, an overhang determination unit 46, and a cargo handling control unit 47.

[0035] The camera 11, laser sensor 12, area detection unit 41, point cloud processing unit 42, image-point cloud matching unit 43, front and rear separation unit 44, luggage interference determination unit 45 and overhang determination unit 46 constitute the above-mentioned packaging state detection system 10.

[0036] The area detection unit 41 detects the areas of the pallet 4 and the luggage 5 in the image data acquired by the camera 11. The area detection unit 41 detects the areas of the pallet 4 and the luggage 5 in pixel units of the image data.

[0037] The area detection unit 41 extracts the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 (see FIG. 3) in the image data. The front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 are surfaces that face the forklift 2 that picks up the luggage. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 correspond to the front areas of the pallet 4 and the luggage 5.

[0038] 5, the area detection unit 41 is configured with a function having a package shape recognition model 20 and learned data 21. The area detection unit 41 reads the learned data 21 into the package shape recognition model 20, thereby extracting the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the image data acquired by the camera 11.

[0039] The package recognition model 20 is an instance segmentation model built using deep learning. Instance segmentation is a technology that estimates the position of individual objects on a pixel-by-pixel basis, and estimates the object position by labeling the target object area in detail and performing learning. The package recognition model 20 is built using data in which the front areas of the pallet 4 and luggage 5 are annotated (added) to images of the pallet 4 and luggage 5.

[0040] The package recognition model 20 has a feature extraction unit 22 that uses trained data 21 to extract features of the package image acquired by the camera 11, and an area recognition unit 23 that uses the trained data 21 and the features of the package image to recognize the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the image data.

[0041] The trained data 21 consists of two types of data: a package image of the pallet 4 and the luggage 5, and a data file containing information about the pallet 4 and the luggage 5 in the package image. The information about the pallet 4 and the luggage 5 in the package image includes pixel numbers of the package image. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the package image are represented by pixel numbers of the package image.

[0042] For example, if the number of types of luggage 5 is two, namely, cardboard boxes and parts boxes, and the number of types of pallets 4 is two, namely, plastic pallets and post pallets, learned data 21 is prepared in which labels are assigned to the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 for a total of four types of packaging images in which the pallets 4 and luggage 5 are photographed.

[0043] The trained data 21 is generated by pre-training a mathematical formula-driven database, as shown in Fig. 6. The mathematical formula-driven database is an image generated from a mathematical formula, and is data that does not require labeling of the image. For example, a dataset (FractalDB) composed of fractal geometric images is used as the mathematical formula-driven database.

[0044] Specifically, as shown in Fig. 6(a), first, pre-training of a formula-driven database 25 is performed using an image classification model 24 to generate pre-trained data 26. Formula-driven supervised learning is used for the pre-training of the formula-driven database 25. The image classification model 24 has a feature extraction unit 27 that extracts features of the formula-driven database 25, and an image classification unit 28 that solves an image classification problem using the features of the formula-driven database 25.

[0045] 6(b), transfer learning is performed using pre-trained data 26 to construct the above-mentioned packaging shape recognition model 20, thereby creating trained data 21. At this time, pre-trained data 26, which has been pre-trained using formula-driven database 25, is read into feature extraction unit 22, and data in which the areas of front surface 4a of pallet 4 and front surface 5a of package 5 are assigned to package shape image 29 in which pallet 4 and package 5 are photographed is subjected to transfer learning, thereby acquiring trained data 21.

[0046] The area detection unit 41 outputs the front area data of the pallet 4 and the luggage 5 in the image data to the image-point cloud matching unit 43, the luggage interference determination unit 45, and the overhang determination unit 46. The front area data of the pallet 4 and the luggage 5 includes the position of the center of gravity of the front surface 4a of the pallet 4 and the position of the center of gravity of the front surface 5a of the luggage 5.

[0047] 1, the point cloud processing unit 42 inputs the point cloud data of the pallet 4 and the luggage 5 acquired by the laser sensor 12 and performs predetermined point cloud processing on the point cloud data. The point cloud data acquired by the laser sensor 12 is represented in the sensor coordinate system.

[0048] Fig. 7 is a flowchart showing the procedure of point cloud processing executed by the point cloud processing unit 42. In Fig. 7, the point cloud processing unit 42 first acquires point cloud data of the laser sensor 12 (step S111).

[0049] Next, the point cloud processing unit 42 converts the point cloud data in the sensor coordinate system into point cloud data in the camera coordinate system using the camera-sensor external parameters acquired in advance (step S112). This makes it possible to calculate the distance from the camera 11 using the point cloud coordinates.

[0050] Next, the point cloud processing unit 42 uses the camera parameters acquired in advance to project the point cloud data in the camera coordinate system onto an image plane to generate a distance image (step S113). The distance image is an image in which distance is represented by color. The point cloud processing unit 42 then outputs the point cloud data in the camera coordinate system and the distance image to the image-point cloud matching unit 43 (step S114).

[0051] Returning to Figure 1, the image-point cloud matching unit 43 matches the image data with the point cloud data based on the front area of ​​the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41 and the point cloud data and distance image in the camera coordinate system obtained by the point cloud processing unit 42, and calculates the distance from the forklift 2 to the pallet 4 and luggage 5.

[0052] Fig. 8 is a flowchart showing the procedure of the calculation process executed by the image-point cloud matching unit 43. In Fig. 8, the image-point cloud matching unit 43 first acquires front area data of the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41, and point cloud data and a range image in the camera coordinate system obtained by the point cloud processing unit 42 (step S121).

[0053] Next, the image-point cloud matching unit 43 extracts a point cloud on the distance image corresponding to the front area of ​​the pallet 4 and luggage 5 in the image data, taking advantage of the correspondence between the pixel coordinates of the areas of the pallet 4 and luggage 5 in the image data and the pixel coordinates of the distance image (step S122).

[0054] Next, the image-point cloud matching unit 43 takes advantage of the correspondence between the point cloud on the distance image and the point cloud in the camera coordinate system to extract a point cloud of the area of ​​the pallet 4 and luggage 5 in the camera coordinate system from the point cloud on the distance image extracted in step S122 (step S123).

[0055] Next, the image-point cloud matching unit 43 extracts a point cloud of the front area of ​​the pallet 4 and luggage 5 in the camera coordinate system using a point cloud processing algorithm such as RANSAC (Random Sample Consensus) that extracts a plane from a point cloud (step S124).

[0056] Next, the image-point cloud matching unit 43 identifies the center coordinates of the pallet 4 and the luggage 5 from the point cloud coordinates of the front areas of the pallet 4 and the luggage 5 in the camera coordinate system (step S125). Then, the image-point cloud matching unit 43 calculates the distance from the camera 11 to the center coordinates of the pallet 4 and the luggage 5 as the distance from the forklift 2 to the front of the pallet 4 and the luggage 5 (step S126). The camera 11 is the origin of the camera coordinate system.

[0057] Next, the image-point cloud matching unit 43 outputs point cloud data of the front area of ​​the pallet 4 and luggage 5 in the camera coordinate system and data on the distance from the forklift 2 to the front of the pallet 4 and luggage 5 to the front / rear separation unit 44 (step S127).

[0058] Returning to Figure 1, the point cloud processing unit 42 and the image-point cloud matching unit 43 constitute a distance calculation unit that extracts a point cloud of the area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41 based on the point cloud data acquired by the laser sensor 12, and calculates the distance from the forklift 2 to the pallet 4 and luggage 5 based on the point cloud of the area of ​​the pallet 4 and luggage 5.

[0059] The front-rear separating unit 44 separates the positional relationship in the depth direction (front-rear direction) of each of the pallets 4 arranged in multiple rows based on the distance from the forklift 2 to the pallets 4 and the cargo 5 calculated by the image-point cloud matching unit 43. The depth direction is the direction perpendicular to the lateral direction.

[0060] The front / rear separating unit 44 recognizes, among the pallets 4 arranged in multiple rows, the pallets 4 whose distance from the forklift 2 is greater than a predetermined specified value as the rear pallet 4Z (see Figure 16) that is shifted to the rear in the depth direction (rear in the front / rear direction), and excludes the rear pallet 4Z and the rear cargo 5Z (see Figure 16) that is the cargo 5 placed on the rear pallet 4Z from the package state detection targets for detecting the package state.

[0061] Fig. 9 is a flowchart showing the procedure of the separation process executed by the front / rear separation unit 44. In Fig. 9, the front / rear separation unit 44 first acquires point cloud data of the front area of ​​the pallet 4 and the cargo 5, and distance data from the forklift 2 to the front of the pallet 4 and the cargo 5 (step S131).

[0062] Next, the front / rear separating unit 44 determines whether the distance from the forklift 2 to the front of the pallet 4 and the package 5 is equal to or greater than a predetermined specified value (step S132).

[0063] When the front / rear separating unit 44 determines that the distance from the forklift 2 to the front of the pallet 4 and the cargo 5 is equal to or greater than a specified value, it recognizes the pallet 4 and the cargo 5 as the rear pallet 4Z and the rear cargo 5Z (step S133).Then, the front / rear separating unit 44 excludes the rear pallet 4Z and the rear cargo 5Z from the package state detection targets (step S134).

[0064] When the front / rear separating unit 44 determines that the distance from the forklift 2 to the front of the pallet 4 and the cargo 5 is less than the specified value, it does not execute steps S133 and S134. Therefore, the pallet 4 and the cargo 5 whose distance from the forklift 2 is shorter than the specified value are not excluded from the packing state detection targets.

[0065] Next, the front / rear separating unit 44 outputs distance data from the forklift 2 to the front of the pallet 4 and the luggage 5 for which the packaging state is to be detected, to the luggage interference determining unit 45 and the protrusion determining unit 46 (step S135).

[0066] Returning to Figure 1, the luggage interference determination unit 45 determines, based on the front area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41, whether a luggage 5 located in a row adjacent to the pallet 4 to be held by the forklift 2 is interfering with a pallet 4 or luggage 5 located in the same row as the pallet 4 to be held by the forklift 2.

[0067] Here, the pallet 4 to be held is referred to as holding pallet 4A (see FIG. 12, etc.). Holding pallet 4A is the bottommost pallet 4. Pallets 4 existing in the same row as holding pallet 4A include holding pallet 4A itself.

[0068] The luggage interference determination unit 45 determines whether luggage 5 present in a row adjacent to the holding pallet 4A is interfering with a pallet 4 or luggage 5 present in the same row as the holding pallet 4A, based on the front areas of the pallet 4 and luggage 5 detected by the area detection unit 41 that have not been excluded from the package state detection target by the front / rear separation unit 44.

[0069] As shown in FIG. 10, the luggage interference determination unit 45 includes a luggage setting unit 45a, a virtual movement amount determination unit 45b, an interference area setting unit 45c, and an interference determination unit 45d.

[0070] The luggage set setting unit 45a sets a plurality of rows of luggage sets S (see FIG. 12(b)) representing the range in which the pallets 4 and luggage 5 are stacked in the vertical direction of the image in the front area of ​​the pallets 4 and luggage 5 in the image data detected by the area detection unit 41. The vertical direction of the image is the Y-axis direction in FIG. 12(a).

[0071] The virtual movement amount determination unit 45b determines the virtual movement amount m (see Figure 12(c)) of the pallets 4 and luggage 5 present in the luggage set S to be handled among the multiple rows of luggage sets S set by the luggage set setting unit 45a, based on the distance from the forklift 2 to the pallet 4 calculated by the image-point cloud matching unit 43.

[0072] The interference area setting unit 45c sets, as an area for determining whether or not a luggage interference occurs, a trajectory range of a pallet 4 and luggage 5 present in a luggage set S to be handled among the multiple rows of luggage sets S set by the luggage set setting unit 45a when the pallet 4 and luggage 5 are virtually moved upward in the image. The upward direction of the image is the direction toward the origin O of the Y axis in FIG. 12(a) (the direction of arrow H).

[0073] The interference area setting unit 45c sets the trajectory range when the pallet 4 and luggage 5 present in the luggage set S to be loaded are virtually moved upward in the image by the virtual movement amount m determined by the virtual movement amount determination unit 45b as the luggage interference determination area R.

[0074] The interference determination unit 45d determines whether the luggage 5 present in the luggage set S adjacent to the luggage set S to be loaded is interfering with the pallet 4 or luggage 5 present in the luggage set S to be loaded by determining whether the luggage interference determination area R set by the interference area setting unit 45c includes luggage 5 present in the luggage set S adjacent to the luggage set S to be loaded.

[0075] FIG. 11 is a flowchart showing the procedure of the luggage interference determination process executed by the luggage interference determination unit 45.

[0076] In Fig. 11, the luggage interference determination unit 45 first acquires front area data of the pallet 4 and luggage 5 in the image data obtained by the area detection unit 41 (step S141). Fig. 12(a) shows an example of the front area of ​​the pallet 4 and luggage 5 in the image data D obtained by the area detection unit 41. In Fig. 12(a), the X-axis direction indicates the left-right direction (horizontal direction) of the pallet 4 and luggage 5, and the Y-axis direction indicates the up-down direction (height direction) of the pallet 4 and luggage 5. In the X-axis direction, the right side is the positive (+) direction, and in the Y-axis direction, the bottom side is the positive (+) direction.

[0077] Next, the luggage interference determination unit 45 creates multiple luggage sets S in the front area of ​​the pallets 4 and luggage 5 in the image data (step S142). As shown in FIG. 12(b), the luggage set S represents a rectangular area including the pallets 4 and luggage 5 that are adjacent to each other in the vertical direction in the image data D, from the center of gravity position G of the front area of ​​the pallets 4 and luggage 5. In the image data D shown in FIG. 12(b), two luggage sets Sa and Sb that are adjacent in the horizontal direction are created. The luggage sets Sa and Sb include two tiers of pallets 4 and luggage 5, one above the other.

[0078] Next, the cargo interference determination unit 45 acquires the distance data from the forklift 2 to the front of the pallet 4 and cargo 5 calculated by the front / rear separating unit 44 (step S143).

[0079] Then, the cargo interference determination unit 45 determines the virtual movement amount m of the pallet 4 and cargo 5 present in the cargo set S to be handled in the image data based on the distance from the forklift 2 to the front of the pallet 4 and cargo 5 (step S144). Here, the virtual movement amount m of the pallet 4 and cargo 5 is the number of pixels equivalent to a predetermined specified amount. The number of pixels equivalent to the specified amount differs depending on the distance from the forklift 2 to the pallet 4 and cargo 5.

[0080] Next, the luggage interference determination unit 45 sets the trajectory range when the pallet 4 and luggage 5 present in the luggage set S to be loaded in the image data are virtually moved upward in the image (toward the negative Y-axis direction) by a virtual movement amount m as the luggage interference determination area R (step S145).

[0081] 12(c), the cargo interference detection region R includes an interference detection region R1, which is the trajectory range when a pallet 4 present in the cargo set S to be handled is virtually moved, and an interference detection region R2, which is the trajectory range when a cargo 5 present in the cargo set S to be handled is virtually moved. The interference detection region R1 is the trajectory range when the lower and upper pallets 4 present in the cargo set S to be handled are moved to positions above the lower and upper pallets 4, respectively, in the image. In this case, the virtual movement amounts m of the pallets 4 and cargo 5 are equal.

[0082] The luggage interference determination unit 45 determines whether luggage 5 of luggage set S other than the loading target is included in luggage interference determination area R of the loading target (step S146). In the image data D shown in FIG. 12(d), luggage 5 of luggage set S other than the loading target is included in luggage interference determination area R of the loading target. Specifically, luggage 5 of luggage set S adjacent to the right is included in interference determination area R1, which is the trajectory range when the upper pallet 4 is virtually moved upward in the image by a virtual movement amount m.

[0083] When the luggage interference determination unit 45 determines that luggage 5 of a luggage set S other than the luggage to be handled is included within the luggage interference determination area R of the luggage to be handled, it determines that luggage 5 of the luggage set S other than the luggage to be handled is interfering with the pallet 4 or luggage 5 of the luggage set S of the luggage to be handled (step S147).

[0084] In the image data D shown in Figure 12(d), it is determined that a piece of luggage 5 from a set S of luggage other than the one to be handled is interfering with the upper pallet 4 of the set S of luggage to be handled. Therefore, that piece of luggage 5 is an interfering piece of luggage 5K that is interfering with the upper pallet 4 of the set S of luggage to be handled.

[0085] Then, the luggage interference determination unit 45 outputs an interference abnormality control signal to the cargo handling control unit 47 (step S148).

[0086] When it is determined in step S146 that no luggage 5 of the luggage set S other than the luggage to be handled is included in the luggage interference determination region R of the luggage to be handled, the luggage interference determination unit 45 determines that the luggage 5 of the luggage set S other than the luggage to be handled is not interfering with the pallet 4 or luggage 5 of the luggage set S of the luggage to be handled (step S149).Then, the luggage interference determination unit 45 outputs a normal control signal to the luggage handling control unit 47 (step S150).

[0087] Here, the luggage set setting unit 45a executes steps S141 and S142. The virtual movement amount determination unit 45b executes steps S143 and S144. The interference area setting unit 45c executes step S145. The interference determination unit 45d executes steps S146, S147, and S149.

[0088] Returning to Figure 1, the protrusion determination unit 46 determines, based on the front area of ​​the pallet 4 and cargo 5 detected by the area detection unit 41, whether the cargo 5 or other pallets 4 placed on the pallet 4 (holding pallet 4A) to be held by the forklift 2 protrudes laterally from the holding pallet 4A by more than a specified amount.

[0089] The protrusion determination unit 46 determines whether the luggage 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount based on the front area of ​​the pallet 4 and luggage 5 detected by the area detection unit 41 that has not been excluded from the package state detection target by the front / rear separation unit 44.

[0090] The protrusion determination unit 46 determines whether the luggage 5 or other pallets 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by more than a specified amount based on the front area of ​​the pallet 4 and luggage 5 that have not been excluded from the package state detection targets by the front / rear separation unit 44 and the distance from the forklift 2 to the luggage 5 or other pallets 4 placed on the holding pallet 4A calculated by the image-point cloud matching unit 43.

[0091] FIG. 13 is a flowchart showing the procedure of the protrusion determination process executed by the protrusion determination unit 46.

[0092] 13, the protrusion determination unit 46 first acquires front area data of the pallet 4 and the luggage 5 in the image data obtained by the area detection unit 41 (step S171). FIG. 14(a) shows an example of the front area of ​​the pallet 4 and the luggage 5 in the image data D obtained by the area detection unit 41. In FIG. 14(a), the X-axis direction and the Y-axis direction are the same as those in FIG. 12(a).

[0093] Next, the overhang determination unit 46 creates multiple package sets S in the front areas of the pallets 4 and packages 5 in the image data (step S172). As shown in FIG. 14(b), the package sets S represent rectangular areas including the pallets 4 and packages 5 that are adjacent to each other in the vertical direction in the image data D, starting from the center of gravity G of the front areas of the pallets 4 and packages 5. In the image data D shown in FIG. 14(b), two package sets Sa and Sb that are adjacent to each other in the horizontal direction are created. The package sets Sa and Sb include two tiers of pallets 4 and packages 5, one above the other.

[0094] Next, as shown in Figure 14(c), the overhang determination unit 46 extends two virtual lines L1 in the image data D from both the left and right ends of the lowest pallet 4, which is the holding pallet 4A of the cargo set S to be loaded, in the vertical direction toward the top end E of the image (step S173).

[0095] Then, as shown in Figure 14(c), the overhang determination unit 46 sets the area in the image data D surrounded by two virtual lines L1, a horizontal line L2 along the bottom edge of the lowest pallet 4, and a horizontal line L3 along the top edge E of the image as the overhang determination area Q of the loading object (step S174).

[0096] Next, the protrusion determination unit 46 determines whether or not there is any luggage 5 or pallet 4 in the luggage set S to be handled that protrudes laterally from the protrusion determination area Q (step S175). In the image data D shown in Figure 14(c), the right end of the upper pallet 4 protrudes laterally from the protrusion determination area Q.

[0097] When the overhang determination unit 46 determines that there is luggage 5 or a pallet 4 that is protruding laterally from the overhang determination area Q, it acquires distance data calculated by the front / rear separation unit 44 from the forklift 2 to the front of the pallet 4 and luggage 5 whose packaging state is to be detected (step S176).

[0098] Then, the overhang determination unit 46 calculates the amount of overhang of the luggage 5 or pallet 4 from the overhang determination area Q based on the distance from the forklift 2 to the luggage 5 or pallet 4 that overhangs laterally from the overhang determination area Q (step S177). As shown in FIG. 15, the length d per pixel varies depending on the distance from the forklift 2 to the luggage 5 or pallet 4, so the amount of overhang f of the luggage 5 or pallet 4 from the overhang determination area Q is calculated from the distance from the forklift 2 to the luggage 5 or pallet 4.

[0099] Next, the overhang determination unit 46 determines whether the overhang amount f of the luggage 5 or pallet 4 from the overhang determination area Q in the luggage set S to be handled is equal to or greater than a predetermined threshold (step S178). The threshold is set to a value that does not affect the loading operation of the pallet 4 and luggage 5 even if the luggage 5 or pallet 4 overhangs the overhang determination area Q in the lateral direction, for example, a value that does not cause the luggage 5 to collapse.

[0100] When the protrusion determination unit 46 determines that the protrusion amount f of the cargo 5 or pallet 4 with respect to the protrusion determination area Q is equal to or greater than the threshold value, it determines that the cargo 5 or other pallet 4 placed on the holding pallet 4A protrudes laterally from the holding pallet 4A by a specified amount or more (step S179).Then, the protrusion determination unit 46 outputs a protrusion abnormality control signal to the cargo handling control unit 47 (step S180).

[0101] When the overhang determination unit 46 determines in step S175 that no luggage 5 or pallet 4 is present that protrudes laterally from the overhang determination area Q, or when the overhang determination unit 46 determines in step S178 that the overhang amount f of the luggage 5 and pallet 4 from the overhang determination area Q is not equal to or greater than the threshold, the overhang determination unit 46 determines that the luggage 5 and other pallets 4 placed on the holding pallet 4A do not protrude laterally from the holding pallet 4A by more than a specified amount (step S181).The overhang determination unit 46 then outputs a normal control signal to the cargo handling control unit 47 (step S182).

[0102] Returning to Fig. 1, the cargo handling control unit 47 controls the drive unit 13 in accordance with the determination results by the luggage interference determination unit 45 and the overhang determination unit 46. When the cargo handling control unit 47 receives a normal control signal from both the luggage interference determination unit 45 and the overhang determination unit 46, it controls the drive unit 13 to perform normal cargo handling control. When the cargo handling control unit 47 receives an interference abnormality control signal from the luggage interference determination unit 45 or an overhang abnormality control signal from the overhang determination unit 46, it controls the drive unit 13 to perform abnormal cargo handling control.

[0103] In the above, when the forklift 2 is used to pick up goods, the forklift 2 travels toward the truck 3. Then, when the forklift 2 reaches the side of the truck 3, the forklift 2 stops temporarily. In this state, the pallets 4 and goods 5 loaded on the truck 3 are imaged by the camera 11, and image data D of the pallets 4 and goods 5 is acquired.

[0104] Then, the front areas of the pallet 4 and luggage 5 in the image data D are extracted. Then, based on the front areas of the pallet 4 and luggage 5 in the image data D, it is determined whether luggage 5 in an adjacent luggage set S interferes with luggage 5 or pallet 4 in the luggage set S to be handled. Also, based on the front areas of the pallet 4 and luggage 5 in the image data D, it is determined whether luggage 5 or other pallets 4 placed on the holding pallet 4A protrude laterally beyond the holding pallet 4A by more than a specified amount.

[0105] Then, when it is determined that the luggage 5 and pallet 4 of the luggage set S to be handled are not interfering with the luggage 5 of the adjacent luggage set S, and when it is determined that the luggage 5 and other pallets 4 placed on the holding pallet 4A do not protrude laterally beyond the holding pallet 4A by more than a specified amount, the drive unit 13 is controlled to unload the luggage set S to be handled.

[0106] As described above, in this embodiment, image data of the pallets 4 and the luggage 5 is acquired by capturing images of the pallets 4 and the luggage 5. Then, in the area of ​​the pallets 4 and the luggage 5 in the image data, multiple rows of luggage sets S are set, representing the range in which the pallets 4 and luggage 5 are stacked in the vertical direction of the image. A trajectory range when a pallet 4 in the luggage set S to be handled among the multiple rows of luggage sets S is virtually moved upward in the image is set as a luggage interference determination region R. Then, by determining whether luggage 5 in a luggage set S adjacent to the luggage set S to be handled is included in the luggage interference determination region R, it is determined whether a pallet 4 in the luggage set S to be handled is interfering with a luggage 5 in the luggage set S adjacent to the luggage set S to be handled. Therefore, it is accurately determined whether a luggage 5 adjacent in the horizontal direction is interfering with the pallet 4 to be handled. This allows accurate detection of whether the packaging state of the pallets 4 and luggage 5 is abnormal.

[0107] Furthermore, in this embodiment, the trajectory range of the pallets 4 and packages 5 present in the package set S to be handled is virtually moved upward in the image, and is set as the package interference determination area R. Then, by determining whether packages 5 present in a package set S adjacent to the package set S to be handled are included in the package interference determination area R, it is determined whether the pallets 4 or packages 5 present in the package set S to be handled are being interfered with by packages 5 present in the package set S adjacent to the package set S to be handled. Therefore, it is accurately determined whether the pallets 4 or packages 5 to be handled are being interfered with by packages 5 adjacent in the horizontal direction. This makes it possible to more accurately detect whether the packaging state of the pallets 4 and packages 5 is abnormal.

[0108] Furthermore, in this embodiment, the luggage interference determination region R is the trajectory range when a pallet 4 present in the luggage set S to be handled is virtually moved to a position above the pallet 4 on the image. For this reason, a range including a position above the pallet 4 present in the luggage set S to be handled on the image is set as the luggage interference determination region R. Therefore, it is possible to more accurately determine whether a luggage 5 adjacent in the horizontal direction is interfering with the pallet 4 to be handled.

[0109] Furthermore, in this embodiment, when the pallets 4 and packages 5 present in the package set S to be handled are virtually moved upward in the image by a predetermined specified amount, the number of pixels corresponding to the specified amount varies depending on the distance from the forklift 2 to the pallets 4 and packages 5. Therefore, a virtual movement amount m of the pallets 4 and packages 5 present in the package set S to be handled is determined according to the distance from the forklift 2 to the pallets 4 and packages 5, and the trajectory range when the pallets 4 and packages 5 present in the package set S to be handled are virtually moved upward in the image by the virtual movement amount m is set as the package interference detection region R. This makes it possible to keep the virtual movement amount m of the pallets 4 and packages 5 present in the package set S to be handled constant regardless of the distance from the forklift 2 to the pallets 4 and packages 5. Therefore, it is possible to more accurately determine whether a package 5 adjacent in the horizontal direction is interfering with the pallet 4 or package 5 to be handled.

[0110] Furthermore, in this embodiment, multiple rows of luggage sets S are set in the front area of ​​the pallet 4 and luggage 5 in the image data. This prevents shadows or the like in the side areas of the pallet 4 and luggage 5 in the image data from affecting the interference determination of the luggage 5. This makes it possible to more accurately determine whether or not a luggage 5 adjacent in the horizontal direction is interfering with the pallet 4 to be handled.

[0111] In addition, in this embodiment, by setting the trajectory range when the pallet 4 and luggage 5 present in the luggage set S to be loaded are virtually moved upward in the image as the luggage interference detection area R, not only can the accuracy of detecting interference abnormalities in the luggage 5 be improved, but the interference detection process can also be simplified.

[0112] Furthermore, in an actual logistics site, as shown in FIG. 16(a), there may be another pallet 4 and another piece of luggage 5 behind the pallet 4 and piece of luggage 5 to be handled. In this case, when the camera 11 captures images of both the front and rear pallets 4 and pieces of luggage 5, image data D showing the front and rear pallets 4 and pieces of luggage 5 is acquired. However, because the image data D is only two-dimensional information, it is not possible to determine whether the pallet 4 and piece of luggage 5 are in front of or behind the camera 11.

[0113] When such image data D is used to detect abnormalities in the packaging state, as shown in Figure 16(b), the cargo interference detection area R for the cargo to be handled includes pallets 4 and cargo 5 other than the cargo to be handled, even though the packaging state is normal. As a result, it is erroneously determined that the pallets 4 and cargo 5 located at the rear (back side) are interfering with the pallets 4 and cargo 5 located at the front (near side).

[0114] On the other hand, in this embodiment, the positional relationship in the depth direction perpendicular to the lateral direction of each pallet 4 arranged in multiple rows is distinguished based on the distance from the forklift 2 to the pallets 4 and packages 5. Therefore, even if a pallet 4 and package 5 are present behind the pallet 4 and package 5 to be handled, it becomes possible to determine whether each pallet 4 arranged in multiple rows is present on the near side (front side) or the far side (rear side). As a result, when detecting the packaging state of the pallets 4 and packages 5, the pallets 4 and packages 5 present on the far side are excluded, thereby improving the detection accuracy of the packaging state of the pallets 4 and packages 5.

[0115] The present invention is not limited to the above embodiment. For example, in the above embodiment, the trajectory range of the pallet 4 and the luggage 5 in the luggage set S to be handled, when virtually moved upward in the image, is set as the luggage interference determination region R in the image data D. However, the present invention is not particularly limited to such an embodiment. For example, the trajectory range of the pallet 4 alone in the luggage set S to be handled, when virtually moved upward in the image, may be set as the luggage interference determination region R.

[0116] In addition, in the above embodiment, the front areas of the pallet 4 and luggage 5 in the image data D are detected using instance segmentation, but this method is not particularly limited, and for example, the front areas of the pallet 4 and luggage 5 in the image data D may be detected using bounding boxes such as object detection.

[0117] Furthermore, in the above embodiment, the front areas of the pallet 4 and the luggage 5 are detected in the image data D, and the packing state of the pallet 4 and the luggage 5 is detected based on the front areas of the pallet 4 and the luggage 5, but this is not particularly limited to such an embodiment. For example, an area including the front, side, bottom, etc. of the pallet 4 and the luggage 5 may be detected in the image data D, and the packing state of the pallet 4 and the luggage 5 may be detected based on the areas of the pallet 4 and the luggage 5.

[0118] Furthermore, in the above embodiment, the camera 11 and the laser sensor 12 are mounted on the forklift 2, but the present invention is not limited to this particular form, and the camera 11 and the laser sensor 12 may be installed on-site.

[0119] Furthermore, in the above embodiment, the laser sensor 12 measures the distance to the pallet 4 and the luggage 5, thereby acquiring point cloud data of the pallet 4 and the luggage 5. However, this is not limited to a particular form, and instead of the laser sensor 12, a sensor capable of measuring the distance to an object and acquiring point cloud data, such as an RGBD camera, may be used.

[0120] Furthermore, in the above embodiment, the virtual movement amount m of the pallets 4 and packages 5 present in the package set S to be handled is determined according to the distance from the forklift 2 to the pallets 4 and packages 5, and the trajectory range when the pallets 4 and packages 5 present in the package set S to be handled are virtually moved upward in the image by the virtual movement amount m is set as the package interference determination area R, but this is not particularly limited to such an embodiment. For example, regardless of the distance from the forklift 2 to the pallets 4 and packages 5, the trajectory range when the pallets 4 and packages 5 present in the package set S to be handled are virtually moved upward in the image by a preset amount may be set as the package interference determination area R.

[0121] Furthermore, in the above embodiment, when the forklift 2 handles the pallet 4 loaded on the loading platform 3a of the truck 3, the state of the packaging of the pallet 4 and the cargo 5 is detected. However, the present invention is not limited to the loading platform 3a of the truck 3, and can also be applied to cases where the pallet 4 is loaded on the floor of a factory, for example. [Explanation of symbols]

[0122] 2...forklift, 4...pallet, 4a...front (front), 5...luggage, 5a...front (front), 10...packing condition detection system, 11...camera (image acquisition unit), 12...laser sensor (point cloud acquisition unit), 41...area detection unit, 42...point cloud processing unit (distance calculation unit), 43...image-point cloud matching unit (distance calculation unit), 45a...luggage set setting unit, 45b...virtual movement amount determination unit, 45c...interference area setting unit, 45d...interference determination unit, D...image data, R...luggage interference determination area, S...luggage set.

Claims

1. A package state detection system that detects the package state of at least one tier of pallets arranged in multiple rows in a horizontal direction when a forklift is used to handle the pallets, and the package state of goods placed on the pallets, an image acquisition unit that captures images of the pallet and the luggage and acquires image data of the pallet and the luggage; an area detection unit that detects areas of the pallet and the luggage in the image data acquired by the image acquisition unit; a luggage set setting unit that sets a plurality of rows of luggage sets representing a range in which the pallet and the luggage are stacked in the vertical direction of the image in the area of ​​the pallet and the luggage in the image data detected by the area detection unit; an interference area setting unit that sets, as a luggage interference determination area, a trajectory range when a pallet present in a luggage set to be handled among the plurality of rows of luggage sets set by the luggage set setting unit is virtually moved upward in the image; A cargo state detection system comprising an interference determination unit that determines whether cargo in a cargo set adjacent to the cargo set to be handled is interfering with a pallet in the cargo set to be handled by determining whether the cargo interference determination area set by the interference area setting unit includes cargo in a cargo set adjacent to the cargo set to be handled.

2. the interference area setting unit sets, as the luggage interference determination area, a trajectory range when a pallet and luggage present in the luggage set to be loaded are virtually moved upward in the image; The interference determination unit determines whether a pallet or piece of luggage in the luggage set to be handled is interfering with a piece of luggage in the luggage set to be handled by determining whether the luggage interference determination area includes luggage in a luggage set adjacent to the luggage set to be handled.

3. The package state detection system according to claim 1, wherein the luggage interference determination area is the trajectory range when a pallet present in the luggage set to be loaded is virtually moved to a position above the pallet in the image.

4. a point cloud acquisition unit that measures distances to the pallet and the package and acquires point cloud data of the pallet and the package; a distance calculation unit that extracts a point cloud of the area of ​​the pallet and the luggage detected by the area detection unit based on the point cloud data acquired by the point cloud acquisition unit, and calculates a distance from the forklift to the pallet and the luggage based on the point cloud of the area of ​​the pallet and the luggage; a virtual movement amount determination unit that determines a virtual movement amount of a pallet present in the load set to be handled in accordance with the distance from the forklift to the pallet calculated by the distance calculation unit, 2. The packaging state detection system according to claim 1, wherein the interference area setting unit sets, as the luggage interference determination area, a trajectory range when a pallet present in the luggage set to be loaded is virtually moved upward on the image by the virtual movement amount determined by the virtual movement amount determination unit.

5. the area detection unit detects a front area of ​​the pallet and the package in the image data acquired by the image acquisition unit; 2. The packaging state detection system according to claim 1, wherein the luggage set setting unit sets the plurality of rows of luggage sets in a front area of ​​the pallet and the luggage in the image data detected by the area detection unit.

Images