Load posture state detection device

Abstract

To provide a load posture state detection device capable of detecting whether the load posture state of a pallet and a load is abnormal.SOLUTION: A load posture state detection device 10 comprises: a camera 11 that acquires load posture image data D of a pallet 4 and a load 5 by imaging the load postures of the pallet 4 and the load 5; a region extraction unit 15 that extracts regions of the pallet 4 and the load 5 in the load posture image data D acquired by the camera 11; and a protrusion determination unit 16c that determines whether the load 5 placed on the pallet 4 to be held by a forklift 2 or another pallet 4 protrudes from the pallet 4 to be held by a prescribed amount or more in a horizontal direction on the basis of the regions of the pallet 4 and the load 5 extracted by the region extraction unit 15.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a packaging state detection device. [Background technology]

[0002] For example, Patent Document 1 describes a technology in which an autonomous forklift equipped with sensors and cameras is used to unload goods loaded on a truck and load them onto an autonomous unmanned guided vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-62964 Summary of the Invention [Problem to be solved by the invention]

[0004] In actual logistics situations, for example, if the load condition (packing state) of the pallet and the cargo becomes abnormal due to the cargo protruding significantly from the pallet due to vibration during transportation, it may be impossible to unload the cargo using normal cargo handling control. For this reason, it is necessary to detect whether the packing state of the pallet and the cargo is abnormal, and if the packing state is abnormal, to take appropriate measures according to the packing state.

[0005] An object of the present invention is to provide a packaging state detection device capable of detecting whether the packaging state of a pallet and luggage is abnormal. [Means for solving the problem]

[0006] (1) One aspect of the present invention is a packaging state detection device that detects the packaging state of a pallet and cargo when a forklift is used to load at least one level of pallets on which cargo is placed, and includes a detection unit that detects the packaging state of the pallet and cargo to obtain packaging state detection data for the pallet and cargo, an area extraction unit that extracts an area of ​​the pallet and cargo in the packaging state detection data obtained by the detection unit, and an overhang determination unit that determines whether a cargo or another pallet placed on a pallet to be held by the forklift protrudes laterally from the pallet to be held by the forklift by more than a specified amount based on the area of ​​the pallet and cargo extracted by the area extraction unit.

[0007] In such a packaging state detection device, the areas of the pallet and the cargo are extracted from the packaging state detection data acquired by detecting the packaging state of the pallet and the cargo. Then, based on the areas of the pallet and the cargo in the packaging state detection data, it is determined whether the cargo or another pallet placed on the pallet to be held by the forklift protrudes laterally from the pallet to be held by the forklift by a specified amount or more. When it is determined that the cargo or another pallet placed on the pallet to be held protrudes laterally from the pallet to be held by a specified amount or more, it is detected that the packaging state of the pallet and the cargo is abnormal. In this way, it is possible to detect whether the packaging state of the pallet and the cargo is abnormal.

[0008] (2) In the above (1), the area extraction unit may extract a front area of ​​the pallet and the package from the package style detection data.

[0009] In this configuration, the influence of the shadows of the side areas of the pallet and the luggage on the protrusion determination of the luggage and the pallet in the package style detection data is suppressed, and therefore it is possible to determine with high accuracy whether the luggage or another pallet placed on the pallet to be held protrudes laterally beyond the pallet to be held by a specified amount or more.

[0010] (3) In (1) or (2) above, the overhang determination unit may create a luggage set representing an area including a pallet and luggage that are adjacent to each other in the vertical direction in the package shape detection data, and determine whether the amount of overhang of the luggage or other pallet from the inner area of ​​two imaginary lines extending in the vertical direction from both left and right ends of the pallet to be held in the luggage set is equal to or greater than a predetermined threshold, thereby determining whether the luggage or other pallet overhangs the pallet to be held by more than a specified amount in the horizontal direction.

[0011] In this configuration, protrusion determination of luggage and pallets is performed for each luggage set that represents an area including a pallet and luggage that are vertically adjacent to each other in the package style detection data. A luggage set corresponds to a unit of loading and unloading performed collectively by a forklift. Therefore, protrusion determination of luggage and pallets can be efficiently performed for each loading and unloading unit. In addition, by setting an inner area between two imaginary lines extending vertically from both the left and right ends of the pallet to be held in the luggage set, it is possible to easily determine whether luggage or other pallets protrude laterally by more than a specified amount from the pallet to be held.

[0012] (4) In any of (1) to (3) above, the detection unit may capture an image of the packaging appearance of the pallet and the luggage, thereby acquiring image data of the packaging appearance of the pallet and the luggage as packaging appearance detection data, and the area extraction unit may extract the areas of the pallet and luggage in the packaging appearance image data on a pixel-by-pixel basis.

[0013] In such a configuration, the areas of the pallet and luggage are extracted in pixel units of the packaging style image data, so that the areas of the pallet and luggage can be extracted with high accuracy in the packaging style detection data.

[0014] (5) In (4) above, the area extraction unit may extract the areas of pallets and luggage in the packaging image data using learned data including packaging images of multiple types of pallets and luggage and pixel information of multiple types of pallets and luggage.

[0015] In such a configuration, by using learned data including the packaging image and pixel information, the areas of the pallet and luggage can be extracted with higher accuracy from the packaging image data.

[0016] (6) In (5) above, the trained data may be generated by transfer learning data obtained by pre-training a formula-driven database, in which pixel information of the pallet and the luggage is annotated to images of the pallet and the luggage.

[0017] In such a configuration, by using pre-learning data obtained by pre-learning a formula-driven database, it is possible to reduce the number of packaging images in which the packaging state of the pallet and luggage is captured, while extracting the areas of the pallet and luggage from the packaging image data with high accuracy. Effect of the Invention

[0018] According to the present invention, it is possible to detect whether the packaging state of the pallet and the packages is abnormal. [Brief description of the drawings]

[0019] [Figure 1] 1 is a block diagram showing the configuration of a cargo handling control device equipped with a package state detection device according to one embodiment of the present invention. [Diagram 2] FIG. 2 is a side view of a forklift on which the cargo handling control device shown in FIG. 1 is mounted. [Diagram 3] FIG. 2 is a plan view showing a state in which a pallet on which cargo is placed is loaded onto the bed of a truck. [Figure 4] 13 is a front view showing a state in which a pallet interferes with an adjacent pallet and a state in which a piece of luggage interferes with an adjacent piece of luggage or pallet. FIG. [Diagram 5] 1 is a front view showing a state in which cargo placed on a pallet protrudes from the pallet, and a state in which an upper pallet protrudes from a lower pallet. FIG. [Figure 6]2 is a block diagram showing the functions of a region extraction unit shown in FIG. 1. [Figure 7] FIG. 7 is a block diagram showing a function for creating the learned data shown in FIG. 6. [Figure 8] 4 is a flowchart showing a procedure of a pallet interference determination process executed by a pallet interference determination unit shown in FIG. 1. [Figure 9] 2 is a front view showing how the pallet interference determination unit shown in FIG. 1 determines whether or not there is pallet interference. FIG. [Figure 10] 4 is a flowchart showing a procedure of a baggage interference determination process executed by a baggage interference determination unit shown in FIG. 1. [Figure 11] 2 is a front view showing how the luggage interference determination unit shown in FIG. 1 determines whether or not there is luggage interference. FIG. [Figure 12] 4 is a flowchart showing a procedure of a protrusion determination process executed by a protrusion determination unit shown in FIG. 1. [Figure 13] 2 is a front view showing how the protrusion determination unit shown in FIG. 1 determines whether or not a package or a pallet is protruding. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same or equivalent elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0021] Fig. 1 is a block diagram showing the configuration of a cargo handling control device equipped with a package state detection device according to one embodiment of the present invention. In Fig. 1, the cargo handling control device 1 is mounted on a forklift 2 as shown in Fig. 2. The forklift 2 is, for example, a countertop forklift.

[0022] The forklift 2 has a body 31, a pair of left and right front wheels 32 which are drive wheels arranged at the front of the body 31, a pair of left and right rear wheels 33 which are steered wheels arranged at the rear of the body 31, a mast 34 attached to the front end of the body 31, a pair of left and right forks 36 which are attached to the mast 34 via lift brackets 35 so that they can be raised and lowered, a lift cylinder 37 which raises and lowers the forks 36 via the lift brackets 35, and a tilt cylinder 38 which tilts the mast 34.

[0023] The cargo handling control device 1 is a device that performs cargo handling control when cargo handling is automatically performed by a forklift 2. Here, as shown in Fig. 3, the cargo handling control device 1 controls the forklift 2 when holding a pallet 4 loaded on a loading platform 3a of a truck 3 with the forks 36 of the forklift 2, that is, when carrying out so-called cargo pick-up.

[0024] The pallet 4 is, for example, a flat pallet made of plastic or wood. The pallet 4 has a square or approximately square shape in a plan view. Cargo 5 is placed on the pallet 4. The pallet 4 is provided with two fork holes 6 into which forks 36 are inserted. The fork holes 6 extend from the front surface (front face) 4a of the pallet 4 toward the rear side.

[0025] A plurality of pallets 4 are loaded on the bed 3a of the truck 3 along the front-to-rear direction of the truck 3. The pallets 4 are arranged so that the front faces 4a face the side of the truck 3. Therefore, the forklift 2 picks up cargo from the side of the truck 3. At this time, the plurality of pallets 4 are arranged side by side in the horizontal direction (left-right direction) as viewed from the forklift 2. The forklift 2 loads and unloads one or more upper and lower tiers of pallets 4 on which cargo 5 is placed (see FIG. 5, etc.). For example, when the forklift 2 picks up cargo from two tiers of pallets 4, the forks 36 are inserted into the fork holes 6 of the lower tier of pallet 4.

[0026] Incidentally, as shown in Fig. 3, when the pallet 4 and the cargo 5 are in a normal packed state, the cargo can be picked up smoothly by the forklift 2. The packed state refers to the way the pallet 4 and the cargo 5 are placed.

[0027] However, as shown in Fig. 4, if the packaging state of the pallets 4 and the cargo 5 is abnormal, it may be impossible to load the cargo using the forklift 2. For example, as shown in Fig. 4(a), if laterally adjacent pallets 4 interfere with each other in the width direction of the truck 3 (the direction in which the pallets 4 are pulled out from the loading platform 3a to the side of the truck 3), the cargo may collapse when the cargo is loaded. Also, as shown in Fig. 4(b), if the cargo 5 placed on a pallet 4 interferes with the cargo 5 placed on the laterally adjacent pallet 4, or if the cargo 5 placed on a pallet 4 interferes with the laterally adjacent pallet 4, the cargo may collapse when the cargo is loaded.

[0028] 5, due to vibrations during transportation by truck 3, packages 5 may protrude significantly laterally from pallets 4, or an upper pallet 4 may protrude significantly laterally from a lower pallet 4. In this case, there is also a possibility that the packages may collapse when unloaded.

[0029] In order to solve such problems, the cargo handling control device 1 detects whether the packing condition of the pallet 4 and the luggage 5 is abnormal, and if the packing condition of the pallet 4 and the luggage 5 is abnormal, performs appropriate control according to the packing condition.

[0030] Therefore, the cargo handling control device 1 is equipped with a packaging state detection device 10 that detects the packaging state of the pallets 4 and the cargo 5. The packaging state detection device 10 is a device that detects whether the packaging state of the pallets 4 and the cargo 5 is normal or abnormal when cargo 5 is placed on a plurality of pallets 4 arranged side by side in the horizontal direction facing the forklift 2 and the forklift 2 handles at least one level of the pallets 4.

[0031] The cargo handling control device 1 including the package state detection device 10 comprises a camera 11, a drive unit 12, an alarm unit 13, and a controller 14.

[0032] The camera 11 is an imaging unit that captures an image in front of the forklift 2. The camera 11 captures an image of the packaging appearance of the pallet 4 and the luggage 5 to obtain packaging appearance image data of the pallet 4 and the luggage 5. The camera 11 constitutes a detection unit that detects the packaging appearance of the pallet 4 and the luggage 5 to obtain packaging appearance image data that is packaging appearance detection data of the pallet 4 and the luggage 5.

[0033] The drive unit 12 has the above-mentioned lift cylinder 37 and tilt cylinder 38, a travel motor (not shown) that rotates the front wheels 32 of the forklift 2, and a steering motor (not shown) that steers the rear wheels 33 of the forklift 2. The alarm unit 13 issues an alarm, for example, by generating an alarm sound.

[0034] The controller 14 is composed of a CPU, RAM, ROM, an input / output interface, etc. The controller 14 has an area extraction unit 15, a packaging style abnormality determination unit 16, and a cargo handling control unit 17. The camera 11, the area extraction unit 15, and the packaging style abnormality determination unit 16 constitute the packaging style state detection device 10 of this embodiment.

[0035] The area extraction unit 15 extracts the areas of the pallet 4 and the luggage 5 from the packing image data acquired by the camera 11. The area extraction unit 15 extracts the areas of the pallet 4 and the luggage 5 from the packing image data in pixel units of the packing image data.

[0036] The area extraction unit 15 extracts the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 (see FIG. 3) in the packing image data. The front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 are surfaces that face the forklift 2 that picks up the luggage. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 correspond to the front areas of the pallet 4 and the luggage 5.

[0037] 6, the area extraction unit 15 is configured with a function having a package shape recognition model 20 and learned data 21. The area extraction unit 15 extracts the area of ​​the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the package shape image data acquired by the camera 11 by having the package shape recognition model 20 read the learned data 21.

[0038] The package recognition model 20 is an instance segmentation model constructed using deep learning. Instance segmentation is a technology that estimates the position of an individual object on a pixel-by-pixel basis, and learns by labeling the target object area in detail to estimate the object position. The package recognition model 20 is constructed using data in which the front areas of the pallet 4 and the luggage 5 are annotated (added) to images in which the pallet 4 and the luggage 5 are photographed.

[0039] The packaging recognition model 20 has a feature extraction unit 22 that uses learned data 21 to extract features of the packaging image captured by the camera 11, and an area recognition unit 23 that uses the learned data 21 and the features of the packaging image to recognize the area of ​​the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data.

[0040] The learned data 21 is composed of two types of data: a packaging image of a pallet 4 and luggage 5, and a data file including information on the pallet 4 and luggage 5 in the packaging image. The information on the pallet 4 and luggage 5 in the packaging image includes pixel numbers of the packaging image. The areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image are represented by pixel numbers of the packaging image.

[0041] For example, if there are two types of luggage 5, namely, cardboard boxes and parts boxes, and there are two types of pallets 4, namely, plastic pallets and post pallets, learned data 21 is prepared in which labels are assigned to the areas of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 for a total of four types of packaging images in which the pallets 4 and luggage 5 are photographed.

[0042] The trained data 21 is generated by pre-training a formula-driven database, as shown in Fig. 7. The formula-driven database is an image generated from a formula, and is data that does not require labeling of the image. For example, a data set (FractalDB) composed of fractal geometric images is used as the formula-driven database.

[0043] 7(a), first, pre-learning of a formula-driven database 25 is performed by an image classification model 24 to generate pre-learned data 26. Formula-driven supervised learning is used for the pre-learning of the formula-driven database 25. The image classification model 24 has a feature extraction unit 27 that extracts features of the formula-driven database 25, and an image classification unit 28 that solves an image classification problem of the features of the formula-driven database 25.

[0044] 7(b), transfer learning is performed using pre-trained data 26 to construct the above-mentioned packaging shape recognition model 20, thereby generating trained data 21. At this time, pre-trained data 26, which has been pre-trained with formula-driven database 25, is read into feature extraction unit 22, and data in which the areas of front surface 4a of pallet 4 and front surface 5a of luggage 5 are added to packaging shape image 29 in which pallet 4 and luggage 5 are photographed is subjected to transfer learning, thereby acquiring trained data 21.

[0045] Returning to FIG. 1, the packaging style abnormality determination unit 16 has a pallet interference determination unit 16a, a baggage interference determination unit 16b, and an overhang determination unit 16c.

[0046] The pallet interference determination unit 16a determines whether the pallet 4 to be detected interferes with a laterally adjacent pallet 4 based on the area of ​​the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data extracted by the area extraction unit 15.

[0047] The luggage interference determination unit 16b determines whether the luggage 5 to be detected will interfere with adjacent luggage 5 or pallet 4 laterally, based on the area of ​​the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data extracted by the area extraction unit 15.

[0048] The protrusion determination unit 16c determines whether or not a cargo 5 or another pallet 4 placed on the pallet 4 to be held by the forks 36 of the forklift 2 protrudes laterally by a specified amount or more from the pallet 4 to be held, based on the areas of the front surface 4a of the pallet 4 and the front surface 5a of the cargo 5 extracted by the area extraction unit 15. The pallet 4 to be held is the pallet 4 into which the forks 36 are inserted.

[0049] 8 is a flowchart showing the procedure of the pallet interference determination process executed by the pallet interference determination unit 16a. This process is executed when an instruction to start cargo handling control is issued by, for example, a manual operation switch or the like.

[0050] In Figure 8, the pallet interference judgment unit 16a first acquires area information (front area information of the pallet 4 and the luggage 5) of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data extracted by the area extraction unit 15 (step S101).

[0051] Next, the pallet interference determination unit 16a creates multiple luggage sets S in the front area of ​​the pallet 4 and luggage 5 in the packaging image data (step S102). Fig. 9(a) shows an example of the front area of ​​the pallet 4 and luggage 5 in the packaging image data D extracted by the area extraction unit 15.

[0052] As shown in FIG. 9(b), the luggage set S represents a rectangular area including pallets 4 and luggage 5 adjacent to each other in the vertical direction in the packaging image data D. In other words, the luggage set S represents an area including only pallets 4 and luggage 5 adjacent to each other in the vertical direction in the packaging image data D, and does not include pallets 4 and luggage 5 adjacent to each other in the left-right direction (horizontal direction). Multiple luggage sets S are created along the horizontal direction in the packaging image data D. In the packaging image data D shown in FIG. 9(b), two luggage sets Sa, Sb adjacent to each other in the horizontal direction are created. The luggage sets Sa, Sb include two pallets 4 and luggage 5 arranged vertically.

[0053] Next, the pallet interference determination unit 16a determines whether the pallet 4 of the luggage set S to be detected among the multiple luggage sets S will come into contact with the pallet 4 of the luggage set S to be loaded that is adjacent in the horizontal direction (step S103). The pallet interference determination unit 16a determines whether the pallet 4 of the luggage set S to be detected will come into contact with the pallet 4 of the luggage set S to be loaded by determining whether the pixels of the pallet 4 of the luggage set S to be detected come into contact with the pixels of the pallet 4 of the luggage set S to be loaded in the packaging shape image data D. The luggage set S to be loaded is the luggage set S including the pallet 4 and luggage 5 to be loaded next.

[0054] In Fig. 9(b), the baggage set Sa is the baggage set S to be handled, and the baggage set Sb is the baggage set S to be detected. Therefore, it is determined whether the pallet 4 of the baggage set Sb to be detected comes into contact with the pallet 4 of the baggage set Sa to be handled.

[0055] When the pallet interference determination unit 16a determines that the pallet 4 of the luggage set S to be detected will come into contact with the pallet 4 of the luggage set S to be loaded that is adjacent in the laterally direction, it calculates the yaw angle, which is the attitude angle of the pallet 4 of the luggage set S to be detected (hereinafter referred to as the contacting pallet 4A) based on the area information of the pallet 4 in the packaging image data D and the previously known shape information of the pallet 4 (step S104).

[0056] The yaw angle of the contacting pallet 4A is 0 degrees when the contacting pallet 4A faces forward relative to the forklift 2. In FIG. 9(c), the pallet 4 of the luggage set Sb to be detected is the contacting pallet 4A that is in contact with the pallet 4 of the luggage set Sa to be handled.

[0057] Then, the pallet interference determination unit 16a determines whether the yaw angle of the contacted pallet 4A is equal to or greater than a predetermined threshold value (step S105). The threshold value is set to an angle that does not affect the loading operation of the pallet 4 of the luggage set S to be loaded and the luggage 5 even if the pallet 4 of the luggage set S to be detected and the luggage set S to be loaded are in contact with each other, for example, an angle that does not cause the luggage 5 to collapse.

[0058] When the pallet interference determination unit 16a determines that the yaw angle of the contacting pallet 4A is equal to or greater than the threshold value, it determines that the pallet 4 of the luggage set S to be detected is interfering with the pallet 4 of the luggage set S to be handled (step S106). Then, the pallet interference determination unit 16a outputs an abnormality control signal to the loading control unit 17 (step S107).

[0059] When the pallet 4 of the luggage set S to be detected is determined in step S103 to be not in contact with the pallet 4 of the laterally adjacent luggage set S to be loaded, or when the pallet interference determination unit 16a determines in step S105 that the yaw angle of the contacting pallet 4A is not equal to or greater than the threshold, it determines that the pallet 4 of the luggage set S to be detected is not interfering with the pallet 4 of the luggage set S to be loaded (step S108). Then, the pallet interference determination unit 16a outputs a normal control signal to the loading control unit 17 (step S109).

[0060] 10 is a flowchart showing the procedure of the luggage interference determination process executed by the luggage interference determination unit 16b. This process is also executed when an instruction to start cargo handling control is issued, similar to the pallet interference determination unit 16a.

[0061] In FIG. 10, the luggage interference determination unit 16b first acquires area information (front area information of the pallet 4 and luggage 5) of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data extracted by the area extraction unit 15 (step S111).

[0062] Next, the luggage interference determination unit 16b creates multiple luggage sets S in the front area of ​​the pallet 4 and luggage 5 in the packaging image data (step S112). Fig. 11(a) shows another example of the front area of ​​the pallet 4 and luggage 5 in the packaging image data D extracted by the area extraction unit 15.

[0063] As shown in Fig. 11(b), similar to step S102 above, the package set S represents a rectangular area including pallets 4 and packages 5 that are adjacent to each other in the vertical direction in the package image data D. Two packages sets Sa and Sb that are adjacent to each other in the horizontal direction are created in the package image data D shown in Fig. 11(b). The packages sets Sa and Sb include two tiers of pallets 4 and packages 5 arranged vertically.

[0064] Next, the luggage interference determination unit 16b sets imaginary lines L1, which are two vertical lines extending in the up-down direction from both the left and right ends of the pallet 4 of the luggage set S to be handled, in the package image data D (step S113). At this time, if the positions of both the left and right ends of the two upper and lower pallets 4 are misaligned, two imaginary lines L1 are set by extending in the up-down direction the ends located on the outer sides in the left-right direction out of the left and right ends of the two upper and lower pallets 4. In the package image data D shown in Fig. 11(c), two imaginary lines L1 are set by extending in the up-down direction from both the left and right ends of the pallet 4 of the luggage set Sa to be handled.

[0065] Next, the luggage interference determination unit 16b sets a luggage interference determination area R of the loading target in the package image data D (step S114). As shown in Fig. 11(c) , the luggage interference determination area R of the loading target is an area surrounded by two virtual lines L1, a horizontal line L2 at the upper end of the package image data D, and a horizontal line L3 which is the bottom surface of the lower pallet 4 in the package image data D. The luggage interference determination area R of the loading target corresponds to the area inside the two virtual lines L1.

[0066] Next, the luggage interference determination unit 16b judges whether the luggage set S to be detected overlaps with the luggage interference determination area R of the loading target (step S115). When the luggage interference determination unit 16b judges that the luggage set S to be detected overlaps with the luggage interference determination area R of the loading target, it determines that the luggage 5 of the luggage set S to be detected is interfering with the luggage 5 or pallet 4 of the luggage set S to be loaded (step S116). Then, the luggage interference determination unit 16b outputs an abnormality control signal to the loading control unit 17 (step S117).

[0067] In the package image data D shown in Fig. 11(c), a part of the detected package set Sb overlaps with the package interference determination area R of the loaded package. Therefore, as shown in Fig. 11(d), it is determined that the package 5 of the detected package set Sb is interfering with the package 5 or pallet 4 of the loaded package set Sa (see W in the figure).

[0068] When the luggage interference determination unit 16b determines that the luggage set S to be detected does not overlap with the luggage interference determination region R of the loading target, it determines that the luggage 5 of the luggage set S to be detected does not interfere with the luggage 5 and pallet 4 of the luggage set S to be loaded (step S118). Then, the luggage interference determination unit 16b outputs a normal control signal to the loading control unit 17 (step S119).

[0069] 12 is a flowchart showing the procedure of the overflow determination process executed by the overflow determination unit 16c. This process is also executed when an instruction to start cargo handling control is issued, similar to the pallet interference determination unit 16a.

[0070] In FIG. 12, the overflow determination unit 16c first acquires area information (front area information of the pallet 4 and the luggage 5) of the front surface 4a of the pallet 4 and the front surface 5a of the luggage 5 in the packaging image data extracted by the area extraction unit 15 (step S121).

[0071] Next, the overflow determination unit 16c creates multiple package sets S in the front area of ​​the pallet 4 and packages 5 in the package image data (step S122). Fig. 13(a) shows yet another example of the front area of ​​the pallet 4 and packages 5 in the package image data D extracted by the area extraction unit 15.

[0072] As shown in Fig. 13(b), similar to step S102 above, the luggage set S represents a rectangular area including pallets 4 and luggage 5 that are adjacent to each other in the vertical direction in the packing image data D. Two pairs of luggage sets Sa and Sb that are adjacent in the horizontal direction are created in the packing image data D shown in Fig. 13(b). The luggage set Sa includes one layer of pallets 4 and luggage 5. The luggage set Sb includes two layers of pallets 4 and luggage 5, one above the other.

[0073] Next, the overhang determination unit 16c sets imaginary lines L1, which are two vertical lines extending in the up-down direction from both the left and right ends of the pallet 4 (hereinafter referred to as the holding pallet 4H) to be held by each package set S, in the package image data D (step S123). The holding pallet 4H is the pallet 4 held by the forks 36 of the forklift 2 as described above. At this time, if there are two or more tiers of pallets 4, the lowest pallet 4 becomes the holding pallet 4H.

[0074] In the packaging image data D shown in Figure 13 (c), two imaginary lines L1 are set extending in the vertical direction from both the left and right ends of the pallet 4 of the luggage set Sa, and two imaginary lines L1 are set extending in the vertical direction from both the left and right ends of the lower pallet 4 of the luggage set Sb.

[0075] Next, the overflow determination unit 16c sets an overflow determination area Q for each package set S in the package image data D (step S124). As shown in Fig. 13(c), the overflow determination area Q is an area surrounded by two imaginary lines L1, a horizontal line L2 at the top end of the package image data D, and a horizontal line L3 which is the bottom surface of the lower pallet 4 in the package image data D. The overflow determination area Q corresponds to the area inside the two imaginary lines L1.

[0076] Next, the overhang determination unit 16c determines whether the overhang amount f of the luggage 5 or pallet 4 from the overhang determination area Q is equal to or greater than a predetermined threshold value for each luggage set S (step S125). The threshold value is set to a value that does not affect the loading operation of the pallet 4 and luggage 5 even if the luggage 5 or pallet 4 overhangs the overhang determination area Q in the lateral direction, for example, a value that does not cause the luggage 5 to collapse.

[0077] When the protrusion determination unit 16c determines that the protrusion amount f of the luggage 5 or pallet 4 with respect to the protrusion determination area Q is equal to or greater than the threshold value, it determines that the luggage 5 or another pallet 4 protrudes laterally from the holding pallet 4H by a specified amount or more (step S126).Then, the protrusion determination unit 16c outputs an abnormality control signal to the cargo handling control unit 17 (step S127).

[0078] In the package image data D shown in Figure 13(c), the protrusion amount f of the luggage 5 from the protrusion judgment area Q in the luggage set Sa is equal to or greater than the threshold value, and the protrusion amount f of the pallet 4 from the protrusion judgment area Q in the luggage set Sb is equal to or greater than the threshold value. Therefore, as shown in Figure 13(d), it is determined that the luggage 5 protrudes laterally from the holding pallet 4H in the luggage set Sa by a specified amount or more, and also that the upper pallet 4 protrudes laterally from the lower pallet 4, which is the holding pallet 4H, in the luggage set Sb by a specified amount or more (see W in the figure).

[0079] When the overhang determination unit 16c determines that the overhang amount f of the luggage 5 and pallet 4 with respect to the overhang determination area Q is not equal to or greater than the threshold value, it determines that the luggage 5 and other pallets 4 do not overhang the holding pallet 4H laterally by a specified amount or more (step S128).Then, the overhang determination unit 16c outputs a normal control signal to the cargo handling control unit 17 (step S129).

[0080] Returning to Fig. 1, the cargo handling control unit 17 controls the drive unit 12 and the alarm unit 13 according to the result of the judgment by the packaging shape abnormality judgment unit 16. When the cargo handling control unit 17 receives normal control signals from all of the pallet interference judgment unit 16a, the luggage interference judgment unit 16b, and the protrusion judgment unit 16c, it controls the drive unit 12 to pick up the cargo of the cargo handling target. When the cargo handling control unit 17 receives an abnormal control signal from any of the pallet interference judgment unit 16a, the luggage interference judgment unit 16b, and the protrusion judgment unit 16c, it controls the alarm unit 13 to issue an alarm.

[0081] In the above, when cargo is picked up by the forklift 2, the forklift 2 travels toward the truck 3. When the forklift 2 reaches the side of the truck 3, the forklift 2 stops once. In this state, the camera 11 captures an image of the packaged state in which the cargo 5 is placed on the multiple pallets 4, and packaged state image data D of the pallets 4 and the cargo 5 is obtained.

[0082] Then, the front areas of the pallet 4 and luggage 5 in the package image data D are extracted. Then, based on the front areas of the pallet 4 and luggage 5 in the package image data D, it is determined whether the pallet 4 of the luggage set S to be detected interferes with the pallet 4 of the luggage set S to be handled that is adjacent in the horizontal direction. Also, based on the front areas of the pallet 4 and luggage 5 in the package image data D, it is determined whether the luggage 5 of the luggage set S to be detected interferes with the luggage 5 or pallet 4 of the luggage set S to be handled. Furthermore, based on the front areas of the pallet 4 and luggage 5 in the package image data D, it is determined whether the luggage 5 placed on the holding pallet 4H or another pallet 4 protrudes laterally from the holding pallet 4H by more than a specified amount.

[0083] When it is determined that the pallet 4 of the luggage set S to be detected does not interfere with the pallet 4 of the luggage set S to be loaded, when it is determined that the luggage 5 of the luggage set S to be detected does not interfere with the luggage 5 and pallet 4 of the luggage set S to be loaded, and when it is determined that the luggage 5 and other pallets 4 placed on the holding pallet 4H do not protrude laterally from the holding pallet 4H by more than a specified amount, the drive unit 12 is controlled to unload the luggage set S to be loaded.

[0084] When it is determined that the pallet 4 of the luggage set S to be detected will interfere with the pallet 4 of the luggage set S to be loaded, an alarm is issued by the alarm unit 13 and the loading of the luggage set S to be detected is halted. When it is determined that luggage 5 of the luggage set S to be detected will interfere with luggage 5 or pallet 4 of the luggage set S to be loaded, an alarm is issued by the alarm unit 13 and the loading of the luggage set S to be detected is halted. When it is determined that luggage 5 or other pallets 4 placed on the holding pallet 4H protrude laterally from the holding pallet 4H by more than a specified amount, an alarm is issued by the alarm unit 13 and the loading of the luggage set S to be detected is halted.

[0085] As described above, in this embodiment, the areas of the pallet 4 and the luggage 5 are extracted from the packaging style image data D acquired by detecting the packaging style of the pallet 4 and the luggage 5. Then, based on the areas of the pallet 4 and the luggage 5 in the packaging style image data D, it is determined whether the luggage 5 or other pallets 4 placed on the pallet 4 (holding pallet 4H) to be held by the forklift 2 protrudes laterally from the holding pallet 4H by a specified amount or more. When it is determined that the luggage 5 or other pallets 4 placed on the holding pallet 4H protrudes laterally from the holding pallet 4H by a specified amount or more, it is detected that the packaging style of the pallet 4 and the luggage 5 is abnormal. In this way, it is possible to detect whether the packaging style of the pallet 4 and the luggage 5 is abnormal. As a result, it is possible to realize appropriate loading and unloading control according to the packaging style of the pallet 4 and the luggage 5.

[0086] Furthermore, in this embodiment, the front areas of the pallet 4 and luggage 5 are extracted from the packaging image data D. This prevents shadows or the like of the side areas of the pallet 4 and luggage 5 from affecting the protrusion determination of the luggage 5 and pallet 4 in the packaging image data D. This makes it possible to determine with high accuracy whether the luggage 5 or other pallets 4 placed on the holding pallet 4H protrudes laterally from the holding pallet 4H by more than a specified amount.

[0087] In this embodiment, a luggage set S is created that represents an area including the pallet 4 and luggage 5 that are in contact with each other in the vertical direction in the package image data D, and it is determined whether the luggage 5 or other pallets 4 protrude from the inner area of ​​the two virtual lines L1 extending in the vertical direction from both the left and right ends of the holding pallet 4H in the luggage set S by a threshold value or more. Therefore, the protrusion determination of the luggage 5 and the pallet 4 is performed for each luggage set S. The luggage set S corresponds to a unit of loading and unloading performed collectively by the forklift 2. Therefore, the protrusion determination of the luggage 5 and the pallet 4 can be efficiently performed for each loading and unloading unit. In addition, by setting the inner area of ​​the two virtual lines L1 extending in the vertical direction from both the left and right ends of the holding pallet 4H in the luggage set S, it is possible to easily determine whether the luggage 5 or other pallets 4 protrude laterally by a specified amount or more from the holding pallet 4H.

[0088] Furthermore, in this embodiment, packaging image data D of the pallet 4 and luggage 5 is acquired by capturing an image of the packaging of the pallet 4 and luggage 5, and the areas of the pallet 4 and luggage 5 in the packaging image data D are extracted in pixel units of the packaging image data D. By extracting the areas of the pallet 4 and luggage 5 in this manner in pixel units of the packaging image data D, the areas of the pallet 4 and luggage 5 can be extracted with high accuracy in the packaging image data D.

[0089] Furthermore, in this embodiment, the regions of the pallet 4 and the luggage 5 are extracted from the packaging image data D using the learned data 21 including packaging images of multiple types of pallets 4 and luggage 5 and pixel information of multiple types of pallets 4 and luggage 5. By using the learned data 21 including the packaging images and pixel information in this manner, the regions of the pallet 4 and luggage 5 can be extracted from the packaging image data D with higher accuracy.

[0090] Furthermore, in this embodiment, the learned data 21 is generated by transfer learning data in which pixel information of the pallet 4 and the luggage 5 is annotated to the packaging appearance images of the pallet 4 and the luggage 5, using pre-learned data 26 obtained by pre-learning the formula-driven database. By using the pre-learned data 26 obtained by pre-learning the formula-driven database in this manner, it is possible to extract the areas of the pallet 4 and the luggage 5 in the packaging appearance image data D with high accuracy while reducing the number of packaging appearance images in which the packaging appearances of the pallet 4 and the luggage 5 are captured.

[0091] In addition, in this embodiment, by constructing a package recognition model 20 using an instance segmentation model using deep learning, the areas of the pallet 4 and luggage 5 can be extracted with even greater accuracy in the package image data D.

[0092] The present invention is not limited to the above embodiment. For example, in the above embodiment, the front area of ​​the pallet 4 and the luggage 5 is extracted from the package image data D, and it is determined whether the luggage 5 or other pallets 4 placed on the holding pallet 4H protrudes laterally from the holding pallet 4H by a specified amount or more based on the front area of ​​the pallet 4 and the luggage 5, but the present invention is not limited to such a form. For example, it is also possible to extract an area including the front, side, and bottom of the pallet 4 and the luggage 5 from the package image data D, and determine whether the luggage 5 or other pallets 4 placed on the holding pallet 4H protrudes laterally from the holding pallet 4H by a specified amount or more based on the area of ​​the pallet 4 and the luggage 5.

[0093] In the above embodiment, it is determined whether the amount of protrusion of the luggage 5 or another pallet 4 from the protrusion determination area Q surrounded by two virtual lines L1 extending in the vertical direction from both the left and right ends of the holding pallet 4H, the horizontal line L2 at the top of the packaging image data D, and the horizontal line L3 which is the bottom surface of the holding pallet 4H is equal to or greater than a threshold value, but the present invention is not limited to such an embodiment. For example, it may be determined whether the amount of protrusion of the luggage 5 or another pallet 4 from the inner area of ​​the two virtual lines L1 extending in the vertical direction from both the left and right ends of the holding pallet 4H is equal to or greater than a threshold value without using the horizontal line L2 at the top of the packaging image data D and the horizontal line L3 which is the bottom surface of the holding pallet 4H.

[0094] In addition, in the above embodiment, the trained data 21 is created using pre-trained data 26 obtained by pre-training a formula-driven database, but the trained data 21 may also be created by increasing the number of packaging images of the pallets 4 and luggage 5 without using the pre-trained data 26.

[0095] In addition, in the above embodiment, the camera 11 captures an image of the packed state of the luggage 5 placed on the pallet 4, thereby acquiring the packing state image data D of the pallet 4 and the luggage 5. However, this is not limited to a particular form, and the packing state detection data of the pallet 4 and the luggage 5 may be acquired by detecting the packed state of the luggage 5 placed on the pallet 4 using a laser sensor or the like.

[0096] In addition, in the above embodiment, the packaging state of the pallet 4 and the cargo 5 is detected when the forklift 2 loads the pallet 4 on the bed 3a of the truck 3. However, the present invention is not limited to the bed 3a of the truck 3, and can also be applied to cases where loading and unloading is performed on the pallet 4 loaded on the floor of a factory, for example. [Explanation of symbols]

[0097] 2...forklift, 4...pallet, 4H...holding pallet (pallet to be held), 4a...front (front), 5...luggage, 5a...front (front), 10...packing state detection device, 11...camera (detection unit), 15...area extraction unit, 16a...protrusion determination unit, 21...trained data, 25...formula-driven database, 26...pre-trained data, 29...packing state image, D...packing state image data (packing state detection data), S...luggage set, Sa, Sb...luggage set, L1...virtual line, Q...protrusion determination area (inner area).

Claims

1. A packaging state detection device that detects a packaging state of at least one pallet on which cargo is placed by a forklift, the device comprising: a detection unit that detects the packaging style of the pallet and the package to obtain packaging style detection data of the pallet and the package; an area extraction unit that extracts an area of ​​the pallet and the package from the packing style detection data acquired by the detection unit; and an overhang determination unit that determines whether a cargo or another pallet placed on the pallet to be held by the forklift protrudes laterally from the pallet to be held by the forklift by more than a specified amount, based on the areas of the pallet and the cargo extracted by the area extraction unit.

2. 2. The packaging state detection device according to claim 1, wherein the area extraction unit extracts a front area of ​​the pallet and the package from the packaging state detection data.

3. The packaging state detection device of claim 1, wherein the overhang determination unit creates a luggage set representing an area including the pallet and the luggage that are adjacent to each other in the vertical direction in the packaging state detection data, and determines whether the amount of overhang of the luggage or the other pallet in the luggage set from the inner area of ​​two imaginary lines extending in the vertical direction from both left and right ends of the pallet to be held is equal to or greater than a predetermined threshold, thereby determining whether the luggage or the other pallet overhangs laterally from the pallet to be held by more than the specified amount.

4. The detection unit captures images of the pallet and the package of the luggage to obtain package style image data of the pallet and the package as the package style detection data, 2. The packaging state detection device according to claim 1, wherein the area extraction unit extracts the areas of the pallet and the packages in the packaging image data in pixel units of the packaging image data.

5. The packaging state detection device according to claim 4, wherein the area extraction unit extracts the areas of the pallet and the luggage in the packaging image data using learned data including packaging images of a plurality of types of pallets and luggage and pixel information of the plurality of types of pallets and luggage.

6. The packaging state detection device according to claim 5, wherein the learned data is generated by transfer learning data obtained by pre-training a formula-driven database, the data being annotated with pixel information of the pallet and the luggage to images of the packaging state of the pallet and the luggage.

Images