Pallet detection device and pallet detection method

Abstract

To provide a pallet detection device capable of accurately detecting a position and a posture of a pallet even when baggage is packed by a package, and a pallet detection method.SOLUTION: A pallet detection device 12 comprises: an acquisition unit (laser sensor 14) which detects reflected light of inspection light emitted toward a pallet and acquires a point group of data corresponding to detection results; a processing unit 17 which calculates a threshold value based on an arrangement state of the point group and an intensity of reflected light at each point and excludes data deviated from the threshold value from the point group; and a detection unit 18 which detects a position and a posture of the pallet on the basis of the point group after processing by the processing unit 17.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a pallet detection device and a pallet detection method. [Background technology]

[0002] A pallet detection device is known that acquires a point cloud of a pallet to be handled from the detection results of a sensor mounted on a forklift, and identifies the position and orientation of the pallet based on the acquired point cloud (see, for example, Patent Document 1). The pallet detection device described in Patent Document 1 detects an approximation line of the point cloud based on a first point cloud, which is a point cloud corresponding to a first detection by the sensor, and calculates the angle of incidence of light from the sensor to the target (pallet) in the second detection based on the approximation line and a second point cloud, which is a point cloud corresponding to a second detection by the sensor. Based on the angle of incidence, a point cloud for identifying the position and orientation of the target is selected from the second point cloud. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-120729 Summary of the Invention [Problem to be solved by the invention]

[0004] In some cases, cargo supported on a pallet is wrapped in a package to prevent it from falling. In this case, if part of the package overlaps with the fork holes of the pallet, the point cloud for the fork holes will also be detected, which may reduce the accuracy of detecting the position and orientation of the pallet.

[0005] An object of the present disclosure is to provide a pallet detection device and a pallet detection method that can accurately detect the position and orientation of a pallet even when the cargo is wrapped in a package. [Means for solving the problem]

[0006] A pallet detection device according to one aspect of the present disclosure comprises an acquisition unit that detects reflected light from inspection light irradiated toward the pallet and acquires a point cloud of data corresponding to the detection results, a processing unit that calculates a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point and excludes data that falls outside the threshold from the point cloud, and a detection unit that detects the position and orientation of the pallet based on the point cloud after processing by the processing unit.

[0007] This pallet detection device focuses on the fact that the intensity of light reflected from a pallet when illuminated with inspection light differs from the intensity of light reflected from the packaging, and calculates a threshold value by taking the intensity of the reflected light into account when calculating the arrangement of a point cloud based on the detection of reflected light. This makes it possible to easily distinguish between point clouds based on reflection from the pallet and point clouds based on reflection from the packaging based on the calculated threshold. By excluding data that falls outside the threshold value from the point cloud, the position and orientation of the pallet can be detected with high accuracy, even when the goods are wrapped in packaging.

[0008] The processing unit may calculate an index value for each point in the point cloud by adding the intensity of reflected light to the distance between the point and a neighboring point of the point cloud, and calculate a threshold value based on the average and standard deviation of the index values ​​calculated for all point clouds. Using such a threshold value allows for accurate distinction between point clouds based on reflection from the pallet and point clouds based on reflection from packaging. Therefore, even when the package contains an object, the position and orientation of the pallet can be detected with even greater accuracy.

[0009] The processing unit may calculate a threshold index value by adding the number of neighboring points for a point in the point cloud to the intensity of the reflected light. Using such a threshold value allows for accurate distinction between point clouds based on reflection from the pallet and point clouds based on reflection from the package. Therefore, even when the package is wrapped around the pallet, the position and orientation of the pallet can be detected with even greater accuracy.

[0010] In calculating the threshold, the processing unit may weight the relationship between the alignment state and the intensity of the reflected light based on the difference between the intensity of the reflected light from the pallet and the intensity of the reflected light from the packages of the packages supported on the pallet. In this case, the threshold can be calculated taking into account the materials of the pallet and the packages. Therefore, the point cloud based on the reflection from the pallet and the point cloud based on the reflection from the packages can be more accurately distinguished.

[0011] The system may further include an imaging unit that images the pallet and the package supported on the pallet, and a determination unit that determines whether the package is wrapped in a package based on the imaging results of the imaging unit, and the processing unit may calculate a threshold based on the arrangement of the point cloud and the intensity of reflected light at each point if the determination unit determines that the package is wrapped in a package, and may calculate a threshold based only on the arrangement of the point cloud if the determination unit determines that the package is not wrapped in a package. When it is determined that the package is not wrapped in a package, the intensity of reflected light is excluded from the parameters used to calculate the threshold, thereby reducing the computational load of the threshold.

[0012] A pallet detection method according to one aspect of the present disclosure comprises an acquisition step of detecting reflected light from inspection light irradiated toward the pallet and acquiring a point cloud of data corresponding to the detection results, an exclusion step of calculating a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point and excluding data that falls outside the threshold from the point cloud, and a detection step of detecting the position and orientation of the pallet based on the point cloud after the exclusion step.

[0013] This pallet detection method focuses on the fact that the intensity of light reflected from a pallet when irradiated with inspection light differs from the intensity of light reflected from the packaging, and calculates a threshold value by taking the intensity of the reflected light into account when calculating the arrangement of a point cloud based on the detection of reflected light. This makes it possible to easily distinguish between point clouds based on reflection from the pallet and point clouds based on reflection from the packaging based on the calculated threshold. By excluding data that falls outside the threshold value from the point cloud, the position and orientation of the pallet can be detected with high accuracy, even when the goods are wrapped in packaging.

[0014] In the excluding step, an index value may be calculated for each point in the point cloud by adding the intensity of reflected light to the distance between the point and a neighboring point of the point cloud, and a threshold value may be calculated based on the average value and standard deviation of the index values ​​calculated for all point clouds. Using such a threshold value allows for accurate distinction between point clouds based on reflection from the pallet and point clouds based on reflection from packaging. Therefore, even when the package is wrapped around the package, the position and orientation of the pallet can be detected with greater accuracy.

[0015] In the excluding step, an index value may be calculated as a threshold value by adding the intensity of reflected light to the number of neighboring points for a point in the point cloud. By using such a threshold value, it is possible to accurately distinguish between point clouds based on reflection from the pallet and point clouds based on reflection from the package. Therefore, even when the package is wrapped in a package, the position and orientation of the pallet can be detected with greater accuracy.

[0016] In the exclusion step, the threshold value may be calculated by weighting the relationship between the alignment state and the intensity of the reflected light based on the difference between the intensity of the reflected light from the pallet and the intensity of the reflected light from the packaging of the package supported by the pallet. In this case, the threshold value can be calculated taking into account the material of the pallet and the material of the packaging. Therefore, the point cloud based on the reflection from the pallet and the point cloud based on the reflection from the packaging can be more accurately distinguished.

[0017] The method may further include an imaging step of imaging the pallet and the package supported on the pallet, and a determination step of determining whether the package is wrapped in a package based on the image results from the imaging step, and in the exclusion step, if the determination step determines that the package is wrapped in a package, a threshold value may be calculated based on the arrangement of the point cloud and the intensity of reflected light at each point, and if the determination step determines that the package is not wrapped in a package, a threshold value may be calculated based only on the arrangement of the point cloud. If it is determined that the package is not wrapped in a package, the intensity of reflected light is excluded from the parameters used to calculate the threshold value, thereby reducing the computational load of the threshold value. [Effects of the Invention]

[0018] According to the present disclosure, the position and orientation of a pallet can be detected with high accuracy even when the cargo is wrapped in a package. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic plan view showing a forklift equipped with a pallet detection device according to an embodiment of the present disclosure, together with a pallet to be handled. [Figure 2] 1 is a block diagram of an automated driving system including a pallet detection device according to an embodiment of the present disclosure. FIG. [Figure 3] FIG. 1 is a diagram showing an example of a package in which cargo supported on a pallet is packaged. [Figure 4] 3 is a flowchart showing a processing procedure executed by the autonomous driving system shown in FIG. 2. [Figure 5] FIG. 10 is a diagram illustrating an example of a point cloud acquired as a detection result of a pallet. [Figure 6] 10 is a flowchart showing details of a filtering process based on an arrangement state and reflection intensity. [Figure 7] 6 is a diagram for explaining calculation of a distance at a relevant point in the point cloud shown in FIG. 5. FIG. [Figure 8] 10 is a flowchart showing details of a filtering process based on an array state. [Figure 9] 10 is a flowchart showing details of a filtering process based on the arrangement state and reflection intensity according to a modified example. [Figure 10] 10 is a flowchart showing details of a filtering process based on an arrangement state according to a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, preferred embodiments of a palette detection device and palette detection method according to an embodiment of the present disclosure will be described in detail with reference to the drawings.

[0021] Fig. 1 is a schematic plan view showing a forklift 1 equipped with a pallet detection device according to an embodiment of the present disclosure, together with a pallet 5 to be handled. In Fig. 1, the forklift 1 includes a vehicle body 2 and a loading device 3 disposed in front of the vehicle body 2 and used for loading and unloading. The loading device 3 includes a mast 4 attached to the front end of the vehicle body 2 and a pair of forks 6 attached to the mast 4 so as to be able to rise and fall and lift the pallet 5.

[0022] The pallet 5 is a loading platform for placing cargo. The pallet 5 is the loading target on which loading by the loading device 3 is about to begin, and is positioned in front of the forklift 1. The pallet 5 is, for example, a flat pallet. Pallets 5 come in a variety of colors and materials. The pallet 5 has a rectangular shape in a plan view. The pallet 5 has a front surface 5a, a rear surface 5b opposite the front surface 5a, and two side surfaces 5c perpendicular to the front surface 5a and rear surface 5b. The front surface 5a faces the forklift 1 when the pallet 5 is lifted by the forks 6. The pallet 5 has two fork holes 7 into which a pair of forks 6 are inserted. The fork holes 7 extend from the front surface 5a to the rear surface 5b of the pallet 5. The fork holes 7 are, for example, rectangular in shape in a front view.

[0023] FIG. 2 is a block diagram of an automatic driving system 11 equipped with a pallet detection device 12 according to one embodiment of the present disclosure. In FIG. 2, the automatic driving system 11 is a system that performs automatic driving of a forklift 1. The automatic driving system 11 is mounted on the forklift 1. The automatic driving system 11 includes the pallet detection device 12, a drive unit 20, and a notification unit 21. Although not specifically shown, the drive unit 20 includes, for example, a travel motor that rotates the drive wheels and a steering motor that steers the steering wheels. The notification unit 21 is, for example, a display or an alarm.

[0024] The pallet detection device 12 has a camera 13, a laser sensor 14, and a control unit 15. The pallet detection device 12 identifies the position and orientation of the pallet 5 based on the point cloud of the pallet 5 acquired by the laser sensor 14. Then, the pallet detection device 12 controls the operation of the forklift 1 based on the identified position and orientation of the pallet 5.

[0025] FIG. 3 illustrates an example of a package L supported on a pallet 5 wrapped in a package W. The package L supported on the pallet 5 may be wrapped in the package W to prevent it from falling. The package W may be, for example, a commercial packaging wrap. The packaging wrap may be, for example, a colorless, transparent plastic wrap. In the example shown in FIG. 3, the package W is wrapped around the side of the package L, and an end Wa of the package W is off the side of the package L and overlaps with the fork holes 7 of the pallet 5. In this case, a point cloud due to the package W may be detected relative to the fork holes 7, potentially reducing the detection accuracy of the position and orientation of the pallet 5. To address this issue, the pallet detection device 12 according to this embodiment focuses on the intensity of reflected light L2 between the pallet 5 and the package W and eliminates the point cloud due to the package W, thereby improving the detection accuracy of the position and orientation of the pallet 5. Each component of the pallet detection device 12 is described in detail below.

[0026] The camera 13 is an imaging unit that captures images of the pallet 5 and the luggage L supported on the pallet 5 (the luggage L is omitted in FIG. 2). The camera 13 captures images of the pallet 5 and the luggage L based on instructions from the control unit 15, and outputs captured image data Im to the determination unit 16. The camera 13 and the laser sensor 14 are installed, for example, on the front side of the vehicle body 2.

[0027] The laser sensor 14 is an acquisition unit that detects reflected light L2 of the inspection light L1 irradiated toward the pallet 5 and acquires a point cloud of data corresponding to the detection results. The point cloud is a set of data points representing a collection of reflected points of the laser light. For example, a 3D (three-dimensional) LiDAR (Light Detection and Ranging) is used as the laser sensor 14. Based on instructions from the control unit 15, the laser sensor 14 irradiates the inspection light L1, which is a laser light, toward the pallet 5. The laser sensor 14 then detects the reflected light L2 reflected by the pallet 5 and acquires a point cloud of data corresponding to the detection results. Here, the laser sensor 14 acquires the coordinate values ​​of each point in the point cloud. The coordinate values ​​will be described later.

[0028] The laser sensor 14 acquires the intensity of reflected light (reflection intensity) at each point of the point cloud, along with the coordinate values ​​of each point of the point cloud. The reflection intensity of the pallet 5 depends on the material of the packaging W, but generally tends to be relatively greater than the reflection intensity of the packaging W. The laser sensor 14 acquires the reflection intensity, for example, as a digital value. The laser sensor 14 acquires the reflection intensity in decimal format, for example, from 0 to 255. The laser sensor 14 may acquire data closer to 0 as the reflection intensity increases, or may acquire data closer to 255 as the reflection intensity increases. The laser sensor 14 outputs the coordinate values ​​and reflection intensity of each point of the point cloud to the processing unit 17.

[0029] The control unit 15 is physically configured with a CPU, RAM, ROM, an input / output interface, etc. Functionally, the control unit 15 has a determination unit 16, a processing unit 17, a detection unit 18, and a drive control unit 19.

[0030] The determination unit 16 recognizes the pallet 5 and the luggage L based on the captured image data Im of the camera 13. The captured image data Im output from the camera 13 is input to the determination unit 16. The determination unit 16 analyzes the captured image data Im and determines whether the luggage L is wrapped in a wrapping material W. The determination unit 16 outputs the determination result to the processing unit 17.

[0031] The processing unit 17 performs a filtering process on the point cloud acquired by the laser sensor 14. The detection results output from the laser sensor 14 are input to the processing unit 17. The detection results include the coordinate values ​​and reflection intensity of each point in the point cloud. As a filtering process, the processing unit 17 calculates a threshold based on the arrangement of the point cloud and the reflection intensity at each point, and excludes data that does not meet the threshold from the point cloud. As a result, the processing unit 17 excludes the point cloud due to the packaging material W from the point cloud included in the detection result, and extracts only the point cloud due to the pallet 5. The processing unit 17 outputs the detection results after the filtering process to the detection unit 18.

[0032] The processing unit 17 switches the content of the filtering process depending on the judgment result output from the judgment unit 16. When the judgment unit 16 judges that the package L is wrapped in a package W, the processing unit 17 calculates a threshold based on the arrangement state of the point cloud and the reflection intensity at each point. When the judgment unit 16 judges that the package L is not wrapped in a package W, the processing unit 17 calculates a threshold based only on the arrangement state of the point cloud.

[0033] The detection unit 18 detects the position and orientation of the pallet 5. The detection result after filtering processing from the processing unit 17 is input to the detection unit 18. The detection unit 18 calculates a plane equation for the front surface 5a of the pallet 5 based on the point cloud included in the detection result after filtering processing. The detection unit 18 then detects the two fork holes 7 based on the calculated plane equation for the front surface 5a of the pallet 5. Next, the detection unit 18 estimates the position and orientation of the pallet 5 using the plane equation for the front surface 5a of the pallet 5. At this time, the detection unit 18 calculates the yaw angle, pitch angle, and roll angle of the pallet 5 as the orientation of the pallet 5. The detection unit 18 calculates the yaw angle and pitch angle of the pallet 5 based on the plane equation for the front surface 5a of the pallet 5, and also calculates the roll angle of the pallet 5 based on the positional relationship of the two fork holes 7. Finally, the detection unit 18 finally determines whether the dimensions of the front surface 5a of the pallet 5, whose position and orientation have been estimated, match predetermined specified values. Then, the detection unit 18 outputs the determination result to the drive control unit 19.

[0034] The drive control unit 19 controls the drive unit 20 and the notification unit 21. The drive control unit 19 receives a determination result from the detection unit 18. When the detection unit 18 determines that the dimensions of the front surface 5a of the pallet 5 match predetermined specified values, the drive control unit 19 controls the drive unit 20 to move the forklift 1 to a position close to and in front of the pallet 5 to be loaded, based on the position and posture of the pallet 5 estimated by the detection unit 18. Furthermore, when the detection unit 18 determines that the dimensions of the front surface 5a of the pallet 5 do not match the predetermined specified values, the drive control unit 19 outputs notification information to the notification unit 21 indicating that an abnormality has occurred.

[0035] Next, the operation of the above-mentioned pallet detection device 12 will be described. Here, the operation of the automatic driving system 11 including the pallet detection device 12 will be described. Figure 4 is a flowchart showing the processing procedure executed by the automatic driving system 11 shown in Figure 2. Note that this processing is started, for example, when automatic driving of the forklift 1 is started in accordance with an instruction.

[0036] 4, the laser sensor 14 irradiates the pallet 5 with laser light (inspection light L1) (step S11). Then, the laser sensor 14 detects reflected light L2 of the laser light (step S12). Next, the laser sensor 14 acquires a point cloud of data corresponding to the detection result (acquisition step; step S13).

[0037] FIG. 5 is a diagram showing an example of a point cloud C1 acquired by the laser sensor 14 as a detection result of the pallet 5. In this embodiment, the laser sensor 14 acquires the point cloud C1 as a three-dimensional point cloud in a three-dimensional coordinate system formed by an X axis, a Y axis perpendicular to the X axis, and a Z axis perpendicular to the X and Y axes. Three-dimensional coordinate values ​​are set for each point P that constitutes the point cloud C1. The point cloud C1 includes a pallet corresponding area C11 and a fork hole corresponding area C12. The pallet corresponding area C11 is a point cloud that corresponds to the pallet 5. In the example of FIG. 5, the front surface of the pallet corresponding area C11 (front surface 5a of the pallet 5) is parallel to the YZ plane defined by the Y axis and the Z axis.

[0038] The fork hole corresponding region C12 corresponds to the two fork holes 7. Because the fork holes 7 are intangible objects, there is generally no point cloud in the fork hole corresponding region C12. On the other hand, as illustrated in FIG. 3, when the end Wa of the package W overlaps with the fork holes 7 of the pallet 5, a package corresponding region C13 appears in the point cloud C1. The package corresponding region C13 is a point cloud corresponding to the package W. In the example of FIG. 3, the end Wa of the package W overlaps with the fork holes 7 of the pallet 5, and therefore in the example of FIG. 5, the package corresponding region C13 is located in the fork hole corresponding region C12. The point density of the package corresponding region C13 tends to be higher than the point density of the pallet corresponding region C11.

[0039] The camera 13 captures an image of the pallet 5 and the package L supported on the pallet 5 (imaging step) and outputs the captured image data Im to the determination unit 16. The determination unit 16 acquires the captured image data Im from the camera 13 (step S14). The determination unit 16 then recognizes the pallet 5 and the package L based on the captured image data Im using image processing technology utilizing deep learning (step S15). Furthermore, if the package L is wrapped in a package W, the determination unit 16 recognizes the package W (step S15). In this case, if the package L is not wrapped in a package W, the determination unit 16 does not recognize the package W. Deep learning is one of the elemental technologies of artificial intelligence. Deep learning is a learning method using a deep neural network that allows a machine to automatically extract features from data without human intervention, provided there is a sufficient amount of data. Deep learning increases information transmission and processing by adding multiple intermediate layers between the input layer and the output layer, making it possible to improve the accuracy and versatility of features and improve prediction accuracy.

[0040] Specifically, the determination unit 16 specifies a frame line that surrounds the pallet 5, the luggage L, and the packaging W in the captured image data Im. The frame line is a rectangular frame called a bounding box in object detection using deep learning. The determination unit 16 then compares the captured image data Im with the specified frame line with the learning data to recognize the pallet 5, the luggage L, and the packaging W.

[0041] The acquisition of the point cloud C1 performed by steps S11 to S13 may be performed simultaneously with the recognition of the pallet 5, luggage L, and packaging W performed by steps S14 and S15, or may be performed after steps S14 and S15.

[0042] After step S15, the determination unit 16 determines whether or not the packaging W has been recognized (determination step; step S16). If the packaging W has been recognized (step S16; YES), the determination unit 16 determines that the package L is wrapped in the packaging W, and the processing unit 17 performs filtering processing based on the arrangement state and reflection intensity (exclusion step; step S17). On the other hand, if the packaging W has not been recognized (step S16; NO), the determination unit 16 determines that the package L is not wrapped in the packaging W, and the processing unit 17 performs filtering processing based on the arrangement state (exclusion step; step S18).

[0043] FIG. 6 is a flowchart showing details of the filtering process (step S17) based on the arrangement state and reflection intensity. In step S17, the processing unit 17 first calculates the distance between the relevant point and neighboring points (step S31). FIG. 7 is a diagram for explaining calculation of the distance at the relevant point P1 in the point cloud C1 shown in FIG. 5. The following explanation will be made with reference to FIGS. 6 and 7. In this embodiment, the arrangement state of the point cloud C1 is defined as the distance between each point P constituting the point cloud C1 and each of the multiple neighboring points. For convenience of explanation, FIG. 7 illustrates the explanation as being the distances D1 and D2 between one point (the relevant point P1) in the point cloud C1 and multiple neighboring points P2 and P3. The number of the multiple neighboring points P2 and P3 may be set in advance. In the example of FIG. 7, the number of the multiple neighboring points is two, P2 and P3, but may be three or more. The relevant point P1 is a point selected from the point cloud C1. The relevant point P1 may be selected, for example, in order from the point whose coordinate value of the point P is closest to the origin.

[0044] The distances D (D1, D2) between the point P1 and each of the multiple neighboring points P2 and P3 are calculated by Equation 1.

number

number

[0045] Next, the processing unit 17 calculates an index value by adding the reflection intensity to the distance D between the point P1 and the neighboring points P2 and P3 (step S32). The index value IV is calculated by Equation 3.

number

[0046] For example, when the difference is relatively large, the values ​​of coefficient A and coefficient B may be determined so that the value of coefficient B, which represents the weight of reflection intensity In, is greater than the value of coefficient A. This increases the weight of reflection intensity In as an index for distinguishing between the pallet 5 and the packaging W. On the other hand, when the difference is relatively small, the values ​​of coefficient A and coefficient B may be determined so that the value of coefficient A, which represents the weight of distance D, is greater than the value of coefficient B. This increases the weight of distance D as an index for distinguishing between the pallet 5 and the packaging W. The values ​​of coefficient A and coefficient B may be set so that the sum of both equals 1. As an example, coefficient A may be set to 0.7 and coefficient B may be set to 0.3. Furthermore, coefficient A and coefficient B may be set to the same value for all points P constituting point group C1, or may be set to different values ​​for each point P.

[0047] For example, the index value IV1 between the point P1 and the neighboring point P2 is calculated based on the formula 3 as shown in the formula 4.

number

[0048] Subsequently, the processing unit 17 averages the index values ​​IV to calculate the average distance DA at each point P (step S33). The average distance DA is calculated by Equation 5.

number

[0049] Next, the processing unit 17 calculates the average value of the average distances DA of all points P (step S34). Specifically, the average value is calculated by summing up the average distances DA of all points P and dividing the sum by the total number of all points P. Thereafter, the processing unit 17 calculates the standard deviation of the average distances DA of all points P (step S35). Specifically, the processing unit 17 sums up the squares of the differences between the average distances DA of each point P and the average value, divides the sum by the total number of all points P, and calculates the standard deviation by taking the square root of the divided value.

[0050] Next, the processing unit 17 calculates the threshold value Th (step S36). The processing unit 17 calculates the threshold value Th based on the average value and standard deviation of the index values ​​IV calculated for all points P (all point groups). The threshold value Th is calculated using Equation 6.

number

[0051] Next, the processing unit 17 determines whether the average distance DA at the relevant point P1 is equal to or less than the threshold value Th (step S37). If the average distance DA at the relevant point P1 is equal to or less than the threshold value Th (step S37; YES), the processing unit 17 determines that the relevant point P1 is a point caused by the pallet 5, and uses the relevant point P1 as a pallet-corresponding area C11 (step S38). On the other hand, if the average distance DA at the relevant point P1 is not equal to or less than the threshold value Th (step S37; NO), the processing unit 17 determines that the relevant point P1 is a point caused by the package W, and excludes the relevant point P1 from the point group C1 (step S39).

[0052] In the processing of steps S37 to S39, if the average distance DA at the relevant point P1 is equal to or less than the threshold value Th (step S37; YES), the processing unit 17 may determine that the relevant point P1 is a point caused by the package W, and may exclude the relevant point P1 from the point group C1. On the other hand, if the average distance DA at the relevant point P1 is not equal to or less than the threshold value Th (step S37; NO), the processing unit 17 may determine that the relevant point P1 is a point caused by the pallet 5, and may use the relevant point P1 as the pallet corresponding area C11. As described above, the laser sensor 14 may acquire data closer to 0 as the reflection intensity In increases, or may acquire data closer to 255 as the reflection intensity In increases. Therefore, the processing unit 17 may change the processing of steps S37 to S39 depending on which data acquisition method the laser sensor 14 employs.

[0053] The processing of steps S37 to S39 completes the filtering process (step S17) based on the arrangement state and reflection intensity In for the point P1. The processing unit 17 performs step S17 on all points P that make up the point group C1. As a result, the processing unit 17 extracts a pallet-corresponding region C11 from the point group C1 and excludes a package-corresponding region C13.

[0054] 8 is a flowchart showing the details of the filtering process (step S18) based on the arrangement state. Only the differences from step S17 will be explained. In step S18, the processing unit 17 calculates the index value IV using Equation 7 based only on the distance D between the point P1 and the neighboring points P2 and P3 (step S41).

number

[0055] Referring again to FIG. 4, after step S17 or step S18, the detection unit 18 calculates a plane equation for the front surface 5a of the pallet 5 based on the point group C1 included in the detection result after the filtering process (step S19). The detection unit 18 uses a robust estimation method such as RANSAC (Random Sample Consensus) to remove point groups other than the point group corresponding to the plane in the point group C1 as outliers, thereby obtaining the plane equation for the front surface 5a of the pallet 5. Robust estimation is a method intended to reduce the influence of outliers included in the measurement values. Note that the plane equation for the front surface 5a of the pallet 5 may be calculated using a least squares method or the like instead of the robust estimation method.

[0056] Next, the detection unit 18 detects the two fork holes 7 based on the plane equation of the front surface 5a of the pallet 5 calculated in step S19 (step S20). At this time, in the point cloud C1, the area with a high point density is detected as the front surface 5a of the pallet 5, and the two areas with a low point density are detected as the fork holes 7.

[0057] Next, the detection unit 18 calculates the position of the pallet 5 and the yaw angle and pitch angle of the pallet 5 based on the plane equation of the front surface 5a of the pallet 5 (step S21). The yaw angle of the pallet 5 is the angle of rotation around the axis of the up-and-down direction (height direction) of the pallet 5. The pitch angle of the pallet 5 is the angle of rotation around the axis of the left-right direction (width direction) of the pallet 5.

[0058] Next, the detection unit 18 calculates the roll angle of the pallet 5 based on the positional relationship of the two fork holes 7 detected in step S20 (step S22). The roll angle of the pallet 5 is the angle of rotation around the axis in the front-to-back direction (depth direction) of the pallet 5. Specifically, the detection unit 18 calculates the center positions of the two fork holes 7, and calculates the roll angle of the pallet 5 from the relationship between the center positions of the two fork holes 7. Note that the calculation to estimate the position of the pallet 5 may also be performed based on the positional relationship of the two fork holes 7.

[0059] Next, the detection unit 18 determines whether the dimensions of each part of the front surface 5a of the pallet 5, the position and orientation of which have been estimated in steps S21 and S22, match predetermined specified values ​​(step S23). The dimensions of each part of the front surface 5a of the pallet 5 include the width and height of the pallet 5, the dimensions of the two fork holes 7, and the distance between the centers of the two fork holes 7. Steps S19 to S23 described above constitute the detection step.

[0060] When the drive control unit 19 determines that the dimensions of each part of the front surface 5a of the pallet 5 match the specified values ​​(step S23; YES), it controls the drive unit 20 to move the forklift 1 to a position close to the front of the pallet 5 to be loaded, based on the position and posture of the pallet 5 (step S24). When the drive control unit 19 determines that the dimensions of each part of the front surface 5a of the pallet 5 do not match the specified values ​​(step S23; NO), it outputs notification information indicating that an abnormality has occurred to the notification unit 21 (step S25). The series of processes executed by the automatic driving system 11 ends with the processing of step S24 or step S25.

[0061] As described above, the pallet detection device 12 according to an embodiment of the present disclosure focuses on the fact that the intensity (reflection intensity In) of the reflected light L2 from the pallet 5 in response to irradiation with the inspection light L1 differs from the reflection intensity In from the package W, and calculates the threshold value Th by taking into account the reflection intensity In in the arrangement of the point cloud C1 based on the detection of the reflected light L2. This makes it possible to easily distinguish between the point cloud C1 based on reflection from the pallet 5 (pallet corresponding area C11) and the point cloud C1 based on reflection from the package W (package corresponding area C13) based on the calculated threshold value Th. By excluding data that falls outside the threshold value Th from the point cloud C1, the position and orientation of the pallet 5 can be detected with high accuracy even when the package L is wrapped in a package W.

[0062] The processing unit 17 calculates an index value IV for each point P by adding the reflection intensity In to the distance D between a point (corresponding point P1) in the point group C1 and neighboring points P2 and P3 of the point P1, and calculates a threshold value Th based on the average value and standard deviation of the index values ​​IV calculated for the entire point group C1. By using such a threshold value Th, the pallet corresponding area C11 and the package corresponding area C13 can be distinguished with high accuracy. Therefore, even when the package L is wrapped in a package W, the position and orientation of the pallet 5 can be detected with high accuracy.

[0063] When calculating the threshold value Th, the processing unit 17 weights the relationship between the arrangement state and the reflection intensity In based on the difference between the reflection intensity In at the pallet 5 and the reflection intensity In at the packaging W of the luggage L supported on the pallet 5. In this case, it is possible to calculate the threshold value Th taking into account the material of the pallet 5 and the material of the packaging W. Therefore, it is possible to distinguish the pallet corresponding area C11 from the packaging corresponding area C13 with even greater accuracy.

[0064] The system further includes a camera 13 that captures images of the pallet 5 and the luggage L supported on the pallet 5, and a judgment unit 16 that judges whether the luggage L is wrapped in a packaging material W based on the image capture results of the camera 13. If the judgment unit 16 judges that the luggage L is wrapped in a packaging material W, the processing unit 17 calculates a threshold value Th based on the arrangement state of the point cloud C1 and the reflection intensity In at each point P, and if the judgment unit 16 judges that the luggage L is not wrapped in a packaging material W, the processing unit 17 calculates the threshold value Th based only on the arrangement state of the point cloud C1. If it is judged that the luggage L is not wrapped in a packaging material W, the reflection intensity In is excluded from the parameters used to calculate the threshold value Th, thereby reducing the computational load of the threshold value Th.

[0065] In a pallet detection method using a pallet detection device 12 according to an embodiment of the present disclosure, the reflection intensity In of the inspection light L1 emitted from the pallet 5 differs from the reflection intensity In of the package W, and a threshold value Th is calculated by taking into account the reflection intensity In in the arrangement of the point cloud C1 based on the detection of the reflected light L2. This makes it possible to easily distinguish between the pallet corresponding area C11 and the package corresponding area C13 based on the calculated threshold value Th. By excluding data that falls outside the threshold value Th from the point cloud C1, the position and orientation of the pallet 5 can be detected with high accuracy even when the package L is wrapped in a package W.

[0066] In the exclusion step S17, an index value IV is calculated for each point P by adding the reflection intensity In to the distance D between a point (corresponding point P1) in the point group C1 and neighboring points P2 and P3 of the point P1, and a threshold value Th is calculated based on the average value and standard deviation of the index values ​​IV calculated for the entire point group C1. By using such a threshold value Th, the pallet corresponding area C11 and the package corresponding area C13 can be distinguished with high accuracy. Therefore, even when the package L is wrapped in a package W, the position and orientation of the pallet 5 can be detected with even higher accuracy.

[0067] In the exclusion step S17, when calculating the threshold value Th, a weight is assigned between the arrangement state and the reflection intensity In based on the difference between the reflection intensity In at the pallet 5 and the reflection intensity In at the packaging W of the luggage L supported on the pallet 5. In this case, the threshold value Th can be calculated taking into account the material of the pallet 5 and the material of the packaging W. Therefore, the pallet corresponding area C11 and the packaging corresponding area C13 can be distinguished with even greater accuracy.

[0068] The method further includes step S14, which includes an imaging step of imaging the pallet 5 and the luggage L supported on the pallet 5, and a judgment step S16 of judging whether or not the luggage L is packaged in a packaging material W based on the imaging results in the imaging step, and in the exclusion step S17, if it is judged in the judgment step S16 that the luggage L is packaged in a packaging material W, a threshold is calculated based on the arrangement state of the point cloud C1 and the reflection intensity In at each point P, and if it is judged in the judgment step S16 that the luggage L is not packaged in a packaging material W, a threshold Th is calculated based only on the arrangement state of the point cloud C1. If it is judged that the luggage L is not packaged in a packaging material W, the reflection intensity In is excluded from the parameters used to calculate the threshold value Th, thereby reducing the computational load of the threshold value Th.

[0069] Although the embodiments of the present disclosure have been described above, the present disclosure is not necessarily limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present disclosure.

[0070] FIG. 9 is a flowchart showing details of the filtering process (exclusion step; step S17A) based on the arrangement state and reflection intensity according to a modified example. FIG. 9 illustrates the filtering process performed when the determination unit 16 recognizes the package W (step S16 in FIG. 4; YES). The processing unit 17 may perform the filtering process in place of step S17. In step S17A, the processing unit 17 first counts the number of neighboring points for the point P1 (step S51). In this modified example, the arrangement state of the point cloud C1 is defined as the number of neighboring points for each point P constituting the point cloud C1. The counting of the number of points is performed, for example, by a point counter held by the processing unit 17. The processing unit 17 calculates the distance between the point P1 and the surrounding points, and if the calculated distance is equal to or less than a preset distance threshold, the point counter is counted up. On the other hand, if the calculated distance is greater than the preset distance threshold, the point counter is not counted up.

[0071] Next, the processing unit 17 calculates the point number threshold PTh by adding the reflection intensity to the number of counted neighboring points (step S52). The point number threshold PTh is calculated by Equation 8.

number

[0072] Next, the processing unit 17 determines whether the number Pn of counted neighboring points at the relevant point P1 is equal to or less than the point threshold PTh (step S53). If the number Pn of counted neighboring points is equal to or less than the point threshold PTh (step S53; YES), the processing unit 17 determines that the relevant point P1 is a point caused by the pallet 5, and uses the relevant point P1 as the pallet-corresponding area C11 (step S54). On the other hand, if the number Pn of counted neighboring points is not equal to or less than the point threshold PTh (step S53; NO), the processing unit 17 determines that the relevant point P1 is a point caused by the package W, and excludes the relevant point P1 from the point group C1 (step S55). With the processing of steps S53 to S55, step S17A for the relevant point P1 is completed. The processing unit 17 performs step S17A for all points P constituting the point group C1. As a result, the processing unit 17 extracts the pallet-corresponding region C11 from the point group C1 and excludes the package-corresponding region C13.

[0073] By employing step S17A, the processing unit 17 calculates an index value as a threshold (point number threshold PTh) by adding the reflection intensity In to the number Pn of points neighboring one point (corresponding point P1) in the point cloud C1. By using such a point number threshold PTh, the pallet corresponding area C11 and the package corresponding area C13 can be distinguished with high accuracy. Therefore, even when the package L is wrapped in a package W, the position and orientation of the pallet 5 can be detected with even higher accuracy.

[0074] In the exclusion step S17A, an index value obtained by adding the reflection intensity In to the number of points Pn of neighboring points for one point (corresponding point P1) in the point cloud C1 is calculated as a threshold (point number threshold PTh). By using such point number threshold PTh, the pallet corresponding area C11 and the package corresponding area C13 can be distinguished with high accuracy. Therefore, even when the package L is wrapped in a package W, the position and orientation of the pallet 5 can be detected with even higher accuracy.

[0075] If the determination unit 16 does not recognize the packaging object W (step S16 in FIG. 4; NO), the processing unit 17 may perform the filtering process by adopting step S18A instead of step S18. FIG. 10 is a flowchart showing the details of the filtering process based on the arrangement state according to a modified example (exclusion step; step S18A). Only the differences from step S17A will be explained. In step S18A, the processing unit 17 calculates the score threshold PTh using Equation 9 based only on the number Pn of counted neighboring points (step S62).

number

[0076] In the flowcharts described above, when the processing unit 17 compares the magnitude relationship of two numerical values, it may use either of the two criteria of "greater than or equal to" and "greater than", or it may use either of the two criteria of "less than or equal to" and "less than".

[0077] The gist of the present disclosure is as follows [1] to

[10] . [1] A pallet detection device comprising: an acquisition unit that detects reflected light from inspection light irradiated toward a pallet and acquires a point cloud of data corresponding to the detection results; a processing unit that calculates a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point and excludes data that falls outside the threshold from the point cloud; and a detection unit that detects the position and orientation of the pallet based on the point cloud after processing by the processing unit. [2] The processing unit calculates an index value at each point by adding the intensity of the reflected light to the distance between a point in the point cloud and a point adjacent to the point, and calculates the threshold value based on the average value and standard deviation of the index values ​​calculated for the entire point cloud. [1] The pallet detection device described in [1]. [3] The palette detection device according to [1] or [2], wherein the processing unit calculates an index value as the threshold value by adding the number of neighboring points for one point in the point cloud to the intensity of the reflected light. [4] A pallet detection device described in any one of [1] to [3], wherein the processing unit, when calculating the threshold value, weights the relationship between the arrangement state and the intensity of the reflected light based on the difference between the intensity of the reflected light on the pallet and the intensity of the reflected light on the packaging of the cargo supported on the pallet. [5] A pallet detection device described in any of [1] to [4], further comprising an imaging unit that images the pallet and the luggage supported on the pallet, and a judgment unit that judges whether the luggage is wrapped in a package based on the imaging results of the imaging unit, wherein the processing unit calculates a threshold value based on the arrangement state of the point cloud and the intensity of the reflected light at each point when the judgment unit judges that the luggage is wrapped in the package, and calculates a threshold value based only on the arrangement state of the point cloud when the judgment unit judges that the luggage is not wrapped in the package. [6] A pallet detection method comprising: an acquisition step of detecting reflected light from inspection light irradiated toward the pallet and acquiring a point cloud of data corresponding to the detection results; an exclusion step of calculating a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point and excluding data that falls outside the threshold from the point cloud; and a detection step of detecting the position and orientation of the pallet based on the point cloud after the exclusion step. [7] In the excluding step, an index value is calculated for each point by adding the intensity of the reflected light to the distance between a point in the point cloud and a neighboring point of the point, and the threshold value is calculated based on the average value and standard deviation of the index values ​​calculated for the entire point cloud. [8] A palette detection method according to [6] or [7], wherein in the exclusion step, an index value obtained by adding the number of neighboring points for a point in the point cloud to the intensity of the reflected light is calculated as the threshold value. [9] A pallet detection method according to any one of [6] to [8], wherein in the exclusion step, when calculating the threshold, a weighting is performed between the arrangement state and the intensity of the reflected light based on the difference between the intensity of the reflected light on the pallet and the intensity of the reflected light on the packaging of the cargo supported on the pallet.

[10] A pallet detection method described in any of [6] to [9], further comprising an imaging step of imaging the pallet and the luggage supported on the pallet, and a judgment step of judging whether the luggage is wrapped in a package based on the imaging results of the imaging step, wherein in the exclusion step, if it is judged in the judgment step that the luggage is wrapped in the package, a threshold is calculated based on the arrangement state of the point cloud and the intensity of the reflected light at each point, and if it is judged in the judgment step that the luggage is not wrapped in the package, a threshold is calculated based only on the arrangement state of the point cloud. [Explanation of symbols]

[0078] 5...pallet, 12...pallet detection device, 13...imaging unit (camera), 14...acquisition unit (laser sensor), 16...judgment unit, 17...processing unit, 18...detection unit, C1...point cloud, D, D1, D2...distance, IV, IV1, IV2...index value, L...baggage, L1...inspection light, L2...reflected light, P...point, P2, P3...neighboring point, Pn...number of points, S16...judgment step, S17, S17A...exclusion step, Th...threshold value, W...packaging body.

Claims

1. an acquisition unit that detects reflected light of inspection light irradiated onto the pallet and acquires a point cloud of data corresponding to the detection results; a processing unit that calculates a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point, and excludes data that deviates from the threshold from the point cloud; A pallet detection device comprising: a detection unit that detects the position and orientation of the pallet based on the point cloud processed by the processing unit.

2. The pallet detection device described in claim 1, wherein the processing unit calculates an index value at each point by adding the intensity of the reflected light to the distance between a point in the point cloud and a point adjacent to the point, and calculates the threshold value based on the average value and standard deviation of the index values ​​calculated for the entire point cloud.

3. 2. The palette detection device according to claim 1, wherein the processing unit calculates, as the threshold value, an index value obtained by adding the intensity of the reflected light to the number of neighboring points for one point in the point cloud.

4. The pallet detection device according to claim 1, wherein the processing unit, when calculating the threshold value, weights the relationship between the arrangement state and the intensity of the reflected light based on the difference between the intensity of the reflected light on the pallet and the intensity of the reflected light on the packaging of the cargo supported on the pallet.

5. an imaging unit that images the pallet and the cargo supported on the pallet; a determination unit that determines whether the package is wrapped in a wrapping material based on the image pickup result of the image pickup unit, A pallet detection device as described in any one of claims 1 to 4, wherein the processing unit calculates a threshold value based on the arrangement state of the point cloud and the intensity of the reflected light at each point when the judgment unit determines that the luggage is wrapped in the packaging material, and calculates a threshold value based only on the arrangement state of the point cloud when the judgment unit determines that the luggage is not wrapped in the packaging material.

6. an acquisition step of detecting reflected light of inspection light irradiated onto the pallet and acquiring a point cloud of data corresponding to the detection results; an exclusion step of calculating a threshold based on the arrangement of the point cloud and the intensity of the reflected light at each point, and excluding data that does not meet the threshold from the point cloud; A pallet detection method comprising: a detection step of detecting the position and orientation of the pallet based on the point cloud after the exclusion step.

7. A palette detection method as described in claim 6, wherein in the exclusion step, an index value is calculated for each point by adding the intensity of the reflected light to the distance between a point in the point cloud and a point adjacent to the point, and the threshold is calculated based on the average value and standard deviation of the index values ​​calculated for all point clouds.

8. 7. The palette detection method according to claim 6, wherein in the excluding step, an index value obtained by adding the intensity of the reflected light to the number of neighboring points for one point in the point cloud is calculated as the threshold value.

9. 7. A pallet detection method as described in claim 6, wherein in the exclusion step, when calculating the threshold, a weighting is performed between the arrangement state and the intensity of the reflected light based on the difference between the intensity of the reflected light on the pallet and the intensity of the reflected light on the packaging of the luggage supported on the pallet.

10. an imaging step of imaging the pallet and the cargo supported on the pallet; a determination step of determining whether the package is wrapped in a wrapping material based on the image pickup result in the image pickup step, A pallet detection method described in any one of claims 6 to 9, wherein in the exclusion step, if it is determined in the judgment step that the luggage is wrapped in the packaging material, a threshold value is calculated based on the arrangement state of the point cloud and the intensity of the reflected light at each point, and if it is determined in the judgment step that the luggage is not wrapped in the packaging material, a threshold value is calculated based only on the arrangement state of the point cloud.

Images