Pallet detection device

Abstract

To provide a pallet detection device capable of detecting the position of a pallet with high accuracy while reducing costs and labor.SOLUTION: A pallet detection device 20 comprises: an acquisition unit that acquires three-dimensional point cloud data including a plurality of measurement points; a grid setting unit 13 that sets a plurality of grid cells Gs on a plane Fp by creating a grid G on the plane Fp estimated based on the point cloud data; a map creation unit 14 that creates a two-dimensional map in which the distance from the plane Fp to the measurement points is calculated as a grid value for each of the plurality of grid cells Gs; a determination unit 15 that determines whether there is an unmeasured grid cell Gs1 where data at the measurement point is missing based on the two-dimensional map; a map correction unit 16 that corrects the two-dimensional map by interpolating the grid value of the unmeasured grid cell Gs1 when it is determined that there is an unmeasured grid cell Gs1; and a position determination unit that determines the position of the pallet 5 based on the two-dimensional map.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a pallet detection device. [Background technology]

[0002] A known conventional pallet detection device is the technology described in Patent Document 1. The pallet detection device described in Patent Document 1 includes an imaging device that captures an image in front of the unmanned forklift, a distance measurement device that measures the distance to a target pallet, an image acquisition unit that acquires the captured image from the imaging device, a pallet type identification unit that has a learning model for combinations of images of multiple types of pallets and the types of pallets and inputs the captured image of the target pallet acquired by the image acquisition unit into the learning model to identify the type of pallet, a pallet position and shape data acquisition unit that acquires position and shape data of the target pallet from the distance measurement device, and a pallet deviation amount detection unit that detects the amount of deviation in the position and posture of the target pallet by comparing stored position and shape data corresponding to the type of target pallet identified by the pallet type identification unit with the position and shape data of the target pallet. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-179331 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology requires the forklift to be equipped with a photographing device such as a camera that acquires image data and a distance measuring device such as LiDAR that acquires point cloud data, which increases costs. Also, a learning model (learning data) is required for each type of pallet, which requires a lot of effort to create the learning data.

[0005] An object of the present invention is to provide a pallet detection device that can detect the position of a pallet with high accuracy while reducing costs and labor. [Means for solving the problem]

[0006] (1) One aspect of the present invention is a pallet detection device that detects a pallet when handling goods with a forklift, the device having a sensor that measures the distance from the forklift to the pallet and that acquires three-dimensional point cloud data including a plurality of measurement points, an acquisition unit that acquires three-dimensional point cloud data including a plurality of measurement points, a plane estimation unit that estimates a plane corresponding to the front of the pallet based on the three-dimensional point cloud data acquired by the acquisition unit, a grid setting unit that sets a plurality of grid cells on the plane by creating a grid on the plane estimated by the plane estimation unit, and a distance calculation unit that calculates the distance from the plane estimated by the plane estimation unit to the measurement points acquired by the acquisition unit for the plurality of grid cells set by the grid setting unit. a determination unit that determines, based on the two-dimensional map created by the map creation unit, whether there are any unmeasured grid cells among the plurality of grid cells for which point cloud data has not been acquired by the acquisition unit and data at measurement points has been missing; a map correction unit that, when the determination unit determines that there are unmeasured grid cells, corrects the two-dimensional map created by the map creation unit by interpolating the grid values ​​of the unmeasured grid cells; and a position determination unit that determines the position of the pallet based on the two-dimensional map created by the map creation unit or the two-dimensional map corrected by the map correction unit.

[0007] In this type of pallet detection device, a sensor measures the distance from the forklift to the pallet and acquires three-dimensional point cloud data containing multiple measurement points. Based on the three-dimensional point cloud data, a plane corresponding to the front of the pallet is estimated. A grid is created on the plane, thereby defining multiple grid cells on the plane. A two-dimensional map is then created, calculating the distances from the plane to the measurement points for each of the grid cells as grid values. Based on the two-dimensional map, it is determined whether any of the grid cells contain unmeasured grid cells for which point cloud data was not acquired by the acquisition unit, resulting in missing measurement point data. If an unmeasured grid cell is determined to exist, the two-dimensional map is corrected by interpolating the grid values ​​of the unmeasured grid cells. The location of the pallet is then determined based on the created or corrected two-dimensional map. Using a sensor to acquire three-dimensional point cloud data in this way eliminates the need for a camera or other device to acquire two-dimensional image data. Furthermore, creating a two-dimensional map allows the location of the pallet to be detected without creating a large amount of training data. This reduces costs and effort. Furthermore, if there are unmeasured grid cells, an appropriate two-dimensional map can be obtained by interpolating the grid values ​​of the unmeasured grid cells. Therefore, the position of the pallet can be detected with high accuracy.

[0008] (2) In (1) above, the pallet detection device may further include a memory unit that stores a template corresponding to pallet information including the dimensions of the pallet, and the position determination unit may determine the position of the pallet by matching the two-dimensional map created by the map creation unit or the two-dimensional map corrected by the map correction unit with the template stored in the memory unit.

[0009] In this configuration, the position of the pallet can be detected with even greater accuracy by matching the two-dimensional map with a template according to pallet information including the dimensions of the pallet.

[0010] (3) In (1) or (2) above, the determination unit determines, based on the two-dimensional map created by the map creation unit, whether there are unmeasured grid cells and no-measurement-point grid cells for which point cloud data has been acquired by the acquisition unit but no measurement points exist, and the map correction unit, when the determination unit determines that there are unmeasured grid cells, corrects the two-dimensional map created by the map creation unit by interpolating grid values ​​of the unmeasured grid cells, and when the determination unit determines that there are no-measurement-point grid cells, does not need to interpolate grid values ​​of the no-measurement-point grid cells.

[0011] In this configuration, even if point cloud data is acquired by the acquisition unit but there are measurement point-free grid cells where no measurement points exist, the grid values ​​of the measurement point-free grid cells are not interpolated, thereby obtaining a more appropriate 2D map and simplifying the detection process.

[0012] (4) In any of the above (1) to (3), the determination unit may determine whether there is an unmeasured grid cell in accordance with the sensor scan order based on the two-dimensional map created by the map creation unit, and the map correction unit may interpolate a grid value of the unmeasured grid cell using grid values ​​of grid cells adjacent to the unmeasured grid cell on the upstream and downstream sides of the unmeasured grid cell in the sensor scan order when the determination unit determines that there is an unmeasured grid cell.

[0013] In such a configuration, a more appropriate two-dimensional map can be obtained through simple calculations by interpolating the grid values ​​of the unmeasured grid cells using the grid values ​​of the grid cells adjacent to the unmeasured grid cells upstream and downstream in the sensor scan order.

[0014] (5) In any of the above (1) to (3), when the determination unit determines that there is an unmeasured grid cell, the map correction unit may interpolate a grid value of the unmeasured grid cell using grid values ​​of a plurality of grid cells arranged around the unmeasured grid cell.

[0015] In such a configuration, a more suitable two-dimensional map can be obtained through simple calculations by interpolating the grid value of the unmeasured grid cell using the grid values ​​of multiple grid cells located around the unmeasured grid cell.

[0016] (6) In any of (1) to (5) above, the plane estimation unit may create a histogram of normal vectors of a plane perpendicular to the ground based on the three-dimensional point cloud data, and estimate a plane corresponding to the front of the pallet using the histogram.

[0017] In this configuration, a histogram of the normal vectors of a plane perpendicular to the ground is created based on three-dimensional point cloud data, and the histogram is used to estimate the plane corresponding to the front of the pallet, thereby obtaining a plane with high estimation accuracy through simple calculations.

[0018] (7) In any of the above (1) to (6), the sensor may be a 3D laser sensor that measures the distance from the forklift to the pallet by irradiating a laser onto the pallet and receiving the reflected laser light.

[0019] In this configuration, the 3D laser sensor irradiates a laser beam onto the pallet to scan it, thereby reliably acquiring three-dimensional point cloud data including multiple measurement points reflected by the pallet. [Effects of the Invention]

[0020] According to the present invention, the position of a pallet can be detected with high accuracy while reducing costs and labor. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram showing the configuration of a travel control device equipped with a pallet detection device according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing a state in which pallets are loaded on the bed of a truck. [Figure 3]1A and 1B are diagrams illustrating examples of scanning patterns of a 3D laser sensor. [Figure 4] 10 is a flowchart showing the procedure of a pallet detection process executed by the controller. [Figure 5] FIG. 10 is a diagram illustrating a normal vector of a plane. [Figure 6] 10 is a graph showing an example of a histogram of normal vectors of a plane. [Figure 7] FIG. 10 is a diagram showing a state in which a grid is created on a plane corresponding to the front of the pallet, and a plurality of grid cells are set on the plane. [Figure 8] 5 is a flowchart showing details of the determination process procedure of step S107 shown in FIG. 4. [Figure 9] FIG. 10 is a diagram showing how the presence or absence of unmeasured grid cells is determined in a two-dimensional map according to the scanning order of a 3D laser sensor. [Figure 10] FIG. 10 is a diagram showing a state in which, when it is determined that there is an unmeasured grid cell in the two-dimensional map shown in FIG. 9, the grid value of the unmeasured grid cell is interpolated. [Figure 11] FIG. 10 illustrates an example of another technique for interpolating clid values ​​for unmeasured grid cells. DETAILED DESCRIPTION OF THE INVENTION

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0023] Figure 1 is a block diagram showing the configuration of a travel control device equipped with a pallet detection device according to one embodiment of the present invention. In Figure 1, the travel control device 1 is mounted on an automatically operated forklift 2. The travel control device 1 is a device that automatically drives the forklift 2 when it handles cargo.

[0024] For example, as shown in FIG. 2, when the forklift 2 is stopped next to the truck 3, the driving control device 1 detects a pallet 5 loaded on the loading platform 4 of the truck 3 and drives the forklift 2 to a loading position where the pallet 5 can be loaded.

[0025] The pallet 5 is a loading platform on which the cargo M is placed. The pallet 5 is, for example, a flat pallet made of plastic (plastic pallet). The pallet 5 is provided with two pallet holes 6 into which a pair of left and right forks (not shown) of the forklift 2 are inserted.

[0026] The driving control device 1 includes a 3D laser sensor 7, a storage unit 8, a drive unit 9, and a controller 10.

[0027] The 3D laser sensor 7 is a sensor that emits a laser toward the pallet 5 and receives the reflected laser light to measure the distance from the forklift 2 to the pallet 5 and output three-dimensional point cloud data. The point cloud is a collection of laser reflection points (measurement points). For example, a LIDAR or the like is used as the 3D laser sensor 7.

[0028] The 3D laser sensor 7 scans in a pattern where the density near the center is higher than the density around the periphery, as shown in Figure 3. This scanning pattern is designed so that the area covered by the laser in space becomes more uniform over time.

[0029] The storage unit 8 stores a template (reference image) according to palette information including the dimensions of the palette 5. The palette information may include the shape of the palette 5 in addition to the dimensions of the palette 5. The template is created in advance based on the palette information.

[0030] Although not shown, the drive unit 9 has a travel motor that drives the forklift 2 and a steering motor that steers the forklift 2.

[0031] The controller 10 is composed of a CPU, RAM, ROM, an input / output interface, etc. The controller 10 has a point cloud acquisition unit 11, a plane estimation unit 12, a grid setting unit 13, a map creation unit 14, a determination unit 15, a map correction unit 16, a matching unit 17, a position calculation unit 18, and a driving control unit 19.

[0032] The point cloud acquisition unit 11 acquires point cloud data (see FIG. 5) from the 3D laser sensor 7. The point cloud acquisition unit 11 cooperates with the 3D laser sensor 7 to configure an acquisition unit that acquires three-dimensional point cloud data.

[0033] The plane estimation unit 12 estimates a plane corresponding to the front surface 5a (see FIG. 2) of the pallet 5 based on the three-dimensional point cloud data acquired by the point cloud acquisition unit 11. The plane estimation unit 12 creates a histogram of normal vectors of a plane perpendicular to the ground based on the three-dimensional point cloud data (see FIGS. 5 and 6), and estimates the plane corresponding to the front surface 5a of the pallet 5 using the histogram.

[0034] The grid setting unit 13 creates a grid on the plane corresponding to the front surface 5a of the pallet 5 estimated by the plane estimation unit 12, thereby setting a plurality of grid cells on the plane (see FIG. 7).

[0035] The map creation unit 14 creates a two-dimensional map (see Figure 9) in which the distance from the plane estimated by the plane estimation unit 12 to the measurement points of the point cloud data acquired by the point cloud acquisition unit 11 is calculated as grid values ​​for multiple grid cells set by the grid setting unit 13.

[0036] Based on the two-dimensional map created by the map creation unit 14, the determination unit 15 determines whether there are any unmeasured grid cells (see grid cell Sx in Figure 9) among the multiple grid cells, where point cloud data has not been acquired by the point cloud acquisition unit 11 and data is missing for measurement points, and any measurement point-less grid cells (see grid cells S2 and S3 in Figure 9) where point cloud data has been acquired by the point cloud acquisition unit 11 but no measurement points exist.

[0037] The determination unit 15 determines whether there are any unmeasured grid cells and any grid cells without measurement points in accordance with the scanning order of the 3D laser sensor 7, based on the two-dimensional map created by the map creation unit .

[0038] When the determination unit 15 determines that there are unmeasured grid cells, the map correction unit 16 corrects the two-dimensional map created by the map creation unit 14 by interpolating the grid values ​​of the unmeasured grid cells (see Figure 10).

[0039] Specifically, when determination unit 15 determines that there are unmeasured grid cells, map correction unit 16 corrects the two-dimensional map created by map creation unit 14 by interpolating the grid values ​​of the unmeasured grid cells, and when determination unit 15 determines that there are grid cells with no measurement points, map correction unit 16 does not interpolate the grid values ​​of the grid cells with no measurement points.

[0040] Furthermore, when the determination unit 15 determines that there is an unmeasured grid cell, the map correction unit 16 interpolates the grid value of the unmeasured grid cell using the grid values ​​of adjacent grid cells on the upstream and downstream sides of the unmeasured grid cell in the scanning order of the 3D laser sensor 7.

[0041] The matching unit 17 matches (collates) the two-dimensional map created by the map creating unit 14 or the two-dimensional map corrected by the map correcting unit 16 with the template stored in the storage unit 8.

[0042] The position calculation unit 18 calculates the position of the pallet 5 based on the matching result by the matching unit 17 .

[0043] The matching unit 17 and the position calculation unit 18 constitute a position determination unit that determines the position of the pallet 5 based on the two-dimensional map created by the map creation unit 14 or the two-dimensional map corrected by the map correction unit 16.

[0044] The travel control unit 19 generates a travel route to the loading and unloading position (described above) based on the position of the pallet 5 calculated by the position calculation unit 18, and controls the drive unit 9 so that the forklift 2 travels along the travel route.

[0045] Here, the 3D laser sensor 7, the memory unit 8, the point cloud acquisition unit 11, the plane estimation unit 12, the grid setting unit 13, the map creation unit 14, the determination unit 15, the map correction unit 16, the matching unit 17, and the position calculation unit 18 of the controller 10 constitute a pallet detection device 20 of this embodiment. The pallet detection device 20 is a device that detects a pallet 5 when loading and unloading is performed by the forklift 2.

[0046] 4 is a flowchart showing the procedure of a pallet detection process executed by the controller 10. This process is executed when the forklift 2 reaches the side of the truck 3 and an instruction is given to start a load-taking operation.

[0047] 4, the controller 10 first acquires point cloud data of the 3D laser sensor 7 (step S101). Then, the controller 10 extracts a plane perpendicular to the ground based on the point cloud data of the 3D laser sensor 7 (step S102). At this time, as shown in FIG. 5, the plane F perpendicular to the ground is extracted using a method such as RANSAC (Random Sample Consensus) or a combination of region growing and eigenvalue analysis of a normal tensor. Here, the plane F perpendicular to the ground includes not only a plane F that is completely perpendicular to the ground, but also a main plane F that is nearly perpendicular to the ground.

[0048] 5 and 6, the controller 10 creates a histogram HG of the normal vector Vf of the plane F (step S103). The horizontal axis of the histogram HG represents the yaw angle of the normal vector Vf, and the vertical axis of the histogram HG represents the frequency of the normal vector Vf.

[0049] Next, the controller 10 estimates a plane Fp corresponding to the front surface 5a of the pallet 5 using the histogram HG of the normal vector Vf of the plane F (step S104). At this time, the controller 10 estimates, for example, the plane F having the highest frequency of the normal vector Vf and including a greater number of measurement points in the point cloud than a specified number, as the plane Fp corresponding to the front surface 5a of the pallet 5 (see FIG. 5).

[0050] 7, the controller 10 orthogonally projects the point cloud perpendicularly onto a plane Fp corresponding to the front surface 5a of the pallet 5, and creates a grid G ​​(grid-like lines) on the plane Fp, thereby setting a plurality of grid cells Gs on the plane Fp (step S105). The grid cells Gs are square in shape, and the dimensions of the grid cells Gs are, for example, several mm or several cm.

[0051] Next, the controller 10 creates a two-dimensional map Mg (see FIG. 9), which is a two-dimensional image in which the distance from the plane Fp corresponding to the front surface 5a of the pallet 5 to the measurement point P is calculated as a grid value for each of the plurality of grid cells Gs (step S106). The grid value is a value obtained by converting the coordinate of the distance from the plane Fp corresponding to the front surface 5a of the pallet 5 to the measurement point P in the vertical direction (depth direction) of the plane Fp. The grid value of the two-dimensional map is expressed, for example, by brightness (light and shade). Specifically, the grid value is set so that the shorter the distance from the plane Fp to the measurement point P, the brighter the grid value.

[0052] Next, the controller 10 determines, based on the two-dimensional map Mg, whether there are any unmeasured grid cells Gs1 and any grid cells Gs2 with no measurement points (see Figure 9) among the multiple grid cells Gs in accordance with the scanning order of the 3D laser sensor 7 (step S107).

[0053] An unmeasured grid cell Gs1 is a grid cell Gs for which no point cloud data was acquired by the 3D laser sensor 7, resulting in a missing (skipped) measurement point. A no-measurement-point grid cell Gs2 is a grid cell Gs for which no measurement point exists, but for which point cloud data was acquired by the 3D laser sensor 7. The no-measurement-point grid cells Gs2 include not only grid cells Gs with no measurement points at all, but also grid cells Gs with few existing measurement points.

[0054] Fig. 8 is a flowchart showing the details of the determination processing procedure of step S107. In Fig. 8, the controller 10 first determines whether the number of measurement points P present in the currently scanned grid cell Gs (current grid cell Gs) is equal to or greater than a predetermined number based on the point cloud data of the 3D laser sensor 7 (step S201). The predetermined number is a threshold value for determining whether the point is the front surface 5a of the pallet 5 or a pallet hole 6 of the pallet 5.

[0055] When the controller 10 determines that the number of measurement points P present in the current grid cell Gs is equal to or greater than a specified number, it determines whether the number of measurement points P present in the previously scanned grid cell Gs (previous grid cell Gs) is equal to or greater than a specified number (step S202).

[0056] When the controller 10 determines that the number of measurement points P present in the previous grid cell Gs is equal to or greater than the specified number, it determines whether the current grid cell Gs is adjacent to the previous grid cell Gs in the scan order of the 3D laser sensor 7 (step S203). As shown in Fig. 7, there are eight adjacent grid cells Gs around one grid cell Gs.

[0057] When the controller 10 determines that the current grid cell Gs is not adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7, it determines that the grid cell Gs located between the current grid cell Gs and the previous grid cell Gs is an unmeasured grid cell Gs1 (step S204).

[0058] The controller 10 does not execute the above step S204 when it determines that the current grid cell Gs is adjacent to the previous grid cell Gs in the scan order of the 3D laser sensor 7. In other words, it is determined that there is no unmeasured grid cell Gs1 between the current grid cell Gs and the previous grid cell Gs.

[0059] When the controller 10 determines in step S202 that the number of measurement points P present in the previous grid cell Gs is less than the specified number, it determines whether the current grid cell Gs is adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7 (step S205).

[0060] When the controller 10 determines that the current grid cell Gs is adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7, it determines that the previous grid cell Gs is a grid cell Gs2 with no measurement point (step S206).When the controller 10 determines that the current grid cell Gs is not adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7, it is impossible to make a determination, and therefore step S206 is not executed.

[0061] When the controller 10 determines in step S201 that the number of measurement points P present in the current grid cell Gs is less than the specified number, it determines whether the number of measurement points P present in the previous grid cell Gs is equal to or greater than the specified number (step S207).

[0062] When the controller 10 determines that the number of measurement points P present in the previous grid cell Gs is equal to or greater than a specified number, it determines whether the current grid cell Gs is adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7 (step S208).

[0063] When the controller 10 determines that the current grid cell Gs is adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7, it determines that the current grid cell Gs is a measurement point-free grid cell Gs2 (step S209). When the controller 10 determines that the current grid cell Gs is not adjacent to the previous grid cell Gs in the scanning order of the 3D laser sensor 7, it is impossible to make a determination, and therefore step S209 is not executed.

[0064] If the controller 10 determines in step S207 that the number of measurement points P present in the previous grid cell Gs is less than the specified number, it is impossible to make a determination, and therefore does not execute steps S204, S206, and S209.

[0065] 4, after executing step S107, the controller 10 determines whether or not an unmeasured grid cell Gs1 exists in the two-dimensional map Mg (step S108). When the controller 10 determines that an unmeasured grid cell Gs1 exists in the two-dimensional map Mg, the controller 10 corrects the two-dimensional map Mg by interpolating the grid value of the unmeasured grid cell Gs1 (step S109).

[0066] At this time, the controller 10 interpolates the grid value of the unmeasured grid cell Gs1 using the grid values ​​of the grid cells Gs adjacent to the unmeasured grid cell Gs1 on the upstream and downstream sides in the scanning order of the 3D laser sensor 7 (see grid cell Sx in FIGS. 9 and 10). Specifically, the controller 10 sets, for example, the average value, median value, or the like of the grid values ​​of the grid cells Gs adjacent to the unmeasured grid cell Gs1 on the upstream and downstream sides in the scanning order of the 3D laser sensor 7 as the grid value of the unmeasured grid cell Gs1.

[0067] Next, the controller 10 performs template matching between the two-dimensional map Mgc (see FIG. 10) corrected in step S109 and the template stored in the storage unit 8, and selects grid cells Gs that correspond to the palette 5 (step S110). At this time, the controller 10 selects grid cells Gs in the two-dimensional map Mgc that closely match the template as grid cells Gs that correspond to the palette 5.

[0068] When the controller 10 determines that there is no unmeasured grid cell Gs1 in the two-dimensional map Mg, it performs template matching between the two-dimensional map Mg created in step S106 and the template stored in the storage unit 8, and selects a grid cell Gs corresponding to the palette 5 (step S111). The method of selecting the grid cell Gs corresponding to the palette 5 is the same as in step S110.

[0069] After executing step S110 or step S111, the controller 10 inversely transforms the position of the grid cell Gs corresponding to the pallet 5 into the coordinate system of the original point cloud data, thereby calculating the position of the pallet 5 (step S112). The coordinate system of the original point cloud data is the coordinate system of the point cloud data relative to the 3D laser sensor 7. Note that the position of the grid cell Gs corresponding to the pallet 5 may be the position of a template that matches the grid cell Gs corresponding to the pallet 5.

[0070] Here, the point cloud acquisition unit 11 executes step S101. The plane estimation unit 12 executes steps S102 to S104. The grid setting unit 13 executes step S105. The map creation unit 14 executes step S106. The determination unit 15 executes steps S107 and S108. The map correction unit 16 executes step S109. The matching unit 17 executes steps S110 and S111. The position calculation unit 18 executes step S112.

[0071] In the pallet detection device 20 as described above, a two-dimensional map Mg (two-dimensional image) is created based on the three-dimensional point cloud data acquired by measurement using the 3D laser sensor 7, with the distance from the plane Fp corresponding to the front surface 5a of the pallet 5 to the measurement point P as a grid value. Then, as shown in FIG. 9 , it is determined whether or not there is an unmeasured grid cell Gs1 in the two-dimensional map Mg according to the scanning order of the 3D laser sensor 7.

[0072] Specifically, first, point cloud data is acquired for grid cell S1. In grid cell S1, measurement point P is detected by the 3D laser sensor 7. Next, point cloud data is acquired for grid cell S2 adjacent to grid cell S1. In grid cell S2, measurement point P is not detected by the 3D laser sensor 7. Therefore, grid cell S1 is determined to be grid cell Gs corresponding to the front surface 5a of the pallet 5. Grid cell S2 is a grid cell Gs2 with no measurement point, and is determined to be grid cell Gs corresponding to the pallet hole 6.

[0073] Next, point cloud data is acquired for grid cell S3 adjacent to grid cell S2. In grid cell S3, measurement point P is not detected by the 3D laser sensor 7. Next, point cloud data is acquired for grid cell S4 adjacent to grid cell S3. In grid cell S4, measurement point P is detected by the 3D laser sensor 7. Therefore, grid cell S3 is determined to be a grid cell Gs2 with no measurement point, and is the grid cell Gs corresponding to the pallet hole 6. Grid cell S4 is determined to be the grid cell Gs corresponding to the front surface 5a of the pallet 5.

[0074] Next, point cloud data is acquired for grid cell S5, which is adjacent to grid cell S4. In grid cell S5, measurement point P is detected by the 3D laser sensor 7. Next, point cloud data is acquired for grid cell S6. In grid cell S6, measurement point P is detected by the 3D laser sensor 7. However, grid cell Sx between grid cell S5 and grid cell S6 is an unmeasured grid cell Gs1 in which measurement point P has not been detected by the 3D laser sensor 7. In other words, a data gap (skipping) has occurred for measurement point P in grid cell Sx.

[0075] If the scan of the 3D laser sensor 7 is rough, even if the laser from the 3D laser sensor 7 actually hits the front surface 5a of the pallet 5, the measurement point P may not be detected by the 3D laser sensor 7, resulting in missing data for the measurement point P.

[0076] 10, the grid value of grid cell Sx, which is an unmeasured grid cell Gs1, is interpolated using the grid values ​​of grid cells S5 and S6 that are adjacent to grid cell Sx on the upstream and downstream sides in the scanning order. The grid value of grid cell Sx is interpolated using, for example, the average value of the grid values ​​of grid cells S5 and S6. Therefore, grid cell Sx becomes grid cell Gs that corresponds to the front surface 5a of pallet 5.

[0077] Next, point cloud data is acquired for grid cell S7 adjacent to grid cell S6. In grid cell S7, measurement point P is detected by the 3D laser sensor 7. Next, point cloud data is acquired for grid cell S8 adjacent to grid cell S7. In grid cell S8, measurement point P is not detected by the 3D laser sensor 7. Therefore, grid cell S7 is determined to be grid cell Gs corresponding to the front surface 5a of the pallet 5. Grid cell S8 is determined to be grid cell Gs2 with no measurement point.

[0078] In this way, a grid cell Gs in which no measurement point P exists is determined to be either a no-measurement-point grid cell Gs2 corresponding to a pallet hole 6 in the pallet 5, or an unmeasured grid cell Gs1 that corresponds to the front surface 5a of the pallet 5 but has not been measured simply because the scan was coarse. If the grid cell Gs in which no measurement point P exists is an unmeasured grid cell Gs1, the grid values ​​of the unmeasured grid cell Gs1 are interpolated to correct the two-dimensional map Mg to a two-dimensional map Mgc. The corrected two-dimensional map Mgc is then matched with the template to calculate the position of the pallet 5.

[0079] Incidentally, in loading and unloading operations using unmanned forklifts, the position of a pallet is detected as a bounding box using a camera, and the orientation of the pallet is measured using LiDAR. Pallet detection using a camera is performed using machine learning such as a deep neural network (DNN), so training images of many pallets, including the pallet being handled, are required. In addition, it is necessary to install both LiDAR to acquire 3D point cloud data and a camera to acquire 2D image data. This increases costs.

[0080] When detecting pallets using LiDAR and DNN, the processing load is heavy because the point cloud is distributed in three dimensions. Also, since there is no training data for pallets, creating the training data is a significant effort.

[0081] Unlike images, 3D measurements essentially provide values ​​viewed from the front at the correct scale, making them detectable by detectors that lack robustness in terms of posture, scale, color brightness, etc. For this reason, it is thought that detectors can be constructed using only dimensional data from CAD, etc., without the need to create large amounts of training data.

[0082] In addition, there are many examples of acquiring depth images from point clouds of depth cameras, but depth cameras are vulnerable to ambient light, have a narrow depth measurement range, and are low in accuracy. Unlike depth cameras, LiDAR is resistant to ambient light, but because it acquires 3D point cloud data by scanning, the density of the point cloud is generally low and there are variations in density.

[0083] To address this issue, in this embodiment, the 3D laser sensor 7 measures the distance from the forklift 2 to the pallet 5, and three-dimensional point cloud data including multiple measurement points P is acquired. Then, based on the three-dimensional point cloud data, a plane Fp corresponding to the front surface 5a of the pallet 5 is estimated. A grid G ​​is created on the plane Fp, and multiple grid cells Gs are set on the plane Fp. A two-dimensional map Mg is then created in which the distances from the plane Fp to the measurement points P for the multiple grid cells Gs are calculated as grid values. Based on the two-dimensional map Mg, it is determined whether there is an unmeasured grid cell Gs1 among the multiple grid cells Gs, where point cloud data was not acquired by the 3D laser sensor 7 and the point cloud acquisition unit 11, resulting in a missing data point for the measurement point P. If it is determined that there is an unmeasured grid cell Gs1, the two-dimensional map Mg is corrected by interpolating the grid value of the unmeasured grid cell Gs1. The position of the pallet 5 is then determined based on the created two-dimensional map Mg or the corrected two-dimensional map Mgc. In this way, by using the 3D laser sensor 7 to acquire three-dimensional point cloud data, a camera or the like for acquiring two-dimensional image data is not required. Furthermore, by creating a two-dimensional map Mg, the position of the pallet 5 can be detected without creating a large amount of training data. This reduces costs and effort. Furthermore, when an unmeasured grid cell Gs1 exists, an appropriate two-dimensional map Mgc can be obtained by interpolating the grid values ​​of the unmeasured grid cell Gs1. Therefore, the position of the pallet 5 can be detected with high accuracy.

[0084] Furthermore, in this embodiment, by matching the two-dimensional map Mg or two-dimensional map Mgc with a template according to pallet information including the dimensions of the pallet 5, the position of the pallet 5 can be detected with even greater accuracy.

[0085] Furthermore, in this embodiment, even if point cloud data is acquired by the 3D laser sensor 7 and the point cloud acquisition unit 11 but there is a no-measurement-point grid cell Gs2 in which no measurement point P exists, the grid values ​​of the no-measurement-point grid cell Gs2 are not interpolated. Therefore, a more appropriate two-dimensional map Mgc can be obtained and the detection process can be simplified.

[0086] Furthermore, in this embodiment, the grid values ​​of the unmeasured grid cell Gs1 are interpolated using the grid values ​​of the grid cells Gs adjacent to the unmeasured grid cell Gs1 on the upstream and downstream sides in the scanning order of the 3D laser sensor 7, thereby obtaining a more appropriate two-dimensional map Mgc through simple calculations.

[0087] In addition, in this embodiment, a histogram HG of the normal vector Vf of a plane F perpendicular to the ground is created based on three-dimensional point cloud data, and the histogram HG is used to estimate a plane Fp corresponding to the front surface 5a of the pallet 5, thereby enabling a highly accurate estimation of the plane Fp to be obtained through simple calculations.

[0088] Furthermore, in this embodiment, the 3D laser sensor 7 emits a laser beam toward the pallet 5 to scan it, so that three-dimensional point cloud data including a plurality of measurement points P reflected by the pallet 5 is reliably acquired.

[0089] 11 is a diagram showing an example of another method for interpolating the grid value of an unmeasured grid cell by the map corrector 16. When the determiner 15 determines that there is an unmeasured grid cell, the map corrector 16 interpolates the grid value of the unmeasured grid cell using the grid values ​​of multiple grid cells arranged around the unmeasured grid cell.

[0090] 11, unmeasured grid cells a1 to a12 are surrounded by grid cells S1 to S24, each having a grid value that is depth information (distance information). Unmeasured grid cells a1 to a12 are interpolated using the average value of the grid values ​​of eight adjacent grid cells Gs that have a grid value. Specifically, the grid values ​​of unmeasured grid cells a1 to a12 are calculated as follows:

[0091] a1=(S1+S2+S3+S7+S10) / 5 a2=(S2+S3+S4+S8+a1) / 5 a6=(a2+S8) / 2 a3=(a6+S8+S4+S5+S6+S9+S11) / 7 a7=(a6+S8+a3+S9+S11+S14) / 6 a10=(a6+a7+S11+S14+S18+S17) / 6 a12=(S21+a10+S17+S23+S22) / 5 a11=(S20+S16+S13+a12+S22+S21) / 6 a8=(S16+S13+a6+a12+a11) / 5 a4=(S12+S10+S7+a1+a2+S13) / 6 a5=(S13+a4+a1+a2+S8+a6+a8) / 7 a9=(a11+a8+a5+a6+a7+a10+S17+a12) / 8

[0092] The present invention is not limited to the above embodiment. For example, in the above embodiment, the position of the pallet 5 is detected by matching the two-dimensional map Mg or the two-dimensional map Mgc with a template, but the orientation of the pallet 5 may also be detected in addition to the position of the pallet 5.

[0093] Furthermore, in the above embodiment, the pallet 5 is detected when the forklift 2 is stopped, but this is not a particular limitation, and the pallet 5 may be detected when the forklift 2 is moving. In this case, it is necessary to correct the amount of movement of the 3D laser sensor 7 using an inertial measurement unit (IMU), an odometer, or the like.

[0094] Furthermore, in the above embodiment, the grid value of the unmeasured grid cell Gs1 is interpolated using the grid values ​​of the grid cells Gs adjacent to the unmeasured grid cell Gs1 on the upstream and downstream sides in the scanning order of the 3D laser sensor 7, or the grid value of the unmeasured grid cell Gs1 is interpolated using the grid values ​​of multiple grid cells Gs arranged around the unmeasured grid cell Gs1, but this is not particularly limited to such a form. For example, the grid value of the unmeasured grid cell Gs1 may be interpolated using the grid values ​​of one or more grid cells Gs positioned in the vicinity of the unmeasured grid cell Gs1.

[0095] Furthermore, in the above embodiment, the pallet 5 is a plastic pallet, but the pallet 5 to be detected is not particularly limited to a plastic pallet, and may be a post pallet, a wooden pallet, or the like.

[0096] Furthermore, in the above embodiment, three-dimensional point cloud data is acquired by the 3D laser sensor 7, but the sensor that acquires the three-dimensional point cloud data is not limited to the 3D laser sensor 7, and for example, a camera or the like that can detect objects in three dimensions may be used. [Explanation of symbols]

[0097] 2...forklift, 5...pallet, 5a...front, 7...3D laser sensor (sensor, acquisition unit), 8...memory unit, 11...point cloud acquisition unit (acquisition unit), 12...plane estimation unit, 13...grid setting unit, 14...map creation unit, 15...determination unit, 16...map correction unit, 17...matching unit (position determination unit), 18...position calculation unit (position determination unit), 20...pallet detection device, P...measurement point, F...plane, Fp...plane, Vf...normal vector, HG...histogram, G...grid, Gs...grid cell, Mg, Mgc...2D map, Gs1...unmeasured grid cell, Gs2...grid cell with no measurement point, a1 to a12...unmeasured grid cells, S1 to S24...grid cells.

Claims

1. A pallet detection device that detects pallets when handling cargo using a forklift, an acquisition unit having a sensor that measures the distance from the forklift to the pallet and that acquires three-dimensional point cloud data including a plurality of measurement points; a plane estimation unit that estimates a plane corresponding to a front surface of the pallet based on the three-dimensional point cloud data acquired by the acquisition unit; a grid setting unit that sets a plurality of grid cells on the plane estimated by the plane estimation unit by creating a grid on the plane; a map creation unit that creates a two-dimensional map in which distances from the plane estimated by the plane estimation unit to the measurement points acquired by the acquisition unit are calculated as grid values ​​for the plurality of grid cells set by the grid setting unit; and a determination unit that determines, based on the two-dimensional map created by the map creation unit, whether or not there is an unmeasured grid cell among the plurality of grid cells where the point cloud data has not been acquired by the acquisition unit and data loss has occurred at the measurement point; and a map correction unit that corrects the two-dimensional map created by the map creation unit by interpolating the clid values ​​of the unmeasured grid cells when the determination unit determines that there are unmeasured grid cells; and a position determination unit that determines the position of the pallet based on the two-dimensional map created by the map creation unit or the two-dimensional map corrected by the map correction unit.

2. a storage unit that stores a template according to pallet information including dimensions of the pallet; 2. The pallet detection device according to claim 1, wherein the position determination unit determines the position of the pallet by matching the two-dimensional map created by the map creation unit or the two-dimensional map corrected by the map correction unit with a template stored in the memory unit.

3. the determination unit determines, based on the two-dimensional map created by the map creation unit, whether or not there are any unmeasured grid cells and any measurement point-free grid cells for which the point cloud data has been acquired by the acquisition unit but no measurement points exist; 2. The pallet detection device according to claim 1, wherein, when the determination unit determines that there is an unmeasured grid cell, the map correction unit corrects the two-dimensional map created by the map creation unit by interpolating grid values ​​of the unmeasured grid cells, and when the determination unit determines that there is a grid cell with no measurement point, the map correction unit does not interpolate grid values ​​of the grid cell with no measurement point.

4. the determination unit determines whether there are any unmeasured grid cells in accordance with a scanning order of the sensor based on the two-dimensional map created by the map creation unit; 2. The pallet detection device according to claim 1, wherein, when the determination unit determines that there is an unmeasured grid cell, the map correction unit interpolates the grid value of the unmeasured grid cell using grid values ​​of grid cells adjacent to the unmeasured grid cell on the upstream and downstream sides in the scanning order of the sensor.

5. 2. The pallet detection device according to claim 1, wherein, when the determination unit determines that there is an unmeasured grid cell, the map correction unit interpolates the grid value of the unmeasured grid cell using grid values ​​of a plurality of grid cells arranged around the unmeasured grid cell.

6. The pallet detection device according to claim 1, wherein the plane estimation unit creates a histogram of normal vectors of a plane perpendicular to the ground based on the three-dimensional point cloud data, and estimates a plane corresponding to the front of the pallet using the histogram.

7. 2. The pallet detection device according to claim 1, wherein the sensor is a 3D laser sensor that measures the distance from the forklift to the pallet by irradiating a laser beam toward the pallet and receiving the reflected laser light.

Images