Palette Detection Device

Abstract

To provide a pallet detecting apparatus capable of stably detecting a pallet irrespective of illuminance conditions of the surroundings of the pallet.SOLUTION: A pallet detecting apparatus 20 comprises a camera 12 that photographs a pallet 5, a storage part 13 that determines a fixed value of exposure of the camera 12 according to usage environment of the camera 12, a photographing control part 23 that controls the camera 12 so as to photograph the pallet 5 with the fixed value of exposure determined by the storage part 13, and a pallet recognizing part 24 that recognizes the pallet 5 on the basis of image data obtained by photographing the pallet 5 with the fixed value of exposure by the camera 12.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a pallet detection device.

Background Art

[0002] In the automatic driving of a forklift or the like, a pallet on which a load is placed may be detected using a camera. Generally, in a detection technique using a camera, a large change in illuminance becomes a major problem. For example, Patent Document 1 describes a technique of predicting a point where an illuminance change equal to or greater than a threshold value is expected on the traveling route of a host vehicle and controlling a camera based on the predicted illuminance change.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When a camera has a function of automatically adjusting exposure, it is possible to adjust the brightness by that function to capture an object clearly. However, when the surroundings are too bright or too dark with respect to the pallet, the exposure of the camera is adjusted according to the illuminance around the pallet, so that the illuminance does not match the pallet itself, and white bleeding or black crushing may occur in the area where the pallet appears in the image data. Further, when the pallet is far from the camera, the occupancy rate of the pallet in the image data becomes small, so that the exposure of the camera is easily adjusted according to the illuminance of the portion other than the pallet, and the area where the pallet appears in the image data tends to become dark. In such image data, even when image processing using AI technology or the like is performed, pallet detection often does not work well.

[0005] An object of the present invention is to provide a pallet detection device capable of stably detecting a pallet regardless of the illuminance conditions around the pallet.

Means for Solving the Problems

[0006] One aspect of the present invention is a pallet detection device for detecting a pallet used when performing a loading and unloading operation by an industrial vehicle, including a camera mounted on the industrial vehicle for photographing the pallet, an exposure determination unit for determining a fixed value of the exposure of the camera according to the use environment of the camera, a photographing control unit for controlling the camera to photograph the pallet with the fixed value of the exposure determined by the exposure determination unit, and a pallet recognition unit for recognizing the pallet based on the image data obtained by photographing the pallet with the fixed value of the exposure by the camera.

[0007] In such a pallet detection device, a fixed value of the exposure of the camera is determined according to the use environment of the camera, and the camera is controlled to photograph the pallet with the fixed value of the exposure. Then, the pallet is recognized based on the image data obtained by photographing the pallet with the fixed value of the exposure by the camera. By photographing the pallet with the fixed value of the exposure according to the use environment of the camera in this way, image data in which the pallet is clearly imaged can be obtained regardless of the illuminance around the pallet. Thereby, the pallet is stably detected regardless of the illuminance conditions around the pallet.

[0008] The exposure determination unit may determine a fixed value of the shutter speed of the camera as the fixed value of the exposure of the camera, and the photographing control unit may control the camera to photograph the pallet with the fixed value of the shutter speed determined by the exposure determination unit. In such a configuration, by determining the fixed value of the shutter speed of the camera, the fixed value of the exposure of the camera is determined without particularly driving a motor or the like. Therefore, the fixed value of the exposure of the camera can be determined without being affected by the vibration of the camera.

[0009] The imaging control unit controls the camera to capture an image of the pallet at the fixed values of a plurality of exposures determined by the exposure determination unit, and the pallet recognition unit may recognize the pallet based on a plurality of image data obtained by capturing the pallet by the camera at the fixed values of the plurality of exposures. In such a configuration, fixed values of a plurality of exposures corresponding to the usage environment of the camera are determined. Then, based on a plurality of image data obtained by capturing the pallet by the camera at the fixed values of the plurality of exposures, the pallet is recognized. Therefore, even when the camera is used in a plurality of different environments, the pallet can be stably detected regardless of the illuminance condition around the pallet.

[0010] The pallet recognition unit may synthesize a plurality of image data and recognize the pallet from the synthesized image data. In such a configuration, by synthesizing a plurality of image data obtained by capturing the pallet by the camera at the fixed values of a plurality of exposures, even when the camera is used in a plurality of different environments, image data in which the pallet is clearly imaged can be obtained, so that it becomes easier to recognize the pallet.

[0011] The imaging control unit may control the camera to capture an image of the pallet at the fixed exposure value determined by the exposure determination unit while the industrial vehicle is stopped. In such a configuration, since the pallet is captured by the camera while the industrial vehicle is stopped, the image data is less likely to be blurred.

Advantages of the Invention

[0012] According to the present invention, the pallet can be stably detected regardless of the illuminance condition around the pallet.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or equivalent elements are denoted by the same reference numerals, and redundant descriptions are omitted.

[0015] FIG. 1 is a schematic plan view showing a forklift equipped with a pallet detection device according to an embodiment of the present invention together with a pallet. In FIG. 1, the forklift 1 is one of industrial vehicles for carrying out cargo handling. The forklift 1 includes a vehicle body 2 and a cargo handling device 3 disposed on the front side of the vehicle body 2 for carrying out cargo handling. The cargo handling device 3 has a mast 4 attached to the front end of the vehicle body 2 and a pair of left and right forks 6 attached to the mast 4 so as to be liftable and for lifting the pallet 5.

[0016] The pallet 5 is a cargo handling platform for loading goods. The pallet 5 is, for example, a flat pallet. The pallet 5 has a rectangular shape in plan view. The pallet 5 is provided with two pallet holes 7 into which each fork 6 of the forklift 1 is inserted. These pallet holes 7 extend from the front surface 5a to the rear surface 5b of the pallet 5. The front surface 5a of the pallet 5 is the surface on the side where the forklift 1 is located. The shape of the pallet hole 7 is rectangular in front view (see FIG. 6).

[0017] Figure 2 is a block diagram showing a schematic configuration of a travel control device including a pallet detection device according to an embodiment of the present invention. In Figure 2, the travel control device 10 is a device that automatically travels the forklift 1 to the location where the pallet 5 is placed. The travel control device 10 is mounted on the forklift 1.

[0018] The travel control device 10 includes a laser sensor 11, a camera 12, a storage unit 13, a controller 14, and a drive unit 15.

[0019] The laser sensor 11 irradiates a laser toward the surroundings of the forklift 1 and receives the reflected light of the laser, thereby detecting the distance to an object existing in the surroundings of the forklift 1. As the laser sensor 11, for example, LIDAR, a laser rangefinder, or the like is used.

[0020] The camera 12 photographs the pallet 5 and acquires image data (a photograph). As the camera 12, for example, a monocular camera or the like is used. The camera 12 can adjust the exposure that determines the illuminance (brightness) of the image data by adjusting the shutter speed and the aperture. Exposure is the amount of light captured by the image sensor when the camera 12 takes a photograph.

[0021] The shutter speed is the time during which the shutter is open. When the shutter speed is increased, the time during which the shutter is open becomes shorter, and the time during which light hits the image sensor becomes shorter, resulting in a darker photograph. When the shutter speed is decreased, the time during which the shutter is open becomes longer, and the time during which light hits the image sensor becomes longer, resulting in a brighter photograph.

[0022] The aperture is the brightness of the image formed on the image sensor through the lens. When the aperture is increased, the image formed on the image sensor becomes darker, resulting in a darker photograph. When the aperture is decreased, the image formed on the image sensor becomes brighter, resulting in a brighter photograph.

[0023] The storage unit 13 stores, as table data, fixed values of the shutter speed according to the usage environment of the camera 12. The storage unit 13 constitutes an exposure determination unit that determines the fixed value of the shutter speed of the camera 12 as the fixed value of the exposure of the camera 12. When there are a plurality of usage environments of the camera 12, fixed values of a plurality of shutter speeds corresponding to the plurality of usage environments are respectively determined.

[0024] For example, when the camera 12 is used outdoors, fixed values of appropriate shutter speeds are respectively determined in outdoor front light, outdoor backlight, and dark places. When the camera 12 is used indoors, fixed values of appropriate shutter speeds are respectively determined in indoor front light, indoor backlight, and dark places. When the camera 12 is used both outdoors and indoors, fixed values of appropriate shutter speeds are respectively determined in outdoor front light, outdoor backlight, indoor front light, indoor backlight, and dark places.

[0025] The fixed value of the shutter speed suitable for the usage environment of the camera 12 is determined by, for example, a person in charge of the manufacturer. The person in charge of the manufacturer investigates the user's usage environment using an illuminance meter or the like, and stores the data of the fixed value of the shutter speed suitable for the usage environment in the storage unit 13.

[0026] The controller 14 is composed of a CPU, a RAM, a ROM, an input / output interface, and the like. The controller 14 inputs the detection data of the laser sensor 11 and the image data of the camera 12, executes predetermined processing, and controls the drive unit 15. The drive unit 15 has, for example, a travel motor that runs the forklift 1 and a steering motor that steers the forklift 1.

[0027] The controller 14 has a self-position estimation unit 21, a drive control unit 22, a shooting control unit 23, a pallet recognition unit 24, an estimation calculation unit 25, and a drive control unit 26.

[0028] Here, the shooting control unit 23 and the pallet recognition unit 24 cooperate with the camera 12 and the storage unit 13 to constitute the pallet detection device 20 of the present embodiment. The pallet detection device 20 is a device that detects the pallet 5 used when the forklift 1 performs a loading and unloading operation.

[0029] When the forklift 1 travels toward the pallet 5, the self-position estimation unit 21 estimates the self-position of the forklift 1 using the detection data of the laser sensor 11 and the map data.

[0030] FIG. 3 is a flowchart showing the procedure of the self-position estimation process executed by the self-position estimation unit 21. This process is executed when the start of the automatic driving of the forklift 1 is instructed.

[0031] In FIG. 3, the self-position estimation unit 21 first acquires the detection data of the laser sensor 11 (step S101). Then, the self-position estimation unit 21 performs an estimation calculation of the self-position of the forklift 1 using the laser SLAM (simultaneous localization and mapping) method (step S102). SLAM is a self-position estimation technology that performs self-position estimation using sensor data and map data. The self-position estimation unit 21 matches the detection data of the laser sensor 11 and the map data to perform an estimation calculation of the self-position of the forklift 1. Then, the self-position estimation unit 21 outputs the self-position data of the forklift 1 to the drive control unit 22 (step S103).

[0032] Subsequently, the self-position estimation unit 21 determines whether the forklift 1 has reached the shooting position based on the self-position of the forklift 1 (step S104). The shooting position is the position where the front surface 5a of the pallet 5 is photographed by the camera 12. The shooting position is a position separated from the front surface 5a of the pallet 5 by a specified distance. The specified distance is a distance such that the entire pallet 5 is included in the image data when the pallet 5 is photographed by the camera 12.

[0033] When the self-position estimation unit 21 determines that the forklift 1 has not reached the shooting position, it executes the above procedures S101 to S104 again. When the self-position estimation unit 21 determines that the forklift 1 has reached the shooting position, it outputs a stop instruction signal for stopping the forklift 1 to the drive control unit 22 (procedure S105). Then, the self-position estimation unit 21 outputs a shooting instruction signal for shooting the pallet 5 by the camera 12 to the shooting control unit 23 (procedure S106).

[0034] Note that the self-position estimation unit 21 estimates and calculates the self-position of the forklift 1 by the laser SLAM method using the detection data of the laser sensor 11, but it is not particularly limited to this form, and the self-position of the forklift 1 may be estimated using the Visual SLAM method, the RTK-GNSS (Realtime Kinematic-Global Navigation Satellite System) positioning method, or an odometry sensor, etc.

[0035] Returning to FIG. 2, the drive control unit 22 controls the drive unit 15 to automatically drive the forklift 1 to the shooting position based on the self-position of the forklift 1 estimated by the self-position estimation unit 21. Also, when the drive control unit 22 receives a stop instruction signal from the self-position estimation unit 21, it controls the drive unit 15 to stop the running of the forklift 1.

[0036] The shooting control unit 23 controls the camera 12 to shoot the pallet 5 with a fixed value of the shutter speed stored in the storage unit 13 in a state where the forklift 1 has stopped.

[0037] FIG. 4 is a flowchart showing the procedure of the shooting control process executed by the shooting control unit 23. In FIG. 4, the shooting control unit 23 first determines whether it has received a shooting instruction signal from the self-position estimation unit 21 (procedure S111). This procedure is repeatedly executed until it is determined that a shooting instruction signal from the self-position estimation unit 21 has been received.

[0038] When the photographing control unit 23 determines that it has received a photographing instruction signal from the self-position estimation unit 21, it reads the data of the fixed value of the shutter speed stored in the storage unit 13 (step S112). As described above, a plurality of fixed values of the shutter speed corresponding to a plurality of usage environments by the user are set in the storage unit 13.

[0039] Then, the photographing control unit 23 controls the camera 12 so as to continuously photograph the pallet 5 with the fixed values of the plurality of shutter speeds stored in the storage unit 13 (step S113). As a result, the pallet 5 is continuously photographed by the camera 12 at different shutter speeds.

[0040] Returning to FIG. 2, the pallet recognition unit 24 recognizes the pallet 5 based on a plurality of image data obtained by photographing the pallet 5 with the fixed values of a plurality of shutter speeds by the camera 12. The pallet recognition unit 24 synthesizes a plurality of image data obtained by photographing the pallet 5 with the fixed values of a plurality of shutter speeds by the camera 12, and recognizes the pallet 5 from the synthesized image data.

[0041] FIG. 5 is a flowchart showing the procedure of the pallet recognition process executed by the pallet recognition unit 24. In FIG. 5, the pallet recognition unit 24 first acquires a plurality of image data by the camera 12 (step S121). As a result, for example, as shown in FIG. 6, two pieces of image data D1 and D2 with different illuminances (brightness) are acquired.

[0042] Then, the pallet recognition unit 24 synthesizes a plurality of image data with different illuminances, for example, by an HDR (High Dynamic Range) function (step S122). By using the HDR function, beautiful photos can be obtained even in photography in backlight or dark places.

[0043] Subsequently, the pallet recognition unit 24 recognizes the pallet 5 from the synthesized image data (synthesized image data), for example, using AI (Artificial Intelligence) technology (step S123). As the AI technology, for example, an image processing technology using deep learning is used.

[0044] Returning to FIG. 2, the estimation calculation unit 25 uses deep learning or the like to perform an estimation calculation of the position and orientation of the pallet 5 recognized by the pallet recognition unit 24. For example, in the composite image data, the estimation calculation unit 25 designates a frame line surrounding the front surface 5a of the pallet 5 and the two pallet holes 7, and estimates the position and orientation of the pallet 5 with respect to the forklift 1. The position of the pallet 5 is the XYZ coordinates of the center position of the pallet 5 with respect to the forklift 1 (see FIG. 1). The orientation of the pallet 5 is the direction of the pallet 5 with respect to the forklift 1.

[0045] Based on the position and orientation of the pallet 5 estimated by the estimation calculation unit 25, the drive control unit 26 controls the drive unit 15 to automatically drive the forklift 1 to the loading / unloading position. The loading / unloading position is the position where each fork 6 of the forklift 1 is inserted into the pallet hole 7 of the pallet 5.

[0046] By the way, when the surroundings of the pallet 5 are too bright or too dark, if the shutter speed of the camera 12 is adjusted according to the illuminance around the pallet 5 by the automatic adjustment function of the camera 12, the illuminance does not match the pallet 5 itself, so white spots or black crushing may occur in the area where the pallet 5 appears in the image data. Also, when the pallet 5 is far from the camera 12, the occupancy rate of the pallet 5 in the image data is small, so the shutter speed of the camera 12 is easily adjusted according to the illuminance of the part other than the pallet 5, and the area where the pallet 5 appears in the image data tends to be dark. Thus, it is difficult to adjust the illuminance by the automatic adjustment function of the shutter speed.

[0047] Therefore, it is conceivable to improve the detection ability of the pallet 5 by AI technology. For example, there is a method of detecting the pallet 5 by having the AI learn images with white gaps and black crush. However, in such a method, it is possible to detect the pallet 5 with image data having few white gaps and black crush, but with image data having many white gaps and black crush, it may not be possible to learn well and the pallet 5 may not be detected. In addition, in order to detect the pallet 5, a huge amount of learning data with white gaps and black crush is required, which is quite inefficient.

[0048] To address such problems, in the present embodiment, a fixed value of the exposure of the camera 12 is determined according to the usage environment of the camera 12, and the camera 12 is controlled to capture the pallet 5 at the fixed value of the exposure. Then, based on the image data obtained by capturing the pallet 5 by the camera 12 at the fixed value of the exposure, the pallet 5 is recognized. By capturing the pallet 5 at the fixed value of the exposure according to the usage environment of the camera 12 in this way, image data in which the pallet 5 is clearly imaged can be obtained regardless of the illuminance around the pallet 5. As a result, the pallet 5 can be stably detected regardless of the illuminance conditions around the pallet 5. In addition, since a huge amount of learning data is not required, the detection of the pallet 5 can be performed efficiently.

[0049] In addition, in the present embodiment, by determining a fixed value of the shutter speed of the camera 12, the fixed value of the exposure of the camera 12 is determined without particularly driving a motor or the like. Therefore, the fixed value of the exposure of the camera 12 can be determined without being affected by the vibration of the camera 12.

[0050] In addition, in the present embodiment, a plurality of fixed values of the exposure according to the usage environment of the camera 12 are determined. Then, based on a plurality of image data obtained by capturing the pallet 5 by the camera 12 at the plurality of fixed values of the exposure, the pallet 5 is recognized. Therefore, even when the camera 12 is used in a plurality of different environments, the pallet 5 can be stably detected regardless of the illuminance conditions around the pallet 5.

[0051] In addition, in the present embodiment, by synthesizing a plurality of image data obtained by photographing the pallet 5 with the camera 12 at a plurality of fixed exposure values, even when the camera 12 is used in a plurality of different environments, a synthesized image data in which the pallet 5 is clearly imaged can be obtained, so that the recognition of the pallet 5 becomes easier.

[0052] In addition, in the present embodiment, since the pallet 5 is photographed by the camera 12 while the forklift 1 is stopped, the image data is less likely to blur even when the shutter speed is particularly slow. In addition, since the position of the pallet 5 on the plurality of image data does not change, it becomes easier to synthesize the plurality of image data, and accurate synthesized image data can be obtained.

[0053] FIG. 7 is a flowchart showing a modified example of the procedure of the pallet recognition process executed by the pallet recognition unit 24, and corresponds to FIG. 5. In FIG. 7, the pallet recognition unit 24 first acquires a plurality of image data by the camera 12 (step S121).

[0054] Then, the pallet recognition unit 24 recognizes the pallet 5 from the plurality of image data using, for example, AI technology (step S125). At this time, the pallet recognition unit 24 may recognize the pallet 5 from all the image data with different illuminances, or may recognize the pallet 5 from the image data in which the pallet 5 is most clearly imaged among the plurality of image data with different illuminances.

[0055] Also in such a modified example, when the camera 12 is used in a plurality of different environments, the pallet 5 is stably detected regardless of the illuminance conditions around the pallet 5.

[0056] Note that the present invention is not limited to the above embodiment. For example, in the above embodiment, as the fixed exposure value of the camera 12, the fixed shutter speed value of the camera 12 is determined, but it is not particularly limited to such a form, and the fixed aperture value of the camera 12 may be determined.

[0057] In the above embodiment, the storage unit 13 stores data of fixed values of a plurality of shutter speeds according to a plurality of usage environments of the camera 12. However, the form is not particularly limited thereto. When there is only one usage environment of the camera 12, fixed values of data of one shutter speed or aperture may be stored in the storage unit 13.

[0058] In the above embodiment, when the pallet 5 is photographed by the camera 12, the data of the fixed value of the shutter speed stored in the storage unit 13 is read as it is. However, the form is not particularly limited thereto. For example, when the pallet 5 is photographed by the camera 12, the user selects and inputs, via an input device, data of a fixed value of a shutter speed suitable for the usage environment of the camera 12 from among the data of the fixed values of the plurality of shutter speeds stored in the storage unit 13, and the pallet 5 may be photographed at the fixed value of the shutter speed. In this case, the input device constitutes an exposure determination unit that determines the fixed value of the exposure of the camera 12.

[0059] In the above embodiment, the pallet 5 is photographed by the camera 12 while the forklift 1 is stopped. However, the form is not particularly limited thereto, and the pallet 5 may be photographed by the camera 12 while the forklift 1 is traveling at a low speed or a very low speed toward the pallet 5.

[0060] In the above embodiment, while estimating the self-position of the forklift 1, the forklift 1 is automatically driven to the photographing position. However, the form is not particularly limited thereto. For example, after the forklift 1 is driven to the photographing position by manual operation, the pallet 5 may be photographed by the camera 12.

[0061] In the above embodiment, the pallet 5 having two pallet holes 7 into which the forks 6 of the forklift 1 are inserted is detected. However, the present invention is also applicable to the detection of a pallet without pallet holes. Further, the present invention is applicable to an industrial vehicle that uses a pallet when performing a handling operation, other than the forklift 1.

Explanation of Reference Numerals

[0062] 1... forklift (industrial vehicle), 5... pallet, 12... camera, 13... memory unit (exposure determination unit), 20... pallet detection device, 23... shooting control unit, 24... pallet recognition unit, D1, D2... image data.

Claims

1. A pallet detection device for detecting a pallet used when performing a handling operation by an industrial vehicle, a camera mounted on the industrial vehicle for photographing the pallet, an exposure determination unit that determines respective fixed values of a plurality of exposures of the camera according to a plurality of use environments of the camera, a photographing control unit that controls the camera to photograph the pallet with the plurality of fixed exposure values determined by the exposure determination unit, and a pallet recognition unit that synthesizes a plurality of image data obtained by photographing the pallet with the plurality of fixed exposure values by the camera once, and recognizes the pallet from the one synthesized image data. A pallet detection device comprising:

2. The exposure determination unit determines fixed values of a plurality of shutter speeds of the camera as the fixed values of the plurality of exposures of the camera, The photographing control unit controls the camera to photograph the pallet with the fixed values of the plurality of shutter speeds determined by the exposure determination unit. The pallet detection device according to Claim 1.

3. The photographing control unit controls the camera to photograph the pallet with the fixed values of the plurality of exposures determined by the exposure determination unit in a state where the industrial vehicle is stopped. The pallet detection device according to Claim 1 or 2.

Images