Forklift and forklift group

Abstract

A forklift 10 comprises: a depth distance detection unit 17 that is attached to a side part of a vehicle body 11 and that detects a depth distance of at least one pallet 30 which exists to the side of a vehicle body 11; and a calculation unit 23 that, during travel, detects depth distances of the pallet at a plurality of locations to acquire a depth distance data group, and that acquires placement information pertaining to the pallet 30 on the basis of the depth distance data group, wherein the calculation unit 23 detects the type of pallet 30 by comparing the depth distance data group with a preset template 41, and acquires, as the placement information, the relative position and relative orientation of the pallet 30 with respect to the vehicle body 11.

Description

Forklifts and forklift fleets

[0001] The present invention relates to a forklift, particularly to a forklift capable of traveling by automatic driving and a group of automatically driving forklifts that operate simultaneously in a plurality of units.

[0002] In recent years, many forklifts that travel unmanned to transport loads have been proposed. For example, Patent Document 1 proposes an autonomous forklift equipped with an object detection unit consisting of a sensor unit and a shape recognition unit, in which the sensor unit is attached to the vehicle body and acquires a three-dimensional point cloud ahead, and the shape recognition unit recognizes the shape and position of an object located ahead from the three-dimensional point cloud acquired by the sensor unit.

[0003] Japanese Patent Application Laid-Open No. 2022-34408

[0004] Conventional self-driving forklifts recognize the shape and position of objects such as pallets by driving the vehicle so that the objects are positioned in front of the vehicle when picking up an object. This required the forklift to move forward to recognize each three-dimensional object, which took time to recognize and process multiple pallets and other objects.

[0005] Therefore, an object of the present invention is to provide a forklift and a group of forklifts that can reduce the time required to recognize and process the arrangement of each three-dimensional object even when there are multiple three-dimensional objects.

[0006] The forklift of the present invention, which solves the above-mentioned problems, comprises a depth distance detection unit attached to the side of the vehicle body to detect the depth distance of one or more three-dimensional objects present to the side of the vehicle body, and a calculation unit which detects the depth distance of the three-dimensional objects at multiple locations while traveling to obtain a group of depth distance data, and obtains position information of the three-dimensional objects based on the group of depth distance data, and the calculation unit is configured to obtain the relative position and relative orientation of the three-dimensional objects with respect to the vehicle body as position information based on the group of depth distance data.

[0007] With this forklift, the depth distance of a three-dimensional object located to the side of the vehicle body can be detected at multiple locations while the forklift is moving using a depth distance detection unit, and the calculation unit can obtain the relative position and orientation of the three-dimensional object with respect to the vehicle body as placement information based on a group of depth distance data containing multiple depth distances.

[0008] According to the present invention, a forklift can acquire the relative position and orientation of a three-dimensional object, such as a cargo pallet, simply by passing near it while traveling, and can continuously acquire and simultaneously recognize the position information of each object, whether there is one or multiple three-dimensional objects. Therefore, the conventional operation of traveling to a position directly opposite each three-dimensional object to recognize its position and orientation is completely unnecessary, significantly reducing the time required to recognize and process each three-dimensional object. Furthermore, because the forklift can recognize the position and orientation of each three-dimensional object on each side simply by traveling, various tasks such as transporting and cleaning up can be efficiently performed without any limit on the number of three-dimensional objects, even if they are arranged in a disorderly manner.

[0009] Furthermore, since the depth distance detection unit is attached to the side of the forklift, rather than the front, the depth distance can be detected even when a load is placed on the front fork. Therefore, the position and orientation of a three-dimensional object can be recognized even while the load is being transported, improving processing efficiency. Therefore, according to the present invention, a forklift can be provided that can recognize the position of each three-dimensional object and reduce the time required to process it, even when there are multiple three-dimensional objects.

[0010] The forklift of the present invention includes a vehicle body detection unit that detects the position and orientation of the vehicle body within a predetermined area when detecting the depth distance, and the calculation unit can obtain the absolute position and orientation of the three-dimensional object within the predetermined area as location information based on the relative position and relative orientation as well as the vehicle body position and orientation. In this way, the absolute position and absolute orientation of the three-dimensional object within the predetermined area can be obtained as location information, making it easy to grasp the location of the three-dimensional object.

[0011] The forklift of the present invention preferably includes a memory unit that stores the position and orientation of the vehicle body along with the relative position and orientation, a travel drive unit that can drive the vehicle body forward, backward, turning, and rotating, and a travel control unit that controls the travel drive unit, and after traveling, the travel control unit generates a travel path from the current position to a predetermined proximity position relative to the three-dimensional object based on the position and orientation of the vehicle body in the memory unit and the placement information of the three-dimensional object, and controls the travel drive unit to travel along the travel path to the predetermined proximity position.

[0012] This allows the vehicle to acquire location information of the three-dimensional object while traveling, store the location information, along with the vehicle body position and orientation in the memory unit, and after traveling, generate a movement path from the current position to a predetermined proximity position relative to the three-dimensional object based on this information, and control the traveling drive unit to travel from the current position to the predetermined proximity position. Therefore, it is possible to detect multiple three-dimensional objects at once, and then travel sequentially to the predetermined proximity position of each three-dimensional object, thereby efficiently performing processing on each three-dimensional object.

[0013] The travel control unit preferably controls the travel drive unit so that the vehicle body travels along a preset reference path during the detection period of the depth distance detection unit, and controls the travel control unit so that after the detection period, the vehicle body travels along the reference path from its current position to a detection position where the depth distance data group was detected, turns at the detection position, and travels to a predetermined approach position. This improves the movement accuracy of the vehicle body, allowing it to accurately travel to the predetermined approach position of each three-dimensional shape part.

[0014] In the present invention, the three-dimensional object may be a loading pallet, and the predetermined proximity position is the insertion position of the fork relative to the pallet insertion opening. A shape detection unit that detects the two-dimensional planar shape of the pallet present on the side of the vehicle body may be provided on the side of the vehicle body, and the calculation unit may be configured to determine a candidate range for the pallet insertion opening from the two-dimensional planar shape and to extract the depth distance data group from the depth distance of each part in the candidate range.

[0015] In this way, the shape detection unit detects the two-dimensional planar shape of the pallet located on the side of the vehicle body to determine the candidate range for the pallet insertion port, and then extracts a group of depth distance data from the depth distance within this candidate range, thereby making it possible to efficiently and accurately obtain the relative position and relative orientation of the pallet insertion port.

[0016] The calculation unit can detect the type of pallet by comparing the group of depth distance data with a preset template, and can obtain the relative positions and orientations of the end faces of the pallet and the pallet insertion slot. This makes it possible to obtain the type of pallet by comparing the group of depth distance data with the preset template, and further obtain the relative positions and orientations of the end faces of the pallet and the pallet insertion slot based on the group of depth distance data, making it easy to recognize the pallet insertion slot and to properly insert the forks of a forklift into the pallet insertion slot.

[0017] The calculation unit may fit a virtual line to the depth distance data group, select three approximate portions along the virtual line, and acquire the type of pallet from the template that best matches the three approximate portions when the three approximate portions are the end faces of the pallet beams and the space between the approximate portions is the insertion slot. This makes it possible to more accurately acquire the relative position and orientation of the pallet with respect to the end face and insertion slot.

[0018] The calculation unit may obtain information about the pallet or the cargo on the pallet from a code included in the two-dimensional planar shape detected by the shape detection unit. In this way, information about the pallet or the cargo on the pallet is obtained from the code included in the two-dimensional planar shape detected by the shape detection unit, so that more information can be obtained to carry out work such as transporting and putting away the pallet, thereby further improving work efficiency.

[0019] The depth distance detection unit can detect groups of depth distance data at multiple height positions, and the calculation unit can obtain the relative position and relative orientation of the pallet end face and the pallet insertion slot based on the multiple groups of depth distance data at multiple height positions. This allows for more accurate determination of the relative position and relative orientation. In this case, the calculation unit may obtain the relative position and relative orientation for each of multiple pallets stacked one above the other based on the multiple groups of depth distance data. In this way, multiple pallets can be recognized efficiently.

[0020] The shape detection unit and depth distance detection unit are composed of a common depth camera, and the calculation unit determines a candidate range for the pallet insertion port from the two-dimensional planar shape acquired by the depth camera, acquires depth distance data groups within the candidate range at multiple heights using the depth camera, fits horizontal virtual straight lines to each of the multiple depth distance data groups, determines positions that match multiple pre-created template pallets and acquires a score, and recognizes the pallet as a template pallet if it has a predetermined number or more depth distance data groups with a score above a threshold, and in this way can calculate the coordinates of the pallet from the position of the forklift at the time the depth distance data groups were acquired by the depth camera.

[0021] The travel control unit may determine a transport route from the insertion position to a preset pallet destination position and control the travel drive unit to travel along the transport route. The calculation unit may detect multiple pallets during a detection period by the depth distance detection unit, store the relative position and relative orientation for each pallet together with the position and orientation of the vehicle body, and, after the detection period, move sequentially to the insertion positions of the multiple pallets based on these to transport each pallet to the pallet destination position.

[0022] The calculation unit may detect a two-dimensional planar shape and a group of depth distance data to the side of a predetermined pallet transport destination position while traveling, and determine whether a pallet can be placed at the predetermined pallet transport destination position based on the two-dimensional planar shape and the group of depth distance data.

[0023] The forklift fleet of the present invention includes a plurality of forklifts as described above, and is configured so that the relative position and orientation of one of the forklifts to a pallet, stored together with the vehicle body position and orientation in the memory unit of the other forklifts, can be used via a network by the travel control units of the other forklifts to transport the pallet. In this way, multiple pallets can be efficiently transported to their destination locations by multiple forklifts.

[0024] According to the present invention, it is possible to provide a forklift and a group of forklifts that can reduce the time required to recognize and process the arrangement of each three-dimensional object even when there are multiple three-dimensional objects.

[0025] 4A is a schematic side view of a forklift according to an embodiment of the present invention; FIG. 4B is a block diagram illustrating the configuration of the forklift shown in FIG. 1; FIG. 4C is a diagram illustrating a travel path of the forklift during a detection period according to an embodiment of the present invention; FIG. 4D is a diagram illustrating the first half of a flowchart illustrating the operation of the forklift according to an embodiment of the present invention; FIG. 4E is a diagram illustrating the second half of the flowchart shown in FIG. 4A; FIG. 4F is a diagram illustrating a state in which a pallet is detected by a forklift according to an embodiment of the present invention; FIG. 4G is a diagram illustrating an example of a travel state when multiple pallets are detected by a forklift according to an embodiment of the present invention; FIG. 4H is a diagram illustrating another example of a travel state when multiple pallets are detected by a forklift according to an embodiment of the present invention; FIG. 4I is a diagram illustrating a process in which a pallet is detected by a forklift according to an embodiment of the present invention, showing a candidate range; FIG. 4J is a diagram illustrating a process in which a pallet is detected by a forklift according to an embodiment of the present invention, showing a slice area; FIG. 4J is a diagram illustrating a process in which a pallet is detected by a forklift according to an embodiment of the present invention, showing a state in which a depth distance group is fitted to a virtual straight line; FIG. 4J is a diagram illustrating a process in which a pallet is detected by a forklift according to an embodiment of the present invention, showing a state in which the depth distance group is compared with a template; FIG. 4I is a diagram illustrating a travel state in which a forklift according to an embodiment of the present invention moves to a predetermined proximity position; FIG. 4J is a diagram illustrating a travel state in which a forklift according to an embodiment of the present invention transports a pallet to a destination position; FIG. 4I is a schematic diagram illustrating an operation in a modified embodiment of the present invention when the end face having the insertion opening of the 13A and 13B are schematic diagrams illustrating a modification of the embodiment of the present invention, and are used to explain the operation when a plurality of pallets are arranged so as to overlap one another on the side of the reference path.

[0026] An embodiment of the present invention will now be described in detail with reference to the accompanying drawings. In this embodiment, an example is used in which an autonomous forklift detects and transports a pallet as a three-dimensional object within a predetermined area such as a warehouse.

[0027] Fig. 1 is a schematic perspective view of a forklift truck according to this embodiment, and Fig. 2 is a block diagram showing the configuration thereof. The forklift truck 10 includes a vehicle body 11, an operation panel 12 for various inputs, a travel drive unit 13 capable of driving the vehicle body 11 forward, backward, turning, and rotating, a depth camera 17 attached to the side of the vehicle body 11 as a depth distance detection unit, a color camera 18 as a shape detection unit, a laser sensor 19 as a vehicle body detection unit that detects the position and orientation of the vehicle body 11 when detected by the depth camera 17, a calculation unit 23 that processes detection information from the depth camera 17, the color camera 18, and the laser sensor 19 and generates control commands for each unit, and a storage unit 25 that stores detection information and processing results from the depth camera 17 as well as detection information and processing results from the laser sensor 19.

[0028] The calculation unit 23 shown in FIG. 2 is a so-called CPU and includes a microcomputer, etc. The storage unit 25 is composed of volatile memory such as DRAM and non-volatile memory such as flash memory and / or HDD. The travel drive unit 13, fork drive unit 11d, operation panel 12, depth camera 17, color camera 18, laser sensor 19, and monitoring sensors such as obstacle sensors, all connected to the calculation unit 23, are connected via an interface circuit (I / O circuit) (not shown). The calculation unit 23, storage unit 25, I / O circuit, etc. may be integrated into a single-board computer or a computer. In the calculation unit 23, the travel control unit 11, the fork drive unit 11d, and programs and applications (hereinafter referred to as "programs") for candidate range extraction, pallet position detection, vehicle position estimation, obstacle detection, etc., which will be described in the flowcharts below, are stored in the storage unit 25. The program stored in the memory unit 25 is executed by the calculation unit 23 and the memory of the memory unit 25, and the travel control unit 15, travel drive unit 13, fork control unit 11e, fork drive unit 11d, etc. are controlled based on signals detected by monitoring sensors and control signals input via the operation panel.

[0029] The vehicle body 11 includes a vehicle body main body 11a, a mast 11b provided vertically at the front of the vehicle body main body 11a, a fork 11c supported on the mast 11b so as to be movable up and down and protruding forward, and a fork drive unit 11d that drives the fork 11c up and down. The fork 11c is controlled by a fork control unit 11e based on the processing results of each detection information, thereby enabling cargo pick-up and unloading operations.

[0030] The travel drive unit 13 is provided at the bottom of the vehicle body 11a, has a plurality of wheels 13a, some or all of which are driven by travel motors 13b, and is configured to be steerable by a steering motor 13c. By adjusting the rotation direction and rotation speed of the wheels 13a and the steering direction and steering speed by the steering motor 13c, the vehicle body 11 can move forward, backward, and turn, and can also turn on the spot, rotating in the same position.

[0031] The driving control unit 15 can control the driving drive unit 13 to make the vehicle body 11 travel along a predetermined route, a short-distance or long-distance route calculated by calculation, etc., and by controlling the rotation direction and rotation speed of the wheels 13a, the steering direction and steering speed, etc., the vehicle body 11 can be made to travel along various routes.

[0032] In the travel control unit 15 of this embodiment, for example, during a detection period in which a pallet 30 is detected and its location information is acquired, the travel drive unit 13 may be controlled so that the vehicle body 11 travels along a predetermined reference path 33 set in advance within a predetermined area 31, as shown in Figure 3. Furthermore, after the detection period, i.e., after pallet detection has been interrupted or terminated at an appropriate point in time, a travel path 34 is generated from the current position, i.e., the position and orientation of the vehicle body 11 at the end of the detection period, based on the location information of the detected pallet 30, to a predetermined proximity position 32 relative to the pallet 30, and the travel drive unit 13 is controlled to travel along the travel path 34.

[0033] This movement path 34 may be, for example, a path that travels the shortest distance from the current position to the specified approach position 32, but in this embodiment, as shown in Figure 9, a path is set that travels along the reference path 33 from the current position to the detection position 36 where the placement information of the pallet 30 is detected, and then turns around on the spot at the detection position 36 to travel to the specified approach position 32.

[0034] Furthermore, after picking up the pallet 30 at the predetermined approach position 32, the travel control unit 15 determines a transport route 35 from the predetermined approach position 32 to a preset pallet destination position 38, and controls the travel drive unit 13 to travel along the transport route 35. This transport route 35 may be, for example, the shortest route from the predetermined approach position 32 to the pallet destination position 38, but in this embodiment, as shown in Figure 10, a route is set that returns from the predetermined approach position 32 to the detection position 36 and travels along the reference route 33 to the pallet destination position 38.

[0035] Here, the predetermined approach position 32 refers to a position where the forklift 10 faces the end face of the pallet 30 where the insertion openings 30a are provided, and where each fork 11c faces directly toward each insertion opening 30a of the pallet 30, and where the protruding direction of the fork 11c is arranged substantially along the axial direction of the insertion opening 30a, or an insertion position 37 where the fork 11c is inserted into the insertion opening 30a. When the predetermined approach position 32 is a position where the fork 11c faces directly toward the insertion opening 30a, the fork 11c advances to the insertion position 37 during loading. The travel control unit 15 controls the travel drive unit 13 to stop the vehicle body 11 or to cause the vehicle body 11 to detour based on detection information from various sensors provided on the vehicle body 11, such as an obstacle sensor 16, an IMU (Inertial Measurement Unit) such as a 6-axis gyro sensor (not shown), and a distance sensor.

[0036] The depth camera 17 detects the depth distance of each part and is attached to the side of the vehicle body 11, or in this embodiment, the side of the mast 11b, facing laterally in a direction perpendicular to the fore-and-aft direction of the vehicle body 11. The depth camera 17 may be installed on one or both sides of the vehicle body 11, and may be installed so that it can be raised and lowered along the mast, or multiple cameras may be installed on the side at intervals in the vertical direction. By being able to detect depth distance data at multiple height positions, for example, the depth distance of a single pallet 30 can be detected at multiple heights, improving recognition accuracy. It is also possible to detect multiple pallets 30 stacked one on top of the other. In this embodiment, the depth camera 17 detects depth distances at multiple locations on the pallet 30, specifically, at multiple locations on the pallet 30 in the horizontal direction at a predetermined height, and transmits depth distance data consisting of the detected multiple depth distances to the calculation unit 23.

[0037] The color camera 18 detects the two-dimensional planar shape, and in this embodiment is provided integrally with the depth camera 17, and like the depth camera 17, is attached to the side of the mast 11b on one or both sides of the vehicle body 11, facing laterally in a direction perpendicular to the fore-and-aft direction of the vehicle body 11. The color camera 18 may be provided so that it can be raised and lowered, like the depth camera, or multiple color cameras 18 may be provided on the side with spacing in the vertical direction. In this embodiment, when the color camera 18 detects a pallet 30 present to the side of the vehicle body 11 while traveling, it captures a color image of the pallet 30 and transmits this color image data to the calculation unit 23 as a two-dimensional planar shape.

[0038] The color camera 18 captures color image data of the two-dimensional planar shape of the pallet 30 present on the side of the vehicle body 11, either continuously or periodically, or when it detects a pallet 30 on the side of the vehicle body 11, as shown in FIG. 7A (described later), and transmits the captured color image data to the calculation unit 23, which then calculates a candidate insertion slot range 40 for the pallet 30 from the two-dimensional planar shape. In this embodiment, the candidate insertion slot range 40 is a range that includes the two insertion slots 30a and the end faces 30b of the three girders located on both sides of each insertion slot 30a. The depth camera 17 may also be configured to detect multiple depth distances within the candidate insertion slot range 40 for the pallet 30.

[0039] The laser sensor 19 detects a detection position 36 where depth distance data is detected by the depth camera 17, i.e., the position and orientation of the vehicle body 11 at the time of detection. In this embodiment, a position sensor is provided on the top of the vehicle body 11 and detects the position and orientation of the vehicle body 11 within a predetermined area 31, such as a logistics warehouse. The position sensor is composed of, for example, a laser scanner and a light receiving unit, and detects the position and orientation of the vehicle body 11 by emitting a laser beam from the laser scanner and receiving the laser beam reflected by reflectors arranged at various locations within the predetermined area 31 with the light receiving unit.

[0040] This laser sensor 19 detects the position and orientation of the forklift 10 within the specified area 31 when it acquires the depth distance of each part within the candidate range 40 of the insertion port of the pallet 30, and transmits this to the calculation unit 23.

[0041] The calculation unit 23 processes the detection information from the operation panel 12, the depth camera 17, the color camera 18, the obstacle detection sensor 16, and the laser sensor 19 to determine an action and generate control commands for each component based on the determined action. Specifically, the determination of an action refers to the execution of a program in the calculation unit 23 to determine an action related to the control of the travel drive unit 13 and the fork drive unit 11d in the body 11 of the forklift 10 based on the control signal from the operation panel 12, the pallet position recognition signal from the depth camera 17 and / or the candidate range extraction signal from the color camera 18, the obstacle detection signal from the obstacle detection sensor 16, and the vehicle position estimation signal from the laser sensor 19. The calculation unit 23 first recognizes the pallet position from the information detected by the depth camera 17 and the color camera 18 while the forklift is traveling. As shown in FIG. 7A (described later), a candidate range 40 for the insertion slot of the pallet 30 is extracted based on color image data from the color camera 18. From the multiple depth distances obtained by the depth camera 17 in the candidate range 40 of the insertion slot of this pallet 30, a group of depth distance data consisting of the depth distances of each part on a virtual plane obtained by slicing the candidate range 40 horizontally at a certain height is obtained, as shown in Fig. 7B described below. At this time, multiple groups of depth distance data at multiple height positions may be obtained.

[0042] The obtained group of depth distance data is compared with a preset template 41 to detect the type of pallet 30. For example, as shown in Figures 8A and 8B (described later), a horizontal imaginary line 42 is fitted to the group of depth distance data, three approximate portions 43 along the imaginary line 42 are selected, and the three approximate portions 43 are set as the end faces 30b of the beams of the pallet 30 and the gaps between the approximate portions 43 are set as the insertion slots 30a. This makes it possible to identify the type of pallet 30 from the template 41 that best matches.

[0043] At the same time, the calculation unit 23 can obtain placement information including the relative position and orientation of the pallet 30 with respect to the vehicle body 11 based on a depth distance data group having a plurality of depth distances. For example, the relative position and orientation of the end face 30b and insertion port 30a of the pallet 30 with respect to the vehicle body 11 may be obtained from the orientation and distance of the depth distance data group fitted to the virtual straight line 42. This allows the calculation unit 23 to recognize the pallet position.

[0044] The calculation unit 23 also estimates the vehicle position from the information detected by the laser sensor 19. The position and orientation of the forklift 10 at the time when the placement information of the pallet 30 is acquired, i.e., at the time when the depth distance data group is acquired, are estimated, and by combining this position and orientation of the forklift 10 with the placement information of the pallet 30, the absolute position and absolute orientation of the pallet 30 within the specified area 31, world coordinates, etc. are acquired.

[0045] The calculation unit 23 of this embodiment recognizes the presence of obstacles or the like that may hinder the operation of the forklift 10 based on detection information from the obstacle sensor 16 and other sensors. Furthermore, various types of information about the pallet 30 or the cargo on the pallet 30 can be obtained from codes contained in the two-dimensional planar shape detected by the color camera 18. Methods for obtaining such various types of information include QR Code (registered trademark) and recognition of surface printing using image processing. The calculation unit 23 of this embodiment determines the behavior of the forklift 10 based on the results of the recognition of the pallet position, estimation of the vehicle position, and recognition of obstacles.

[0046] The calculation unit 23 of this embodiment is provided with a travel control unit 15 that controls the travel drive unit 13 and a fork control unit 11e that controls the fork drive unit 11d. The travel control unit 15 generates travel commands and steering commands based on the action-determined information, and can control the travel motor 13b and the steering motor 13c. The fork control unit 11e generates fork control commands based on the action-determined information, and can control the fork drive unit 11d.

[0047] The memory unit 25 stores pallet arrangement information including the relative position and relative orientation of the pallet 30, vehicle body arrangement including the position and orientation of the vehicle body 11 at the time of detection, arrangement information such as the absolute position and absolute orientation of the pallet 30 or world coordinates, pre-set templates, various pre-set information regarding the pallet 30 and the specified area 31, etc., which can be used by the calculation unit 23 as needed.

[0048] Next, an operation of the forklift 10 to detect a plurality of pallets 30 arranged within a predetermined area 31, such as a logistics warehouse, and transport them to a predetermined pallet destination position 38 will be described. FIGS. 4A and 4B are flowcharts showing the flow of the operation of detecting and transporting a pallet 30. In FIGS. 4A and 4B, the display positions indicated by the symbol A are consecutive, and the display positions indicated by the symbol B are consecutive. First, as shown in FIG. 3, the forklift 10 travels along a predetermined reference path 33 within the predetermined area 31 by the travel control unit 15 controlling the travel drive unit 13 (S11). This period is a detection period for detecting pallets 30 present within the predetermined area 31 and obtaining information such as the placement of the pallets 30.

[0049] As shown in Figure 5, when a pallet 30 is present to the side of the vehicle body 11 while the forklift 10 is traveling, the color camera 18 captures an image of the pallet 30 to the side of the vehicle body 11 while the forklift 10 is traveling, and color image data is acquired as a two-dimensional planar shape (S12). The color image data detected during the detection period is transmitted to the calculation unit 23. At this time, a single pallet 30 may be detected as shown in Figure 5, but multiple pallets 30 may also be detected successively or collectively while the forklift 10 is traveling.

[0050] Fig. 6A is a diagram illustrating an example of a traveling state when a forklift according to an embodiment of the present invention detects multiple pallets, and Fig. 6B is a diagram illustrating another example of the traveling state. The multiple pallets 30 do not need to be arranged side by side in the traveling direction of the vehicle body 11. For example, as shown in Fig. 6A, the multiple pallets 30 may be arranged diagonally with respect to the traveling direction of the vehicle body 11 on the reference path 33, or as shown in Fig. 6B, each pallet 30 may be arranged at an arbitrary inclination with respect to the traveling direction of the reference path 33.

[0051] FIG. 7A illustrates a process for detecting a pallet using a forklift according to an embodiment of the present invention, showing a candidate range 40 for the insertion slot of the pallet 30. FIG. 7B illustrates a slice region in the process for detecting the pallet. FIG. 8A illustrates a process for detecting a pallet using a forklift according to an embodiment of the present invention, showing a state in which a group of depth distances is fitted to a virtual line. FIG. 8B illustrates a process for detecting a pallet using a forklift according to an embodiment of the present invention, showing a state in which the pallet is matched to a template. As shown in FIG. 7A , the calculation unit 23 sets a candidate range 40 for the insertion slot of the pallet 30 based on color image data acquired by the color camera 18 (S13). In this embodiment, the candidate range 40 for the insertion slot of the pallet 30 may be determined by machine learning based on the color image data. For example, a recognition model trained on the shape of the target pallet 30 may be prepared in advance, and an image captured during detection may be input into the training model to obtain a circumscribed rectangle of the surface including the insertion slot of the pallet 30, and the rectangle may be used as the candidate range 40.

[0052] When multiple palettes 30 exist in the color image data, multiple candidate ranges 40 are determined. In this embodiment, multiple palettes 30 are arranged horizontally on the left and right, and multiple palettes 30 are stacked vertically. Therefore, multiple candidate ranges 40 are determined in the horizontal and vertical directions for the color image data.

[0053] Meanwhile, the depth camera 17 captures depth images and detects the depth distance of each part. The depth images detected during the detection period are transmitted to the calculation unit 23 (S14). The calculation unit 23 extracts a depth distance data group (candidate point group) consisting of depth distances at multiple positions from the depth distances of each part of the pallet 30 in the candidate range 40 (S15).

[0054] As shown in Fig. 7B, the calculation unit 23 sets slice regions at a certain height in the horizontal direction for each candidate range 40 of the insertion slot of each pallet 30 based on the depth distances of each part in the candidate range 40 of the insertion slot of each pallet 30 detected by the depth camera 17, and acquires a group of depth distance data for each slice region. In this embodiment, fine slice regions are set for each candidate range 40, and groups of depth distance data at multiple heights are acquired. The multiple groups of horizontal depth distance data acquired for each slice region are each projected onto a virtual horizontal plane, as shown in Fig. 8A, and virtual line fitting is performed to fit a virtual line 42 on the virtual horizontal plane (S16).

[0055] Furthermore, the depth distance data group to which the virtual straight line 42 has been fitted is compared with various pre-created palette templates as shown in Figure 8B to find the matching position, obtain a score, and perform template matching to score all slice areas in the same way (S16).

[0056] Meanwhile, the laser sensor 19 detects the position and orientation of the forklift 10 within the specified area 31 at the time the depth distance is acquired by the depth camera 17, and transmits this to the calculation unit 23, thereby acquiring vehicle body positioning information (S17).

[0057] If the template matching finds that there are a predetermined number or more positions within the candidate range 40 where the score is equal to or greater than the threshold, the calculation unit 23 determines that the detected pallet 30 is a pallet of the template 41, and acquires the type of pallet. At the same time, the relative orientation of the end face 30b of the pallet 30 where the insertion port 30a is provided, with respect to the vehicle body 11, is acquired from the virtual straight line 42. In addition, the relative positions of the end face 30b of the girders of the pallet 30 and the insertion port 30a, with respect to the vehicle body 11, are acquired from the depth distance data group fitted to the virtual straight line 42. The calculation unit 23 can then combine the relative orientation and relative position with the vehicle body arrangement information to acquire pallet arrangement information (S18).

[0058] In this way, the detection of the relative orientation and relative position of the pallet 30 with respect to the vehicle body 11 is performed sequentially for each pallet 30 detected during the detection period for the pallet 30, and the pallet placement information including the relative position and relative orientation for each acquired pallet 30 is recorded in the memory unit 25 together with vehicle placement information including the position and orientation of the vehicle body 11 (S19).

[0059] The detection period continues, and it is determined whether the preset pallet detection period has been interrupted or ended (S20). If it has been determined that the period should be continued, the detection period continues by traveling along the reference path.

[0060] On the other hand, after the pallet detection period is interrupted or ended, each pallet 30 is sequentially unloaded based on the pallet arrangement information of each pallet 30 recorded in the memory unit 25 and the vehicle body arrangement information of the vehicle body 11 recorded together with this pallet arrangement information, and the pallet is transported to a predetermined pallet transport destination position.

[0061] First, at a position where detection of the pallet 30 was interrupted or ended, the laser sensor 19 acquires the current position, including the current position and orientation of the vehicle body, and transmits this to the travel control unit 15 (S21). The travel control unit 15 generates a travel path 34 from the current position to a predetermined approach position 32 relative to the pallet 30 (S21). As shown in Fig. 9 , by controlling the travel drive unit 13, the forklift 10 travels from the current position to a detection position 36 along a reference path 33, turns around on the spot at the detection position 36, and travels to the predetermined approach position 32 (S22).

[0062] The forklift 10 is turned around on the spot at the detection position to correspond to the relative orientation of the pallet 30, and is brought directly face-to-face with the end face of the pallet 30 where the insertion opening 30a is provided, thereby reaching the predetermined approach position 32 (S23). Then, each fork 11c is placed at a position corresponding to the insertion opening 30a, and is moved forward to perform the operation of picking up the pallet 30 at the insertion position 37 (S24).

[0063] Thereafter, the travel control unit 15 generates a transport path 35 from the insertion position 37 to a preset pallet destination position 38 (S25), and the travel drive unit 13 is controlled to transport the pallet 30 along the transport path 35 (S26). The transport path 35 is set so that the forklift 10 travels to the pallet destination position 38, for example, by traveling along the reference path 33 from the predetermined approach position 32 to near the pallet destination position 38, and then turning around on the spot at that position. The forklift 10 travels until it reaches the pallet destination position 38 (S27), and the pallet 30 is unloaded at the pallet destination position 38 (S28).

[0064] When the forklift 10 travels along the transport path 35 and passes by the side of the pallet destination position 38, the calculation unit 23 detects the two-dimensional planar shape and depth distance data to determine whether a pallet 30 can be placed at the pallet destination position 38, and determines whether the next pallet 30 can be transported.

[0065] The same operation is repeated for the multiple pallets 30 detected during the detection period, and the pallets 30 are transported sequentially. When the transport of all pallets 30 detected during the travel period has been completed (S29), it is determined whether to continue the operation of detecting the pallets 30 and transporting them to the pallet transport destination position 38 (S30). If the operation is to continue, the detection period is started again, and the forklift 10 starts traveling along the reference path 33 (S11).

[0066] According to the forklift 10 of this embodiment, the depth distance of the pallet 30 located to the side of the vehicle body 11 is detected while the forklift 10 is moving using the depth camera 17, and the calculation unit 23 obtains the relative position and relative orientation of the pallet 30 with respect to the vehicle body 11 as placement information based on a group of depth distance data having multiple depth distances.

[0067] The forklift 10 can acquire the location information of the pallets 30 simply by passing near the pallets 30 while traveling, and whether there is one or multiple pallets 30, the location information of each pallet 30 is continuously acquired and recognized all at once while traveling. In this case, there is no need to travel to a position directly opposite each pallet 30 to recognize the position and orientation of the pallet 30, and the time required to recognize and process each pallet 30 can be significantly reduced. Furthermore, because the position and orientation of each pallet 30 can be recognized by the forklift 10 traveling, various processes such as transportation and cleanup can be efficiently performed without any limit on the number of pallets 30, even if they are arranged in a disorderly manner.

[0068] Because the depth camera 17 is attached to the side of the forklift 10, rather than the front, the depth distance can be detected by the depth camera 17 even when a load is placed on the front fork 11c. Therefore, the position and orientation of other pallets 30 can be recognized even while a load is being transported, improving processing efficiency. For example, it is possible to check whether the pallet destination position 38 is empty while a load is being transported, or to transport the pallet 30 stacked one above the other. Even when there are multiple pallets 30, the time required to recognize the position of each pallet 30 and process it can be reduced.

[0069] The forklift 10 of this embodiment is equipped with a laser sensor 19 that detects the position and orientation of the vehicle body 11 within the specified area 31 when detecting the depth distance, and the calculation unit 23 can obtain the absolute position and orientation of the pallet 30 within the specified area 31 and world coordinates as placement information based on the relative position and relative orientation as well as the position and orientation of the vehicle body 11, so the placement of the pallet 30 can be easily grasped.

[0070] The forklift 10 of this embodiment is equipped with a memory unit 25 that stores the position and orientation of the vehicle body 11 along with the placement information of the pallets 30, a travel drive unit 13, and a travel control unit 15. After traveling during the detection period, the travel control unit 15 generates a travel path 34 from the current position to a predetermined approach position 32 based on the position and orientation of the vehicle body 11 and the placement information of the pallets 30 stored in the memory unit 25, and the travel drive unit 13 and the travel control unit 15 travel along the travel path 34 to the predetermined approach position 32. Therefore, during traveling during the detection period, multiple pallets 30 can be detected at once, and then the forklift 10 can travel sequentially to the predetermined approach position 32 of each three-dimensional object, allowing efficient processing of each pallet 30.

[0071] In the forklift 10 of this embodiment, the travel control unit 15 controls the travel drive unit 13 to travel along a preset reference path 33 during the detection period of the depth camera 17, and after the detection period, controls the travel drive unit 13 to travel along the reference path 33 from the current position to a detection position 36 where a group of depth distance data is detected, turn at the detection position 36, and travel to the predetermined approach position 32. This improves the movement accuracy of the vehicle body 11, allowing it to accurately travel to the predetermined approach position 32 of each three-dimensional object. Furthermore, generation of a travel path from the current position to the predetermined approach position 32 and control of the travel control unit 15 are simplified.

[0072] The forklift 10 of this embodiment is equipped with a color camera 18 as a shape detection unit that detects the two-dimensional planar shape of the pallet 30 located to the side of the vehicle body 11, and determines a candidate range 40 for the insertion opening of the pallet 30 from the two-dimensional planar shape, while the depth camera 17 extracts a group of depth distance data for each part within the candidate range 40. The color camera 18 detects the two-dimensional planar shape of the pallet 30 located to the side of the vehicle body 11 to determine the candidate range 40 for the insertion opening of the pallet 30, and depth distances within this candidate range 40 can be detected in a focused manner by setting multiple slice regions, etc., so that the relative position and relative orientation of the insertion opening 30a of the pallet 30 can be obtained efficiently and accurately.

[0073] The forklift 10 of this embodiment obtains the type of pallet 30 by comparing the depth distance data group with a pre-set template 41, and further obtains the relative position and relative orientation of the end face 30b and insertion port 30a of that type of pallet 30 based on the depth distance data group, thereby accurately recognizing the insertion port 30a of the pallet 30 and being able to properly insert the forks 11c of the forklift 10 into the insertion port 30a of the pallet 30.

[0074] In the forklift 10 of this embodiment, the calculation unit 23 can obtain information about the pallet 30 or the cargo on the pallet 30 from the code contained in the two-dimensional planar shape detected by the color camera 18. This allows for obtaining a large amount of information and processing such as transporting and clearing away the pallet, allowing for more appropriate work to be performed and improving efficiency.

[0075] The above embodiment can be modified as appropriate within the scope of the present invention. In the above embodiment, an example was shown in which a single forklift 10 transports multiple pallets 30 in a predetermined area 31 to the pallet destination position, but it is also possible to transport multiple pallets 30 to the pallet destination position 38 using a group of forklifts including multiple forklifts 10.

[0076] In this case, the location information of the relative position and orientation with respect to the pallet 30, which is stored in the memory unit 25 of one of the forklifts 10 together with the position and orientation of the vehicle body 11, may be used by the travel control unit 15 of the other forklifts 10 via the network. The other forklifts 10 can use this location information to generate a movement route 34 from their current position to a predetermined proximity position 32, move along this route, and pick up the pallet 30. In this way, multiple pallets 30 can be efficiently transported to a pallet destination position 38 by multiple forklifts 10.

[0077] In the above embodiment, each pallet 30 in the predetermined area 31 is arranged with its insertion port 30a facing the reference path 33, but as shown in Figure 11, the present invention can also be applied to cases where the end face of the pallet 30 having the insertion port 30a does not face the reference path 33. In that case, the forklift 10 may travel along each side of the pallet 30 and detect the end face 30b having the insertion port 30a of the pallet 30 on the side of the vehicle body 11.

[0078] In the above embodiment, an example has been described in which a group of horizontal depth distance data is acquired by detecting the horizontal depth distance using the depth camera 17. However, instead of a depth camera, a two-dimensional laser scanner may be installed on the side of the vehicle body 11 so that it can scan vertically, as shown in FIG. 12. In this case, it is possible to acquire three-dimensional position information by scanning the depth distance of each part vertically while traveling (see the area enclosed by the two-dot chain line in FIG. 12). This improves accuracy and resolution compared to when a depth camera is used.

[0079] The above describes an example in which pallets 30 are arranged in a single layer along the reference path 33. FIG. 13A shows a modified example of the embodiment of the present invention, and is a schematic diagram illustrating the operation when multiple pallets are arranged sideways of the reference path so as to overlap one another. FIG. 13B is a schematic diagram illustrating another operation of FIG. 13A , and FIG. 13C is a schematic diagram illustrating yet another operation of FIG. 13A . As shown in FIGS. 13A to 13C , not only is the example in which pallets 30 are arranged in a single layer along the reference path 33, but a group of pallets may also be arranged sideways of the reference path 33 so as to overlap one another. In this case, when the forklift 10 travels along the reference path 33, as shown in FIG. 13A , the forklift 10 may detect multiple pallets 30 arranged closest to the reference path 33, transport each of them to the pallet destination position 38 (see FIG. 10 ), and then, as shown in FIGS. 13B and 13C , the forklift 10 may travel along the reference path again, detect the pallets 30 again, and transport the pallets. This process may be repeated. This allows multiple pallets 30 to be transported sequentially in parallel, making the transport of multiple pallets 30 easier.

[0080] In the above embodiment, a pallet 30 was used as an example of a three-dimensional object, but the present invention can also be applied to various three-dimensional objects that can be transported by a forklift 10, such as nesters, mesh pallets, shelves, etc.

[0081] In the above embodiment, a depth camera 17 is used that is commonly used in the depth distance detection unit and the shape detection unit. However, for example, a two-dimensional laser scanner, a stereo camera, a TOF (Time of Flight) camera, or a monocular camera may also be used. In the case of a two-dimensional laser scanner, three-dimensional information can be obtained by scanning the vehicle body 11 vertically while it is traveling, which provides better accuracy and resolution than a depth camera. It is also possible to provide the depth distance detection unit and the shape detection unit separately on the side of the vehicle body. A stereo camera, TOF camera, monocular camera, etc. may be used as the depth distance detection unit, and a monocular camera, three-dimensional laser scanner, etc. may be used as the shape detection unit.

[0082] In the above embodiment, the forklift 10 detects the pallets 30 while traveling within a predetermined area 31. However, the present invention can also be applied to a case in which the forklift 10 detects the pallets while traveling within an undefined area. For example, the present invention can be applied to a case in which the pallets 30 are retrieved from the bed of a truck parked in an open space. In this case, the forklift 10 first travels along an appropriately set travel path, and the depth camera 17 on the side of the vehicle body 11 detects multiple pallets 30 on the bed at once. Color images are captured, and the depth distances of each part are obtained. This allows each pallet 30 to be retrieved, as in the above embodiment. Note that the present invention is not limited to an example in which the forklift 10 detects the pallets 30 while traveling along a predetermined reference path 33. The present invention can also be applied to a case in which the forklift 10 detects the pallets 30 or other loadable items while traveling along an arbitrary travel path determined by various processes.

[0083] 10...Forklift, 11...Vehicle body, 11a...Vehicle body main body, 11b...Mast, 11c...Fork, 11d...Fork drive unit, 11e...Fork control unit, 12...Operation panel, 13...Travel drive unit, 13a...Wheels, 13b...Travel motor, 13c...Steering motor, 15...Travel control unit, 16...Obstacle sensor, 17...Depth camera, 18...Color camera, 19...Laser sensor, 23...Calculation unit, 25...Memory unit, 30...Pallet, 30a...Inlet, 30b...End face, 31...Predetermined area, 32...Predetermined proximity position, 33...Reference path, 34...Movement path, 35...Transport path, 36...Detection position, 37...Insertion position, 38...Pallet destination position, 40...Candidate range, 41...Template, 42...Virtual straight line, 43...Approximation unit

Claims

1. An object distance detection unit that is attached to a side portion of a vehicle body and detects a depth distance of one or more three-dimensional objects existing on the side of the vehicle body; an arithmetic unit that detects the depth distance of the three-dimensional object at a plurality of locations while the vehicle is traveling to obtain an object distance data group, and obtains arrangement information of the three-dimensional object based on the object distance data group; and a forklift, wherein the arithmetic unit obtains the relative position and relative orientation of the three-dimensional object with respect to the vehicle body as the arrangement information based on the object distance data group.

2. The forklift according to claim 1, further comprising a vehicle body detection unit that detects the position and orientation of the vehicle body within a predetermined region during the detection of the object distance, and the arithmetic unit obtains the absolute position and absolute orientation of the three-dimensional object within the predetermined region as the arrangement information based on the relative position and relative orientation and the position and orientation of the vehicle body.

3. A storage unit that stores the relative position and relative orientation, and the position and orientation of the vehicle body; a traveling drive unit that can drive the vehicle body forward, backward, turn, and pivot; and a traveling control unit that controls the traveling drive unit, wherein the traveling control unit generates a movement path from the current position to a predetermined proximity position with respect to the three-dimensional object based on the position and orientation of the vehicle body and the arrangement information of the three-dimensional object in the storage unit after traveling, and controls the traveling drive unit to travel to the predetermined proximity position along the movement path.

4. The forklift according to claim 3, wherein the traveling control unit controls the traveling drive unit so that the vehicle body travels along a preset reference path during the detection period of the object distance detection unit, and after the detection period, the traveling control unit controls the vehicle body to travel along the reference path from the current position to the detection position where the object distance data group is detected, and turn at the detection position and travel to the predetermined proximity position.

5. The forklift according to claim 3, wherein the three-dimensional object is a pallet for loading, and the predetermined proximity position is the insertion position of the fork with respect to the insertion port of the pallet.

6. The forklift according to claim 5, further comprising a shape detection unit that detects a two-dimensional planar shape of the three-dimensional object existing on the side of the vehicle body, wherein the calculation unit obtains a candidate range of the insertion port of the pallet from the two-dimensional planar shape and extracts the depth distance data group from the depth distances of respective parts in the candidate range.

7. The forklift according to claim 5, wherein the calculation unit detects the type of the pallet by comparing the depth distance data group with a preset template, and obtains the relative position and the relative orientation facing the end face of the pallet and the insertion port of the pallet.

8. The forklift according to claim 7, wherein the calculation unit fits a virtual straight line to the depth distance data group, selects three approximate parts along the virtual straight line, and obtains the type of the pallet from the template that most matches when the three approximate parts are used as the end faces of the digits of the pallet and the spaces between the approximate parts are used as the insertion ports.

9. The forklift according to claim 6, wherein the calculation unit obtains information on the pallet or the load on the pallet from a code included in the two-dimensional planar shape detected by the shape detection unit.

10. The forklift according to claim 6, wherein the depth distance detection unit can detect the depth distance data group at a plurality of height positions, and the calculation unit obtains the relative position and the relative orientation of the end face of the pallet and the insertion port of the pallet based on the plurality of depth distance data groups at the plurality of height positions.

11. The forklift according to claim 7, wherein the depth distance detection unit can detect the depth distance data group at a plurality of height positions, and the calculation unit obtains the relative position and the relative orientation for each of the plurality of pallets stacked vertically based on the plurality of depth distance data groups.

12. The shape detection unit and the depth distance detection unit are composed of a common depth camera. The calculation unit obtains a candidate range of the insertion port of the pallet from the two-dimensional planar shape acquired by the depth camera, acquires a group of depth distance data within the candidate range at a plurality of heights by the depth camera, fits a horizontal virtual straight line to each of the plurality of groups of depth distance data, obtains a position that matches the pallet of a plurality of templates created in advance, and acquires a score. When there are a predetermined number or more of the groups of depth distance data whose scores are equal to or higher than a threshold value, it is recognized that the pallet is the pallet of the template, and the coordinates of the pallet are calculated from the position of the forklift at the time when the group of depth distance data is acquired by the depth camera. The forklift according to claim 6.

13. The travel control unit obtains a transport route from the insertion position to a preset pallet transport destination position, and controls the travel drive unit to travel along the transport route. The forklift according to claim 7.

14. The calculation unit detects a plurality of the pallets during the detection period of the depth distance detection unit, stores the relative position and the relative orientation for each pallet together with the position and the orientation of the vehicle body, and sequentially moves to the insertion positions of the plurality of pallets based on these after the detection period to transport each pallet to the pallet transport destination position. The forklift according to claim 13.

15. The calculation unit detects the two-dimensional planar shape and the group of depth distance data while traveling on the side of a preset pallet transport destination position, and determines whether the preset pallet transport destination position can place the pallet based on the two-dimensional planar shape and the group of depth distance data. The forklift according to claim 6.

16. A plurality of forklifts according to any one of claims 5 to 15 are provided. By using the relative position and the relative orientation with respect to the pallet stored in the storage unit of any one of the forklifts together with the position and the orientation of the vehicle body by the travel control unit of another forklift via a network, the other forklift can transport the pallet. A group of forklifts.

Images