Cargo handling support system and on-board sensor device

Abstract

This technology provides the ability to identify unloading locations at low cost using existing cargo handling vehicles. [Solution] The cargo handling support system 1 is detachably attached to the cargo handling equipment of a cargo handling vehicle independently of the power system of the cargo handling vehicle and includes an on-board sensor device 100 that acquires sensor information indicating the position and orientation of the cargo handling equipment over time; a relative position calculation unit 213 that calculates a second relative position between each of a plurality of reference parts provided at predetermined positions in the work area, where the first relative positions of the plurality of reference parts are known and the cargo handling equipment; a trajectory correction unit 303 that corrects the movement trajectory of the cargo handling equipment obtained based on sensor information based on the first relative position and the second relative position; and a server 300 that includes a position identification unit 304 that identifies the unloading position on the corrected movement trajectory based on the time of unloading of cargo obtained based on sensor information and the corrected movement trajectory.

Description

Technical Field

[0001] The present invention relates to cargo handling support technology.

Background Art

[0002] Cargo handling vehicles such as forklifts can perform cargo handling at desired locations for the cargo to be transported. On the other hand, when the cargo is placed at an unspecified position by the cargo handling vehicle, the position of the placed cargo cannot be managed. This becomes an obstacle in improving the efficiency of the cargo handling work.

[0003] In contrast, in the technology described in Patent Document 1, the cargo storage position is detected based on the current running position information of the manned forklift truck calculated from the detection output of a movement distance sensor that detects the movement distances of the left and right wheels.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] Special sensors such as the movement distance sensor used in the technology described in Patent Document 1 are not provided in ordinary cargo handling vehicles. Therefore, the user needs to modify an existing cargo handling vehicle to be able to use such a special sensor, for example. In this case, the labor and installation costs for installing the special sensor increase. Alternatively, the user can newly purchase a cargo handling vehicle equipped with a special sensor, but this causes an increase in cost.

[0006] In view of the above, an object of the present invention is to provide a technology that enables the unloading position to be specified at low cost using an existing cargo handling vehicle.

Means for Solving the Problems

[0007] One aspect of the present invention is a cargo handling support system comprising: an on-board sensor device that can be detachably attached to a cargo handling device that performs cargo handling on the cargo handling vehicle and holds the cargo that has been handled, independently of the power system of the cargo handling vehicle, and acquires sensor information indicating the position and orientation of the cargo handling device over time; a relative position calculation unit that calculates a second relative position between each of a plurality of reference units provided at predetermined positions in the work area of ​​the cargo handling vehicle, wherein the first relative positions of the plurality of reference units are known; a server comprising: a trajectory correction unit that corrects the movement trajectory of the cargo handling device within the work area, which is determined based on the sensor information, based on the first relative position and the second relative position; and a position identification unit that identifies the unloading location on the corrected movement trajectory, based on the time of unloading of the cargo by the cargo handling vehicle, which is determined based on the sensor information, and the corrected movement trajectory.

[0008] Another aspect of the present invention is an on-board sensor device comprising: a sensor unit that acquires sensor information indicating over time the position and orientation of a cargo handling device that performs cargo handling on a cargo handling vehicle and holds the cargo that has been handled; a work detection unit that detects the unloading of the cargo by the cargo handling vehicle based on the sensor information and generates work information indicating the time of the detected unloading; and a trajectory calculation unit that calculates the movement trajectory of the cargo handling device based on the sensor information, wherein a plurality of reference units are provided at predetermined positions in the work area of ​​the cargo handling vehicle, the first relative positions of the plurality of reference units are known, the second relative position between each of the plurality of reference units and the cargo handling device is calculated, the movement trajectory is corrected based on the first relative position and the second relative position, and the device is detachably attached to the cargo handling device of the cargo handling vehicle independently of the power system of the cargo handling vehicle.

[0009] Furthermore, any combination of the above components, or any substitution of components or expressions of the present invention between methods, apparatus, systems, etc., is also valid as an embodiment of the present invention. [Effects of the Invention]

[0010] According to the present invention, it is possible to determine the unloading location at low cost using existing cargo handling vehicles. [Brief explanation of the drawing]

[0011] [Figure 1] This figure illustrates a schematic configuration of a cargo handling support system according to the first embodiment. [Figure 2] This figure shows an example of the application of the cargo handling support system according to the first embodiment. [Figure 3] This diagram illustrates a movement trajectory generated by sensor information. [Figure 4] This is a functional block diagram of the cargo handling support system according to the first embodiment. [Figure 5] This is a conceptual diagram illustrating a pause graph. [Figure 6] This diagram illustrates an example of pose graph optimization when the position of a single provision is measured at multiple points, and these measurements contain errors. [Figure 7] This diagram illustrates the corrected movement trajectory of a manned forklift's cargo handling equipment in the world coordinate system of an unmanned forklift, as well as the identified loading and unloading positions. [Figure 8] This is a sequence diagram showing the processing of the cargo handling support system according to the first embodiment. [Figure 9] This is a functional block diagram of the cargo handling support system according to the second embodiment. [Figure 10] This is a sequence diagram showing the processing of the cargo handling support system according to the second embodiment. [Modes for carrying out the invention]

[0012] The present invention will be described below with reference to the drawings, based on preferred embodiments. The same or equivalent components, members, and processes shown in each drawing will be denoted by the same reference numerals, and redundant descriptions will be omitted as appropriate. Furthermore, the embodiments are illustrative and not limiting to the invention, and not all features or combinations thereof described in the embodiments are necessarily essential to the invention.

[0013] First Embodiment Figure 1 illustrates a schematic configuration of a cargo handling support system 1 according to the first embodiment. The cargo handling support system 1 comprises an on-board sensor device 100, an information providing device 200, a server 300, and an unmanned forklift 500. For example, the on-board sensor device 100 is attached to the cargo handling device 610 of a manned forklift 400, which is a forklift operated by an operator.

[0014] The manned forklift 400 and the unmanned forklift 500 each comprise a body 601, front wheels 602 and rear wheels 603, a mast 604, and a cargo handling device 610. The mast 604 is located in front of the body 601. The cargo handling device 610 is a device that handles cargo and holds the handled cargo. The cargo handling device 610 includes a lifting body 605, forks 606, and a backrest 607. The lifting body 605 is driven by a power source such as a hydraulic actuator (not shown in Figure 1) and moves up and down along the mast 604. Forks 606 for supporting loads are attached to the lifting body 606. The lifting body 605 raises and lowers the forks 606 through its lifting motion. Backrests 607 are attached to the forks 606 to prevent cargo loaded on the forks 606 from falling behind the mast 604. The backrest 607 can be raised and lowered together with the forks 606 by raising and lowering the lifting body 605. In other words, in the cargo handling device 610, cargo handling is performed by raising and lowering the forks 606 and backrest 607 together with the lifting body 605. If the lifting body 605 moves not only vertically but also in the forward, backward, left, and right directions relative to the vehicle body 601, the forks 606 and backrest 607 will also move together with this movement.

[0015] The on-board sensor device 100 acquires sensor information related to the movement and loading / unloading operations of the manned forklift 400 over time. The on-board sensor device 100 is configured to transmit the acquired information to the server 300 via a predetermined communication network. The on-board sensor device 100 is an external sensor device configured to be attached to the loading / unloading device 610 of the manned forklift 400 independently of the power system of the manned forklift 400. For example, the on-board sensor device 100 has a mounting part (not shown) made of a magnet or adhesive sheet, and is detachably attached to the loading / unloading device 610 of the manned forklift 400 via the mounting part. The on-board sensor device 100 is purchased separately from the manned forklift 400 and retrofitted to the loading / unloading device 610 of the manned forklift 400.

[0016] The information providing device 200 provides information for correcting the movement trajectory of the cargo handling device 610 of the manned forklift 400. The information providing device 200 includes a first providing unit 210A and a second providing unit 210B, which are installed at predetermined locations in the work area 20 where the manned forklift 400 and the unmanned forklift 500 perform cargo handling operations. In this embodiment, the information providing device 200 is configured to transmit information to the server 300 via a predetermined communication network. The information providing device 200 may also transmit information via the on-board sensor device 100 by wireless communication with the on-board sensor device 100.

[0017] The server 300 corrects the movement trajectory of the cargo handling device 610 of the manned forklift 400 and identifies the unloading location where the cargo was placed by the manned forklift 400 on the corrected movement trajectory. The server 300 transmits a cargo handling instruction, including the identified unloading location, to the unmanned forklift 500, instructing the unmanned forklift 500 to perform cargo handling operations at the identified unloading location. The method for correcting the movement trajectory and the method for identifying the cargo unloading location will be described later.

[0018] The unmanned forklift 500 is an autonomously operating forklift. The unmanned forklift 500 is autonomously controlled, for example using artificial intelligence, to perform loading and unloading operations on cargo unloaded by the manned forklift 400. The manned forklifts 400 and 500 are examples of cargo handling vehicles used to transport goods such as pallets.

[0019] Figure 2 shows an example of the application of the cargo handling support system 1 of this embodiment. Here, it is assumed that an autonomous forklift 500 capable of autonomous driving and automatic cargo handling operations will be adopted to resolve the shortage of personnel in cargo handling operations and to improve efficiency. However, currently, the autonomous forklift 500 has poor work efficiency, and it may take a long time to unload all the cargo stored on the truck bed. As a result, for example, the departure of the truck to its next destination may be delayed, hindering the efficiency of cargo loading and unloading. Therefore, as shown in Figure 2, it is being considered that a manned forklift 400 will first transport the cargo 11 stored in the cargo storage area 10, such as the truck bed, and place it in another location within the work area 20, and then the autonomous forklift 500 will transport the placed cargo 11. This will allow the unloading of the cargo 11 stored on the truck bed to be completed quickly, thereby preventing delays in the truck's departure.

[0020] On the other hand, if the cargo 11 is placed in an unspecified location by the manned forklift 400, the unmanned forklift 500 may not be able to determine the unloading location of the cargo 11, which can lead to a further deterioration of work efficiency. To address this, in this embodiment, the vehicle-mounted sensor device 100 is installed on the cargo handling device 610 of the manned forklift 400, thereby acquiring the movement trajectory of the cargo handling device 610 of the manned forklift 400. This movement trajectory is used to determine the unloading location of the cargo 11.

[0021] The on-board sensor device 100 of this embodiment is independent of the power system of the manned forklift 400 and can be detachably attached to the cargo handling device 610 of the manned forklift 400. Therefore, it is possible to acquire the movement trajectory of the cargo handling device 610 of the manned forklift 400 at low cost and easily without requiring modifications to the existing forklift.

[0022] On the other hand, when using inexpensive sensors such as an inertial measurement unit (IMU) or a camera in the in-vehicle sensor device 100, errors can accumulate in the movement trajectory detected by the in-vehicle sensor device 100 due to sensor drift, and these errors may increase over time.

[0023] Figure 3 illustrates a movement trajectory generated by sensor information from the on-board sensor device 100. Figure 3 shows the movement trajectory T1 of the cargo handling device 610 of the manned forklift 400 in the local coordinate system of the manned forklift 400. In Figure 3, P1, P3, and P5 are loading positions, and P2, P4, and P6 are unloading positions. In Figure 3, the cargo handling device 610 of the manned forklift 400 moves in the order P1→P2→P3→P4→P5→P6. As shown in Figure 3, the movement trajectory T1 generated by sensor information from the on-board sensor device 100 may contain errors due to drift of the on-board sensor device 100. To address this sensor drift issue, using high-precision sensors such as LiDAR (Light Detection and Ranging) in the on-board sensor device 100 would increase costs. It is also conceivable that cargo could be fitted with wireless tags and its location could be determined using wireless tag readers installed in the environment, but incoming cargo may not always have wireless tags attached.

[0024] To address this sensor drift, in this embodiment, the movement trajectory is corrected using an information providing device 200 that can be installed at low cost within the work area 20. This correction method will be described later. This makes it possible to obtain an appropriate movement path for the manned forklift 400, and since it is not necessary to provide correction means such as magnetic tape throughout the entire work area, as in the technology described in Patent Document 1, for example, costs can be reduced.

[0025] In this embodiment, since the unloading location of cargo can be identified using the vehicle-mounted sensor device 100 and information providing device 200, which can be used at low cost, it is possible to improve the efficiency of cargo handling operations at a low cost.

[0026] Based on the above, the cargo handling support system 1 of this embodiment will now be described in more detail.

[0027] Figure 4 is a functional block diagram of the cargo handling support system 1 of the first embodiment. Each functional block shown in the figure can be realized hardware-wise using components such as a computer processor, CPU, and memory, as well as electronic circuits and mechanical devices, and software-wise using computer programs, etc. However, here, the functional blocks are depicted as being realized through the coordination of these components. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways through combinations of hardware and software.

[0028] The in-vehicle sensor device 100 comprises a first sensor unit 110, a processing unit 120, a storage unit 130, a communication unit 140, and a battery 150.

[0029] The first sensor unit 110 acquires sensor information indicating the position and orientation of the cargo handling device 610 of the manned forklift 400 over time. This sensor information makes it possible to acquire the position and orientation of cargo held by the cargo handling device 610. The first sensor unit 110 in this embodiment includes an IMU (Inertial Measurement Unit) containing a three-axis acceleration sensor and a gyro sensor, and acquires sensor information by three-dimensionally measuring the acceleration and angular velocity of the manned forklift 400. The first sensor unit 110 stores the acquired sensor information in the storage unit 130.

[0030] The processing unit 120 performs various calculations in the in-vehicle sensor device 100. The processing unit 120 includes a work detection unit 121 and a trajectory calculation unit 122.

[0031] The work detection unit 121 detects the start of cargo handling operations, loading, unloading, and completion of cargo handling operations based on sensor information, and generates work information indicating the detected work content and the time of detection. For example, the work information includes a work list that links the detected work content and the time of detection. For example, the work detection unit 121 detects these based on the acceleration in the sensor information from the IMU of the first sensor unit 110. For example, the work detection unit 121 detects the start of cargo handling operations when a change in acceleration is measured after a predetermined period of time during which there has been no change in the measured acceleration value. For example, the work detection unit 121 detects the completion of cargo handling operations when a predetermined period of time has passed during which there has been no change in the measured acceleration value.

[0032] The detection method for loading and unloading cargo will now be explained. For example, the work detection unit 121 detects loading and unloading cargo based on the changes in acceleration that occur during loading and unloading. When loading and unloading cargo is performed by a forklift, there is a moment when the forks and cargo come into contact. Also, the movement of the forklift when inserting or removing the forks from cargo such as pallets is different from other movements. By detecting the acceleration that occurs during loading and unloading cargo, it is possible to detect loading and unloading cargo, respectively.

[0033] Furthermore, if detecting cargo loading and unloading is difficult using only scalar values ​​such as acceleration magnitude, machine learning can be used. By recording multiple time-series changes in acceleration caused by cargo loading and unloading, and training a machine learning model in conjunction with time-series changes in acceleration under normal conditions, it becomes possible to appropriately detect cargo loading and unloading.

[0034] Furthermore, the on-board sensor device 100 may also include switches and photoelectric sensors. When cargo is unloaded by a forklift, the forks are inserted into the pallet. If physical contact or changes in brightness caused by the insertion of the forks into the pallet can be detected, the presence or absence of cargo can be detected.

[0035] Furthermore, the start of cargo handling operations, loading, unloading, and completion of cargo handling operations may also be detected when the operator operates a user interface (button, touch panel, etc.) provided on the on-board sensor device 100, or when a predetermined operation is performed on the operator's mobile terminal.

[0036] The trajectory calculation unit 122 calculates the movement trajectory of the cargo handling device 610 of the manned forklift 400 based on sensor information. The movement trajectory of the cargo handling device 610 indicates the time-dependent position of the cargo handling device 610 in the local coordinate system of the manned forklift 400. For example, the trajectory calculation unit 122 calculates the movement trajectory by integrating the measured acceleration values ​​in the sensor information of the IMU of the first sensor unit 110 over time. Alternatively, the movement trajectory may be calculated by estimating the time-dependent position of the cargo handling device 610 from the sensor information of the IMU using methods such as dead reckoning or a Kalman filter.

[0037] The memory unit 130 stores various data such as algorithms and sensor information for executing processing in the in-vehicle sensor device 100.

[0038] The battery 150 supplies power to the first sensor unit 110, the processing unit 120, the storage unit 130, and the communication unit 140. The battery 150 is charged using a predetermined charging device for the in-vehicle sensor device 100.

[0039] The information providing device 200 of this embodiment includes a first providing unit 210A located at a first position within the work area 20, and a second providing unit 210B located at a second position different from the first position within the work area 20. When the first providing unit 210A and the second providing unit 210B are not specifically distinguished, they may simply be referred to as the providing unit 210. The providing unit 210 is installed at a predetermined position (first position or second position) near a passage that is repeatedly passed through by a manned forklift 400 transporting cargo within the work area 20. The first relative position between the first providing unit 210A and the second providing unit 210B is assumed to be known. The information providing device 200 of this embodiment detects the manned forklift 400. In this context, "detecting the manned forklift 400" includes not only detecting the entire manned forklift 400, but also detecting parts of the manned forklift 400 such as the cargo handling equipment 610 and the on-board sensor device 100 installed on the manned forklift 400. The information providing device 200 in this embodiment provides the server 300 with a second relative position, which is the relative position between the detected cargo handling equipment 610 of the manned forklift 400 and the providing unit 210 that detected the manned forklift 400, as information for correcting the movement trajectory.

[0040] The configuration of the information providing device 200 varies depending on the sensor used in the first sensor unit 110. For example, when an IMU is used as the first sensor unit 110, as in this embodiment, the providing unit 210 may use a camera, LiDAR (Light Detection and Ranging), photoelectric sensor, etc., as the second sensor unit 211 described later. The providing unit 210 in this embodiment is an example of a reference unit.

[0041] The supply unit 210 includes a second sensor unit 211, an object detection unit 212, a relative position calculation unit 213, and a communication unit 214. The second sensor unit 211 includes a detection range such as a passage through which a manned forklift 400 passes during cargo handling operations. The second sensor unit 211 in this embodiment includes a camera that captures subjects within the detection range and generates image information.

[0042] The object detection unit 212 detects the manned forklift 400 from the output data of the second sensor unit 211 and acquires its detection position as the position of the cargo handling device 610. In this embodiment, the object detection unit 212 detects the manned forklift 400 from the image information of the second sensor unit 211 (camera) and acquires its detection position as the position of the cargo handling device 610. The object detection unit 212 may also be deemed to have detected the manned forklift 400 and acquired its detection position by detecting the on-board sensor device 100 attached to the manned forklift 400 and acquiring its detection position. Alternatively, the on-board sensor device 100 may be provided with a two-dimensional code such as an AR marker, and the detection unit may determine that the manned forklift 400 has been detected in response to the detection of an AD marker and acquire its detection position. This can improve the accuracy of detecting the position of the manned forklift 400. The second sensor unit 211 may be, for example, a LiDAR or a photoelectric sensor.

[0043] The relative position calculation unit 213 calculates a second relative position, which is the relative position between the detected cargo handling device 610 of the manned forklift 400 and the supply unit 210 that detected the manned forklift 400. For example, if the first supply unit 210A detects the manned forklift 400, it calculates the second relative position between the detected cargo handling device 610 of the manned forklift 400 and the first supply unit 210A. Also, for example, if the second supply unit 210B detects the manned forklift 400, it calculates the second relative position between the detected cargo handling device 610 of the manned forklift 400 and the second supply unit 210B. The second relative position here is, for example, the relative position of the first supply unit 210A or the second supply unit 210B with respect to the detected cargo handling device 610 of the manned forklift 400.

[0044] In response to the calculation of the second relative position, the communication unit 214 transmits the calculated second relative position and the time when the manned forklift 400 was detected to the server 300 as correction information.

[0045] The server 300 includes a storage unit 301, a graph generation unit 302, a trajectory correction unit 303, a location identification unit 304, a cargo handling instruction unit 305, and a communication unit 306.

[0046] The memory unit 301 has already stored a known first relative position.

[0047] The graph generation unit 302 generates a pose graph showing the position and orientation of the cargo handling device 610 of the manned forklift 400 at multiple points in time, based on the known first relative position stored in the storage unit 301, the movement trajectory acquired via the communication unit 306, and correction information (see Figure 5(b)).

[0048] Figure 5 is a conceptual diagram showing an example of a pause graph. Figure 5(a) shows an example of the true movement trajectory of the cargo handling device 610 of the manned forklift 400, and Figure 5(b) shows an example of the movement trajectory including errors due to sensor drift of the on-board sensor device 100. The pause graph is a graph in which the position and orientation of the cargo handling device 610 of the manned forklift 400 and the position of the supply unit 210 that detected the manned forklift 400 at each time point are used as nodes, and the vectors connecting them are used as edges (constraints).

[0049] In Figure 5(a), node x t0 ~x t7 Nodes x indicate the true position and orientation of the loading / unloading device 610 of the manned forklift 400 at each time t0 to t7, and nodes m0 and m1 indicate the true positions of the first and second positions where the supply section 210 is provided. t1 and x t5 The first supply unit 210A and the second supply unit 210B respectively detect the manned forklift 400, and node x t1 and x t5 Assume that the relative positions of nodes m0 and m1 with respect to the given node are obtained as correction information. In the example in Figure 5, the known first relative position and the second relative position included in the correction information are assumed to be error-free.

[0050] In Figure 5(b), node x' t0 ~x' t7shows the position and orientation of the cargo handling device 610 of the manned forklift 400 measured by the in-vehicle sensor device 100 at each time t0 to t7. Node x' t0 ~x' t7 contains errors. The graph generation unit 302 adjusts the nodes x' of the pose graph to reflect the position and orientation of the cargo handling device 610 of the manned forklift 400 at each time in the movement trajectory t0 ~x' t7 to create.

[0051] In Fig. 5(b), node m 00 represents the position of the cargo handling device 610 of the manned forklift 400 at node x' t1 and the position of the first providing unit 210A calculated from the second relative position of the first providing unit that is detected for the cargo handling device 610 of the manned forklift 400 in the correction information. When the time when the first providing unit 210A detects the manned forklift 400 is set as t1, the graph generation unit 302 calculates from the second relative position of the first providing unit 210A with respect to the position and orientation of node x' t1 at time t1, and creates node m 00 to create.

[0052] In Fig. 5(b), node m 10 is the position of the second providing unit 210B calculated from the known first relative position between the position of node m 00 and the providing unit 210. The graph generation unit 302 creates node m 00 from the known first relative position between the providing units 210 with respect to the position of node m 10 to create.

[0053] In Fig. 5(b), node m 11 is node x' t5The position and orientation of the loading / unloading device 610 of the manned forklift 400 and the position of the second supply unit 210B calculated from the second relative position of the second supply unit 210B with respect to the detected loading / unloading device 610 of the manned forklift 400 in the correction information are shown. The graph generation unit 302, when the time t5 is the time when the second supply unit 210B detected the manned forklift 400, shows node x' at time t5. t5 From the second relative position of the second providing part 210B, with reference to the position and orientation of the node m 11 Create.

[0054] In Figure 5(b), node m 01 , node m 11 The position of the first providing unit 210A is calculated from the position of node m and the known first relative position between the providing unit 210. The graph generation unit 302 generates node m 11 From the known first relative position between the providing parts 210 with respect to the position of node m 10 Create.

[0055] As shown in Figure 5(b), node x' t1 Node m indicates the position of the first supply unit 210A, which is assumed to be based on its position and orientation. 00 and node x' t5 Node m indicates the position of the first supply unit 210A, which is assumed to be based on its position and orientation. 01 The position is shifted. Also, node x' t1 Node m indicates the position of the second supply unit 210B, which is assumed to be based on the position and orientation of the node. 10 and node x' t5 Node m indicates the position of the second supply unit 210B, which is assumed to be based on the position and orientation of the node. 11 The position is shifted. Node x' t0 ~x' t7 If it does not include the error, node m 00 and node m 01 , and node m 10 and node m 11 The positions of each should coincide. Therefore, node x' t0 ~x' t7 It can be seen that there is an error.

[0056] Returning to Figure 4, the trajectory correction unit 303 corrects the movement trajectory based on the position and orientation of the manned forklift 400's loading / unloading device 610 at multiple points in time in the pose graph, which has been adjusted by optimizing the pose graph.

[0057] Figure 5(c) shows an example of optimizing the pause graph to minimize the error between nodes. Node m in Figure 5(b) 00 and node m 01 Positional deviation and node m 10 and node m 11 To minimize the positional misalignment, the relative positional relationship between nodes is transformed as shown in Figure 5(c). In pose graph optimization, node x t The edges in between are deformed to minimize the error, while node x t1 and node m 00 Edge between node x t5 and node m 11 Edges and nodes m between them 00 and node m 11 The relative positions of each node are deformed, while the edges between them remain unchanged.

[0058] The trajectory correction unit 303 corrects the movement trajectory to reflect the position and orientation of the loading / unloading device 610 of the manned forklift 400 at each point in time in the optimized pose graph, which is achieved by deforming the edges in such a way as to minimize edge errors. This makes it possible to obtain a movement trajectory with reduced errors caused by sensor drift of the on-board sensor device 100.

[0059] In the example in Figure 5, the explanation was based on the assumption that there were no errors in the measurement. However, here, using Figure 6, we will explain an example of pose graph optimization when the position of a single supply unit 210 is measured at multiple points and the measurement contains errors. Figure 6(a) shows node x, which indicates the true position and orientation of the cargo handling device 610 of the manned forklift 400 at times t1 to t3. t1 ~x t3This shows node m, which indicates the true position of the supply unit 210. As shown in Figure 6(a), all nodes x at times t1 to t3 t1 ~x t3 The position of the supply unit 210 is measured.

[0060] Figure 6(b) shows the position of the measurement unit at each time t1 to t3, based on the position and orientation of the loading / unloading device 610 of the manned forklift 400, including the error measured by the on-board sensor device 100, with node m indicating the position of the measurement unit. t1 ~m t3 This is shown. Here, it is assumed that the degree of measurement error of the supply unit 210 has been measured and determined in advance. Node m t1 ~m t3 The dotted ellipse surrounding the area represents the range of error. Therefore, during pose graph optimization, the edges (constraints) for measurement of the providing unit 210 are deformed only within this range of error.

[0061] Figure 6(c) shows the position and orientation of the manned forklift 400 at each time t1 to t3, obtained by pose graph optimization. Figure 6(d) shows node x t3 The position of the measurement unit measured by node x shows how it changed before and after optimization. t3 This is represented in the local coordinate system. As shown in Figure 6(d), it can be seen that the deformation of the constraint on the measurement of the providing part 210 is within the range of error.

[0062] Returning to Figure 4, the positioning unit 304 identifies the unloading location of the cargo placed by the manned forklift 400 based on the corrected movement trajectory and the unloading time in the work information. For example, the positioning unit 304 converts the corrected movement trajectory, which is represented in the local coordinate system of the manned forklift 400, to the world coordinate system of the unmanned forklift 500. The positioning unit 304 identifies the unloading location of the cargo in the world coordinate system by identifying the position corresponding to the unloading time in the movement trajectory in the world coordinate system.

[0063] Furthermore, the position identification unit 304 may identify the cargo's loading location in the world coordinate system by identifying the position corresponding to the loading time in the work information on the movement trajectory in the world coordinate system. For example, when transporting cargo that was loaded on the bed of a truck, it becomes possible to identify where the cargo that was originally on the truck bed was placed. Therefore, if the arrangement information of the cargo on the truck bed is provided in advance by an inbound delivery slip or the like, it becomes possible to determine the location of the received cargo in detail. In addition, if the cargo is assigned identification code information such as a barcode and the manned forklift 400 is equipped with a means to read it, it becomes possible to identify the cargo by comparing the time the cargo's identification code information was read with the loading time and unloading time.

[0064] Figure 7 illustrates the corrected movement trajectory T2 of the cargo handling device 610 of the manned forklift 400 in the world coordinate system of the unmanned forklift 500, as well as the identified loading and unloading positions. In Figure 7, P'1, P'3, and P'5 are loading positions, and P'2, P'4, and P'6 are unloading positions. As shown in Figure 7, the manned forklift 400 is thought to move in the order of positions P'1 → P'2 → P'3 → P'4 → P'5 → P'6, bypassing obstacles 30 such as trucks and pallet storage racks. By identifying the unloading positions, it is possible to determine the travel path of the unmanned forklift 500 so that it heads towards P'2, P'4, and P'6.

[0065] The cargo handling instruction unit 305, for example, in response to the identification of the cargo unloading location, transmits cargo handling instructions, including the unloading location in the world coordinate system, via the communication unit 306.

[0066] The unmanned forklift 500 comprises a travel path generation unit 501, a control unit 502, a drive unit 503, and a communication unit 504. The travel path generation unit 501 generates a travel path using known travel path generation techniques based on the unloading locations included in the cargo handling instructions received from the server 300 via the communication unit 504. For example, the travel path generation unit 501 generates a travel path that passes through a specified unloading location. For example, the travel path generation unit 501 generates a travel path that travels back and forth between each unloading location and the destination location of the unloaded cargo.

[0067] The control unit 502 controls the drive units 503 of the unmanned forklift 500, such as the travel motor and steering actuator (not shown), to drive the unmanned forklift 500 according to a generated travel path, based on the position information of the unmanned forklift 500 obtained from sensors such as GNSS (Global Navigation Satellite System) and LIDAR. Furthermore, when cargo is detected at the unloading position by sensors such as LIDAR and cameras mounted on the unmanned forklift 500, the control unit controls the drive units 503, such as actuators that drive the forks, to perform predetermined operations such as inserting the forks of the unmanned forklift 500.

[0068] Figure 8 is a sequence diagram showing the processing of the cargo handling support system 1 of this embodiment. In each step of Figure 8, the on-board sensor device 100 is assumed to be continuously acquiring sensor information, and the information providing device 200 is assumed to be continuously acquiring image information.

[0069] In step S101, the work detection unit 121 determines whether it has detected the start of cargo handling operations by the manned forklift 400 based on sensor information, etc. If the start of cargo handling operations is detected (Yes in step S101), the process proceeds to step S102. If the start of cargo handling operations is not detected (No in step S101), the process returns to step S101, and step S101 is repeatedly executed until the start of cargo handling operations is detected.

[0070] In step S102, the work detection unit 121 determines whether it has detected the completion of the loading / unloading operation of the manned forklift 400 based on sensor information, etc. If the completion of the loading / unloading operation is detected (Yes in step S102), the process proceeds to step S103. If the completion of the loading / unloading operation is not detected (No in step S102), the process returns to step S102, and step S102 is repeatedly executed until the completion of the loading / unloading operation is detected.

[0071] In step S103, the work detection unit 121 generates work information based on the sensor information.

[0072] In step S104, the trajectory calculation unit 122 calculates the movement trajectory of the cargo handling device 610 of the manned forklift 400 based on the sensor information.

[0073] In step S105, the communication unit 140 transmits work information and movement trajectory to the server 300. After step S105, the processing of the on-board sensor device 100 is completed, and the on-board sensor device 100 waits until the start of cargo handling work is detected in step S101.

[0074] In step S106, the object detection unit 212 of the information providing device 200 determines whether it has detected a manned forklift 400 based on the image information from the second sensor unit 211. If a manned forklift 400 is detected (Yes in step S106), the process proceeds to step S107. If a manned forklift 400 is not detected (No in step S106), the process returns to step S106, and step S106 is repeatedly executed until a manned forklift 400 is detected.

[0075] In step S107, the object detection unit 212 acquires the detection position of the manned forklift 400 as the position of the cargo handling device 610 based on the image information.

[0076] In step S108, the relative position calculation unit 213 calculates a second relative position based on the acquired detection position of the manned forklift 400 and the position of the supply unit 210 that detected the manned forklift 400.

[0077] In step S109, the communication unit 214 transmits correction information to the server, including the calculated second relative position and the time when the manned forklift 400 was detected. After step S109, the information providing device 200 finishes processing and waits until the manned forklift 400, which is the target of detection in step S106, is detected. That is, when the manned forklift 400 passes through the detection range of the second sensor unit 211 again and is detected, another correction information generated based on that detection result is transmitted to the server 300.

[0078] In step S110, the graph generation unit 302 generates a pause graph based on the known first relative position, the movement trajectory received via the communication unit 306, and the correction information.

[0079] In step S111, the trajectory correction unit 303 corrects the movement trajectory by optimizing the pose graph.

[0080] In step S112, the positioning unit 304 identifies the unloading location of the cargo based on the corrected movement trajectory and the unloading time in the work information.

[0081] In step S113, the cargo handling instruction unit 305 transmits cargo handling instructions, including the unloading location, via the communication unit 306. After step S113, the server 300 finishes processing and waits until it receives, for example, the work information and movement trajectory from step S105 and the correction information from step S109.

[0082] In step S114, the travel path generation unit 501, in response to receiving a cargo handling instruction via the communication unit 504, generates a travel path for the unmanned forklift 500 based on the unloading location included in the cargo handling instruction.

[0083] In step S115, the control unit 502 controls the drive unit 503 to move the unmanned forklift 500 according to the generated travel path and perform cargo handling operations.

[0084] In step S116, the control unit 502 determines whether the loading and unloading operations of the unmanned forklift 500 are complete. For example, the control unit 502 determines that the loading and unloading operations of the unmanned forklift 500 are complete when all cargo at the unloading location has been transported and the end of the travel path has been reached. If the loading and unloading operations of the unmanned forklift 500 are complete (Yes in step S116), the processing of the unmanned forklift 500 ends. If the loading and unloading operations of the unmanned forklift 500 are not complete (No in step S116), the process returns to step S115, and step S116 is repeatedly executed until the loading and unloading operations of the unmanned forklift 500 are complete.

[0085] As described above, according to this embodiment, the manned forklift 400 can be attached to the manned forklift 400 independently of its electrical system and can be detachably mounted on the manned forklift 400. Therefore, it is possible to acquire the movement trajectory of the cargo handling device 610 of the manned forklift 400 at low cost and easily, without requiring any modification of the existing forklift. Furthermore, according to this embodiment, since the movement trajectory can be corrected using an information providing device 200 that can be installed at low cost within the work area 20, it is possible to identify the cargo unloading position at low cost. Moreover, since both the movement trajectory acquisition function and the cargo handling operation detection function can be realized by the sensor information output from the first sensor unit 110, it is possible to effectively reduce costs.

[0086] Second Embodiment A second embodiment of the present invention will be described below. In the drawings and description of the second embodiment, components and members that are the same or equivalent as those in the first embodiment will be denoted by the same reference numerals. Descriptions that overlap with those of the first embodiment will be omitted as appropriate, and the description will focus on the configurations that differ from those of the first embodiment.

[0087] Figure 9 is a functional block diagram of the cargo handling support system 1 of the second embodiment. As shown in Figure 9, the first providing unit 210A and the second providing unit 210B of the information providing device 200 each include an AR marker 215. The information providing device 200 of this embodiment provides information indicating a reference position for calculating a second relative position, such as the AR marker 215, as information for correcting the movement trajectory.

[0088] The first sensor unit 110 of the second embodiment includes a camera that generates image information as sensor information. In this case, the work detection unit 121 can detect cargo handling operations using techniques such as template matching or deep learning from the image information. If the on-board sensor device 100 alone does not have sufficient computing resources for detecting cargo handling operations, the image information may be sent to the server 300, and the server 300 may perform the cargo handling operation detection process. In addition, the trajectory calculation unit 122 can measure the movement trajectory by calculating the amount of movement between two consecutive images in the image information.

[0089] The processing unit 120 of the in-vehicle sensor device 100 includes an object detection unit 212 and a relative position calculation unit 213. The object detection unit 212 in this embodiment detects the AR markers 215 of the first providing unit 210A and the second providing unit 210B based on image information, which is sensor information from the first sensor unit 110, and acquires their detection positions. A known method is used for detecting the position of the AR markers using a camera. The relative position calculation unit 213 in this embodiment calculates a second relative distance between the cargo handling device 610 of the manned forklift 400 and the detected AR marker 215.

[0090] Figure 10 is a sequence diagram showing the processing of the cargo handling support system 1 of the second embodiment. Steps S201, S206, S207, and S209-S215 in Figure 10 are the same as steps S101, S103, S104, and S110-S116 in Figure 8, except for points that are specifically mentioned, so their explanation is omitted.

[0091] In step S202, the object detection unit 212 of the in-vehicle sensor device 100 determines whether it has detected the providing unit 210. For example, the object detection unit 212 determines that it has detected the providing unit 210 if it detects the AR marker 215 provided on the providing unit 210 based on the image information, which is the sensor information of the first sensor unit 110. If the providing unit 210 is detected (Yes in step S202), the process proceeds to step S203. If the providing unit 210 is not detected (No in step S202), the process proceeds to step S205.

[0092] In step S203, the object detection unit 212 of the in-vehicle sensor device 100 acquires the detection position of the AR marker 215 as the detection position of the providing unit 210.

[0093] In step S204, the relative position calculation unit 213 of the in-vehicle sensor device 100 calculates a second relative position based on the detected position of the providing unit 210 that was acquired.

[0094] In step S205, the work detection unit 121 determines whether it has detected the completion of the loading / unloading operation of the manned forklift 400. If it has detected the completion of the loading / unloading operation (Yes in step S205), the process proceeds to step S206. If it has not detected the completion of the loading / unloading operation (No in step S205), the process returns to step S202, and steps S202 to S205 are repeatedly executed until the completion of the loading / unloading operation is detected.

[0095] After steps S206 and S207, in step S208, the communication unit 140 transmits correction information, work information, and movement trajectory to the server 300. The correction information here includes the calculated second relative position and the time when the providing unit 210 was detected.

[0096] In the second embodiment as well, the information providing device 200 can be installed in the work area 20 at low cost, making it possible to identify the cargo unloading location at low cost.

[0097] Those skilled in the art will understand that the present invention is not limited to the embodiments described above, and that various design changes and modifications are possible, and that such modifications also fall within the scope of the present invention. Such modifications will be described below.

[0098] In the embodiments, a forklift was used as an example of a cargo handling vehicle, but the application of the present invention is not limited to this, and can be applied to various cargo handling vehicles such as hand lifts (hand pallets), AGVs (Automatic Guided Vehicles), and fork loaders (wheel loaders with forks).

[0099] In the embodiment, an example is shown in which two providing sections 210 are provided, but the application of the present invention is not limited to this, and two or more providing sections 210 may be provided at different locations within the work area 20. In this case, the first relative positions of each of the two or more providing sections 210 can be measured in advance and known.

[0100] The trajectory calculation unit 122 and the work detection unit 121 may be provided on the server 300. In this case, the sensor information from the first sensor unit 110 is transmitted to the server 300, and the server 300 calculates the movement trajectory and generates work information based on that sensor information.

[0101] The object detection unit 212 and the relative position calculation unit 213 may be provided on the server 300. In this case, the sensor information from the first sensor unit 110 or the second sensor unit 211 is transmitted to the server 300, and the server 300 calculates the detection position and the second relative position based on that sensor information.

[0102] The first sensor unit 110 may be a combination of a camera and an IMU. In this case, the movement trajectory may be calculated using a method such as VIO (Visual Inertial Odometer) using the camera and IMU.

[0103] The work information is intended to show the detected work content and detection time among the start of cargo handling operations, unloading, and completion of cargo handling operations, but is not limited to this; it is sufficient to show at least the detection time of unloading.

[0104] Forklift 400 may be an autonomous, unmanned forklift, while forklift 500 may be a manned forklift operated by an operator.

[0105] In summary, the cargo handling support system 1 of the embodiment includes an on-board sensor device 100 that can be detachably attached to a cargo handling device 610 that performs cargo handling on a cargo handling vehicle and holds the handled cargo, independently of the power system of the cargo handling vehicle, and acquires sensor information indicating the position and orientation of the cargo handling vehicle over time; a relative position calculation unit 213 that calculates a second relative position between each of a plurality of reference parts provided at predetermined positions in the work area of ​​the cargo handling vehicle, where the first relative positions of the plurality of reference parts are known and the cargo handling device; a server 300 which includes a trajectory correction unit 303 that corrects the movement trajectory of the cargo handling device in the work area, which is determined based on the sensor information, based on the first relative position and the second relative position; and a position identification unit 304 that identifies the unloading position on the corrected movement trajectory, based on the time of unloading of cargo by the cargo handling vehicle, which is determined based on the sensor information, and the corrected movement trajectory. This configuration allows for the device to be attached to the manned forklift 400 independently of its electrical system and detachably. Therefore, it is possible to acquire the movement trajectory of the manned forklift 400 at low cost and with ease, without requiring any modifications to the existing forklift. Furthermore, it becomes possible to pinpoint the unloading location of cargo at low cost.

[0106] The cargo handling support system 1 of this embodiment further includes a graph generation unit 302 that generates a pose graph showing the position and orientation of the cargo handling device at multiple points in time based on the movement trajectory, a first relative position, and a second relative position, and a trajectory correction unit 303 corrects the movement trajectory based on the position and orientation of the cargo handling device shown in the adjusted pose graph by optimizing the pose graph. With this configuration, the error in the movement trajectory can be minimized, making it possible to accurately determine the unloading position of the cargo.

[0107] In this embodiment, the server 300 further comprises a cargo handling instruction unit 305 that transmits cargo handling instructions including the unloading location, and the cargo handling support system 1 further comprises an unmanned cargo handling vehicle that generates a travel route based on the unloading location in the cargo handling instructions and is capable of autonomously traveling within the work area according to the travel route. With this configuration, the unmanned cargo handling vehicle can grasp the unloading location of the cargo, thus suppressing a deterioration in the efficiency of cargo handling operations.

[0108] In this embodiment, the position identification unit 304 identifies the location of the cargo pickup on the corrected movement trajectory based on the time of cargo pickup by the cargo handling vehicle, which is determined based on sensor information, and the corrected movement trajectory. With this configuration, it is possible to determine where the picked-up cargo was unloaded. Therefore, if information on the arrangement of cargo on the truck bed is provided in advance by an inbound delivery slip or the like, it becomes possible to determine the location of the incoming cargo in detail.

[0109] In this embodiment, the on-board sensor device 100 includes an inertial measurement unit, and the reference unit includes a camera, an object detection unit 212 that detects a cargo handling vehicle based on image information from the camera, a relative position calculation unit 213 that calculates a second relative position between the detected cargo handling device 610 of the cargo handling vehicle and the reference unit that detected the cargo handling vehicle, and a communication unit 214 that transmits the second relative position to the server 300. With this configuration, it is possible to obtain an appropriate movement trajectory at low cost.

[0110] In this embodiment, an AR marker 215 is provided in the reference section, and the in-vehicle sensor device 100 includes a camera, an object detection unit 212 that detects the AR marker 215 from the image information of the camera, a relative position calculation unit 213 that calculates a second relative position between the detected AR marker 215 and the cargo handling device 610 of the cargo handling vehicle that detected the AR marker 215, and a communication unit 140 that transmits the second relative position to the server 300. With this configuration, it is possible to obtain an appropriate movement trajectory at low cost.

[0111] The vehicle-mounted sensor device 100 of the embodiment includes a sensor unit 110 that acquires sensor information indicating the position and orientation of a cargo handling device 610 that performs cargo handling and holds the handled cargo in a cargo handling vehicle that transports cargo; a work detection unit 121 that detects the unloading of cargo by the cargo handling vehicle based on the sensor information and generates work information indicating the time of the detected unloading; and a trajectory calculation unit 122 that calculates the movement trajectory of the cargo handling device 610 based on the sensor information. The vehicle-mounted sensor device 100 is detachably attached to the cargo handling device 610 of the cargo handling vehicle, and is independent of the power system of the cargo handling vehicle. This configuration makes it possible to provide an in-vehicle sensor device 100 that is inexpensive, can be easily installed on a cargo handling vehicle, and enables obtaining appropriate movement trajectories and unloading positions. [Explanation of Symbols]

[0112] 100 In-vehicle sensor devices 110 First Sensor Unit 120 Processing Unit 121 Work detection unit 122 Trajectory calculation section 130, 301 Storage section 140, 214, 306, 504 Communications Department 150 batteries 200 Information provision device 210 Provision Department 211 Second Sensor Unit 212 Object detection unit 213 Relative position calculation unit 300 servers 302 Graph Generation Unit 303 Trajectory correction section 304 Location identification part 305 Cargo Handling Instructions Department 400, 500 forklifts 501 Travel Path Generation Unit 502 Control Unit 503 Drive Unit

Claims

1. An on-board sensor device that is detachably attached to a cargo handling device that operates independently of the power system of a cargo handling vehicle and holds the cargo that has been handled, and acquires sensor information indicating the position and orientation of the cargo handling device over time, A relative position calculation unit that calculates a second relative position between each of the multiple reference points, where the first relative positions of the multiple reference points are known, and the cargo handling device, wherein the multiple reference points are provided at predetermined locations in the work area of ​​the cargo handling vehicle, It is a server, A trajectory correction unit corrects the movement trajectory of the cargo handling device within the work area, which is determined based on the sensor information, based on the first relative position and the second relative position. A position identification unit that identifies the unloading location on the corrected movement trajectory based on the time of unloading by the cargo handling vehicle determined based on the sensor information and the corrected movement trajectory, Including a server, Equipped with, Cargo handling support system.

2. The system further comprises a graph generation unit that generates a pose graph showing the position and orientation of the cargo handling device at multiple points in time, based on the aforementioned movement trajectory, the first relative position, and the second relative position. The trajectory correction unit corrects the movement trajectory based on the position and orientation of the cargo handling device shown by the pose graph, which has been adjusted by optimizing the pose graph. The cargo handling support system according to claim 1.

3. The server further comprises a cargo handling instruction unit that transmits cargo handling instructions including the unloading location, The cargo handling support system further comprises an unmanned cargo handling vehicle that generates a travel route based on the unloading location during the cargo handling instruction and is capable of autonomously traveling within the work area according to the travel route. The cargo handling support system according to claim 1.

4. The cargo handling support system according to claim 1, wherein the position determination unit determines the location of the cargo pickup on the corrected movement trajectory based on the time of cargo pickup by the cargo handling vehicle determined based on the sensor information and the corrected movement trajectory.

5. The in-vehicle sensor device includes an inertial measurement unit, The cargo handling support system according to claim 1, wherein the reference unit comprises a camera, an object detection unit that detects the cargo handling vehicle based on image information from the camera, a relative position calculation unit that calculates the second relative position between the cargo handling device of the detected cargo handling vehicle and the reference unit that detected the cargo handling vehicle, and a communication unit that transmits the second relative position to the server.

6. An AR marker is provided in the aforementioned reference section. The cargo handling support system according to claim 1, wherein the in-vehicle sensor device comprises a camera, an object detection unit that detects the AR marker from image information of the camera, a relative position calculation unit that calculates the second relative position between the detected AR marker and the cargo handling device of the cargo handling vehicle that detected the AR marker, and a communication unit that transmits the second relative position to the server.

7. A sensor unit that acquires sensor information indicating the position and orientation of a cargo handling device that holds the cargo after it has been handled in a cargo handling vehicle, A work detection unit detects the unloading of the cargo by the cargo handling vehicle based on the sensor information and generates work information indicating the time of the detected unloading. A trajectory calculation unit calculates the movement trajectory of the cargo handling device based on the sensor information, Equipped with, A plurality of reference points are provided at predetermined locations in the work area of ​​the cargo handling vehicle, and the second relative position between each of the plurality of reference points, where the first relative position between the plurality of reference points is known, and the cargo handling device is calculated. Based on the first relative position and the second relative position, the movement trajectory is corrected. An on-board sensor device that can be detachably attached to the cargo handling equipment of the cargo handling vehicle, independently of the power system of the cargo handling vehicle.

Images