Cargo Handling System

Abstract

To provide a cargo handling system capable of accurately estimating a position and a posture of a cargo-carrying platform.SOLUTION: A cargo handling system (100) comprises an image acquiring device (110) for acquiring an image of a cargo-carrying platform part (51) of a conveying vehicle (50), and a control device (120) for estimating a position and a posture of a cargo-carrying platform (51n). The control device (120) extracts an outer shape straight line (L) of the cargo-carrying platform part (51) from the image acquired by the image acquiring device (110), and estimates the position and the posture of the cargo-carrying platform (51n) based on the outer shape straight line (L).SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a cargo handling system. [Background technology]

[0002] A cargo handling machine such as a forklift may handle cargo (such as loading or unloading) onto the loading platform of a transport vehicle. Patent Document 1 discloses a system for creating a highly efficient cargo handling plan. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-015574 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing a cargo handling process, it is preferable to be able to accurately grasp the position and orientation of the loading platform of a transport vehicle. An object of the present invention is to provide a cargo handling system that can accurately estimate the position and orientation of the loading platform. [Means for solving the problem]

[0005] The cargo handling system according to the present invention comprises: an image capturing device for capturing an image of the loading platform of the transport vehicle; a control device that estimates the position and orientation of the loading platform; Equipped with The control device From the image acquired by the image acquisition device , a visible edge of the outer or inner wall surrounding the loading platform portion that appears in the image Extract the outline straight line, The invisible outline lines of the loading platform portion that do not appear in the image or the intersections of the invisible outline lines are determined based on the visible outline lines, and the visible outline lines and the intersections of the invisible outline lines are determined. Outline straight line or the intersection The position and orientation of the platform are estimated based on the above. [Effects of the Invention]

[0006] According to the present invention, it is possible to obtain an effect that the position and orientation of the loading platform can be accurately estimated. [Brief explanation of the drawings]

[0007] [Figure 1] 1A is a schematic diagram showing a cargo handling system according to an embodiment of the present invention, and FIG. 1B is a block diagram showing the cargo handling system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a perspective view showing an example of a transport vehicle. [Figure 3] FIG. 2 is a diagram showing an example of a cargo handling plan. [Figure 4] 10 is a flowchart showing a cargo handling control process. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0009] FIG. 1 is a schematic diagram (A) and a block diagram (B) showing a cargo handling system according to an embodiment of the present invention.

[0010] The cargo handling system 100 of this embodiment is a system that performs cargo handling-related processes at a cargo handling site G. The cargo handling site G refers to a location where a cargo handling machine 30 (FIG. 3) handles cargo onto a transport vehicle 50. The concept of cargo handling includes transporting cargo, loading, and unloading. The cargo handling machine 30 is, for example, a forklift, but may have any configuration as long as it is capable of handling cargo. The cargo handling machine 30 may perform cargo handling by automatic driving, or may be driven by a driver. The transport vehicle 50 is a vehicle having a loading platform 51n, such as a truck. The transport vehicle 50 that is the target of processing by the cargo handling system 100 may be one type of vehicle defined by a specific format, multiple known types of vehicles, or any multiple types of vehicles.

[0011] In this embodiment, the cargo handling-related processing performed by the cargo handling system 100 is mainly the determination of a cargo handling plan to be executed by the cargo handling machine 30. A cargo handling plan refers to a cargo handling plan that defines the travel route, loading location and order, or unloading location and order for the cargo handling by the cargo handling machine 30 (transporting cargo, loading onto the loading platform 51n, or unloading from the loading platform 51n). Note that the cargo handling-related processing performed by the cargo handling system 100 may be any processing related to cargo handling, such as processing to support the cargo handling of the cargo handling machine 30 and / or the transport vehicle 50. The cargo handling support processing may include, for example, issuing a warning alert or notifying information useful for cargo handling (such as the height and inclination of the loading platform 51n).

[0012] The cargo handling system 100 includes an image capturing device 110 that captures an image of the cargo bed 51 of the transport vehicle 50, and a control device 120 that estimates the position and attitude of the cargo bed 51.

[0013] The image acquisition device 110 may be a digital camera having a lens and an image sensor. In this case, the image is a two-dimensional image, such as an RGB image or a black-and-white image. The image acquisition device 110 may be a scanning device that acquires a scanned image by scanning an object with light irradiated thereon and measuring the reflected wave. In this case, the image is a two-dimensional scanned image or a three-dimensional scanned image.

[0014] The image acquisition device 110 may include a distance sensor 111, and the acquired image may include distance information. The distance sensor 111 may be any device capable of measuring the distances to multiple points in an image, such as a LiDAR (Light Detection and Ranging) or a compound eye camera. The image acquisition device 110 may also function as the distance sensor 111.

[0015] The image acquisition device 110 may output the acquired image to a monitor. Then, a manager or a worker may monitor the loading dock G by checking the monitor.

[0016] The control device 120 is a computer equipped with a CPU (Central Processing Unit) that executes a program. The control device 120 has a learning model 121 that has been machine-learned, a storage unit that stores a program 122 for cargo handling control processing and control data 123, and a cargo handling plan database 124 that stores multiple cargo handling plans. The learning model 121 may be an AI (Artificial Intelligence) that has a deep-learned neural network.

[0017] The control device 120 extracts the outline straight line L of the loading platform 51 from the image acquired by the image acquisition device 110, and estimates the position and orientation of the loading platform 51n based on the extracted outline straight line L. If the image includes distance information, the control device 120 may estimate the position and orientation of the loading platform 51n in three-dimensional coordinates based on the extracted outline straight line and the distance information.

[0018] The learning model 121 is provided in advance with a plurality of teacher data indicating images of a plurality of transport vehicles and the outline straight lines of the loading platform 51 of the transport vehicles, and the learning model 121 is machine-learned using the plurality of teacher data. The control device 120 can extract the outline straight line L of the loading platform 51 from the image of the transport vehicle 50 using the learning model 121. Specifically, when the control device 120 inputs image data of the transport vehicle 50 acquired by the image acquisition device 110 to the learning model 121, it can receive data indicating the outline straight line of the loading platform 51 in the image data from the learning model 121.

[0019] The control device 120 may perform image analysis on the image acquired by the image acquisition device 110 to extract the outline straight line L of the loading platform portion 51.

[0020] FIG. 2 is a perspective view showing an example of a transport vehicle.

[0021] The extracted outline straight line L of the loading platform 51 includes the outer or inner edges E1 to E5 of the walls 52a to 52c surrounding the loading platform 51n. Furthermore, the outline straight line L may include portions corresponding to the edge E6 of the wing 53 of the transport vehicle 50 and the edges E7 to E10 of the tailgate 54 of the transport vehicle 50.

[0022] As shown in Figures 1(A) and 2, the control device 120 can calculate the position of the outline line L in three-dimensional coordinates from the outline line L extracted from the image and distance information in the image. Furthermore, it can calculate the position of the intersection P of two intersecting outline lines L in three-dimensional coordinates. For example, the position of the intersection P in Figure 1(A) in three-dimensional coordinates can be calculated by adding the vector A3 to the calculated intersection point P in the image (vector A3 from the image acquisition device 110) to the vector A1+A2 from the reference point O1 of the loading dock G to the image acquisition device 110.

[0023] When calculating the position of the intersection point P, the intersection point P does not have to appear in the image. That is, as long as two intersecting outline lines L appear in the image, the control device 120 can calculate the position of the intersection point not appearing in the image as the point where the extensions of the two outline lines L intersect. Furthermore, as shown in FIG. 2, the control device 120 can also calculate the position of the outline line Lv not appearing in the image and its intersection point Pv from the outline line L that appears in the image. That is, the geometric relationship between the various parts of the loading platform 51 may be known. For example, the surfaces of the opposing walls of the loading platform 51 may be parallel, or the interior space of the loading platform 51 may have a specific shape (e.g., a rectangular parallelepiped). In such a case, the control device 120 can use the known geometric relationship to calculate the positions of the outline line Lv not appearing in the image and its intersection point Pv (e.g., its intersection with the right, upper, and lower inner edges of the rear wall of the loading platform 51) from the outline line L that appears in the image.

[0024] The control device 120 then estimates the position and orientation of the platform 51n based on the outline lines L, Lv and the positions of the intersections P, Pv calculated as described above. The control device 120 has, as control data 123, sample data indicating the outline lines L and intersections P of the platform unit 51 for multiple types of transport vehicles 50, and which of these represent the periphery of the platform 51n. By comparing the sample data with the calculated outline lines L, Lv and intersections P, Pv, the control device 120 can identify the outline lines L, Lv and intersections P, Pv that represent the periphery of the platform 51n and accurately estimate the position and orientation of the platform 51n.

[0025] The control device 120 may further calculate the presence of the wing 53 and the gate 54, the opening / closing ratio of the wing 53, and the angle and protrusion amount of the gate 54 based on the outline straight line L of the wing 53 and the outline straight line L of the gate 54. By such calculations, for example, when the opening or closing of the wing 53 stops halfway, the control device 120 can issue a warning (e.g., a stop warning) to the cargo handling machine 30 and / or the transport vehicle 50. Furthermore, when the gate 54 is stopped at an angle that protrudes forward, the control device 120 can issue a warning (proximity warning) to the cargo handling machine 30 and / or the transport vehicle 50. The cargo handling system 100 may include an alarm device 130 that outputs an indicator light or an alarm sound, and the control device 120 may issue the above-mentioned warning via these. Furthermore, the cargo handling system 100 may include a control mechanism that acts on the operating mechanism of the cargo handling machine 30, and may perform operating operations such as stopping the cargo handling machine 30 or preventing the cargo handling machine 30 from approaching in the above-mentioned cases. The driving operation command may be transmitted from the control device 120 to the control mechanism of the cargo handling machine 30 by wireless communication or wired communication.

[0026] FIG. 3 is a diagram showing an example of a cargo handling plan.

[0027] The control device 120 determines a cargo handling plan to be executed by the cargo handling machine 30 based on the estimated position and attitude of the loading platform 51n. As shown in Fig. 3, the cargo handling plan shows a route c11 along which the cargo handling machine 30 moves from the waiting area G2 for the cargo handling machine 30 to the loading area G, routes c12, c1, and c2 along which the cargo handling machine 30 travels back and forth between the warehouse G1 and the loading platform 51n of the transport vehicle 50 to load or unload multiple times, and the locations and order of loading or unloading on the loading platform 51n. The cargo handling plan may also include the height of the load f (the height at which the forks are raised or lowered) and the inclination angle of the load f (the tilt angle of the forks) when the cargo handling machine 30 loads or unloads the load. In warehouse G1, the cargo handling machine 30 may transport cargo f from shelf H or transport cargo f to shelf H, or the cargo handling machine 30 may deliver cargo f at a predetermined location in warehouse G1, and in warehouse G1, cargo f may be transported between the predetermined location and shelf H by a belt conveyor or the like.

[0028] The cargo handling plan data determined by the control device 120 may be sent to the cargo handling machine 30, and the cargo handling machine 30 may perform cargo handling by automatic operation in accordance with the cargo handling plan. Alternatively, the cargo handling plan data may be presented to the operator of the cargo handling machine 30, and the operator may operate the cargo handling machine 30 in accordance with the cargo handling plan data, thereby performing cargo handling in accordance with the cargo handling plan.

[0029] During the loading and unloading process by the loading and unloading machine 30, the weight of the load f on the loading platform 51n may change, causing the posture of the loading platform 51n to change. For this reason, during the loading and unloading process, the image acquisition device 110 continuously acquires images of the transport vehicle 50, and the control device 120 continuously estimates the position and posture of the loading platform 51n. If the estimation result differs from the previous estimation result, the control device 120 updates the estimation result of the position and posture of the loading platform 51n and may also correct the loading and unloading plan in response to the update. The correction of the loading and unloading plan may include correction of the height of the load f (the height at which the forks are raised and lowered) when loading or unloading onto or from the loading platform 51n, as well as correction of the inclination angle of the load (the tilt angle of the forks).

[0030] <Cargo handling control processing> FIG. 4 is a flowchart showing the cargo handling control process.

[0031] Next, the cargo handling control process executed by the cargo handling system 100 will be described. When the transport vehicle 50 stops at the loading dock G and opens the loading platform 51n, the image acquisition device 110 acquires an image of the transport vehicle 50 (step S1). The image data is then sent to the control device 120, which extracts the outline line L of the loading platform 51 from the image data using the learning model 121 (step S2). Furthermore, based on the distance information in the image, the control device 120 calculates the position of the outline line L, the position of the intersection point P, the position of the outline line Lv that does not appear in the image, and the position of the intersection point Pv on the three-dimensional coordinate system. Then, the control device 120 compares the calculation result of step S3 with the sample data, and identifies the outline line and intersection points that are the periphery of the loading platform 51n from the calculation result of step S3, thereby estimating the position and posture of the loading platform 51n (step S4).

[0032] Next, the control device 120 creates a cargo handling plan corresponding to the position and posture of the loading platform 51n based on the estimation result of step S4 (step S5). The control device 120 previously stores multiple base data for cargo handling plans corresponding to multiple sizes of loading platforms in the loading plan database 124. The control device 120 may create a cargo handling plan corresponding to the loading platform 51n by extracting base data corresponding to the estimated size of the loading platform 51n from the multiple base data and modifying the route c1 (FIG. 3) that moves around the transport vehicle 50 using the extracted base data. Here, the control device 120 performs a modification by translating and rotating the route c1 to match the position of the loading platform 51n, and then connects the modified route c1 with the route c2 that continues thereto, thereby creating a route for the loading plan. The modification by translating and rotating the route c1 may be achieved by a worker manually operating the screen while viewing the image. In this case, the image may display the route c1 of the loading plan before modification and the loading platform 51n as the estimated result.

[0033] Furthermore, the control device 120 determines the loading and unloading plan by setting the position of the load f, the height of the load f (the height to which the forks are raised and lowered), and the inclination angle of the load f (the tilt angle of the forks) during loading or unloading in the loading and unloading plan to values ​​corresponding to the estimated position and posture of the loading platform 51n.

[0034] Once the cargo handling plan is determined, the control device 120 transmits the cargo handling plan data to the cargo handling machine 30 (step S6). As a result, the cargo handling machine 30 carries out cargo handling processing in accordance with the cargo handling plan.

[0035] The cargo handling system 100 repeatedly executes a loop process (steps S7 to S13) while cargo handling processing is in progress. In the loop process, first, the image acquisition device 110 acquires an image of the transport vehicle 50 (step S7). Next, the control device 120 extracts the outline straight lines of the loading platform 51 and estimates the position and posture of the loading platform 51n (step S8). The control device 120 then compares the estimated values ​​used when determining the cargo handling plan with the estimation results of step S8 to determine whether there is a difference greater than a predetermined value (step S9). If the result of the determination is YES, the control device 120 updates the estimation results of the position and posture of the loading platform 51n (step S10) and corrects the cargo handling plan based on the updated estimation results (step S11). That is, in step S11, the control device 120 corrects the path c1 of movement of the loading / unloading machine 30 around the transport vehicle 50 and the height and inclination angle of the loaded or unloaded load f to match the height and inclination angle of the updated loading platform 51n.

[0036] Then, the control device 120 transmits the corrected cargo handling plan data to the cargo handling machine 30 (step S12). By transmitting the data, the cargo handling machine 30 executes the remaining cargo handling process using the corrected cargo handling plan data.

[0037] Also, in the loop processing, the control device 120 determines whether or not there is a notification of completion of loading and unloading in the loading and unloading plan sent from the loading and unloading machine 30 (step S13), and if there is a notification of completion of loading and unloading, it exits the loop processing and ends the loading and unloading processing.

[0038] As described above, according to the cargo handling system 100 of this embodiment, the control device 120 extracts the outline straight line L of the loading platform 51 from an image of the transport vehicle 50, and estimates the position and orientation of the loading platform 51n based on the outline straight line L. The outline straight line L of the loading platform 51 is easy to extract accurately. Therefore, the control device 120 can accurately estimate the position and orientation of the loading platform 51n. Furthermore, extracting the outline straight line L of the loading platform 51 does not require complicated work such as attaching a marker for position detection. Therefore, the control device 120 can accurately estimate the position and orientation of the loading platform 51n without requiring the operator's effort.

[0039] Furthermore, according to the cargo handling system 100 of this embodiment, the outer straight line L of the loading platform 51 is extracted using a machine-learned learning model 121. Therefore, the control device 120 can reduce erroneous recognition of the outer straight line L due to the background of the transport vehicle 50, etc. Furthermore, even if an inexpensive photographing device is used as the image acquisition device 110, the control device 120 can accurately extract the outer straight line L.

[0040] Furthermore, according to the cargo handling system 100 of this embodiment, the image acquisition device 110 includes a distance sensor 111, and the control device 120 calculates the positions of the extracted outline line L and its intersection point P in three-dimensional coordinates. The control device 120 then estimates the position and orientation of the cargo platform 51n in three-dimensional coordinates. Therefore, it is possible to obtain an estimated result of the cargo platform 51n that is useful even in a cargo handling site G where horizontality is not guaranteed. It is also possible to obtain an estimated result of the cargo platform 51n that is useful when handling a heavy load f in which the height or inclination of the cargo platform 51n changes due to loading and unloading.

[0041] In the above embodiment, a configuration for estimating the position and orientation of the loading platform 51n in three-dimensional coordinates has been described, but for example, when the loading area G is horizontal and the height of the loading platform 51n is constant, the control device 120 may estimate the position and orientation of the loading platform 51 in two-dimensional coordinates. For example, the image acquisition device 110 may acquire an image of the transport vehicle 50 from above, and the control device 120 may acquire the outline of the loading platform 51 when viewed from above, thereby estimating the position and orientation of the loading platform 51n in two-dimensional coordinates (planar coordinates of the loading area G). Based on the results of this estimation, the control device 120 can create a loading plan that matches the position and orientation of the loading platform 51n.

[0042] Furthermore, in the cargo handling system 100 according to this embodiment, the control device 120 extracts contour lines L representing the outer or inner edges E1 to E5 of the walls 52a to 52c surrounding the loading platform 51n of the transport vehicle 50. Therefore, from these contour lines L, it is possible to determine the contour lines Lv and intersection points Pv of the loading platform portion 51 that do not appear in the image. Furthermore, the control device 120 extracts contour lines L representing the edges E6 of the wing 53 of the transport vehicle 50 or the edges E7 to E10 of the gate 54. Therefore, from these extraction results, it is possible to determine the opening / closing degree of the wing 53 or the protrusion amount of the gate 54, and based on this information, it is possible to provide highly convenient cargo handling assistance.

[0043] Furthermore, according to the cargo handling system 100 of this embodiment, a cargo handling plan to be executed by the cargo handling machine 30 is created based on the estimation results of the position and attitude of the loading platform 51n obtained as described above. Therefore, the cargo handling plan based on accurate estimation results allows the cargo handling machine 30 to efficiently carry out cargo handling processing.

[0044] Furthermore, according to the cargo handling system 100 of this embodiment, the control device 120 continues the estimation process for the loading platform 51n of the transport vehicle 50 during loading and updates the estimation result. Therefore, when the position or posture of the loading platform 51n changes due to loading and unloading, an updated estimation result corresponding to the change can be obtained.

[0045] Furthermore, according to the cargo handling system 100 of this embodiment, the control device 120 corrects the cargo handling plan based on the updated estimated position and attitude of the loading platform 51n. Therefore, even if the position or attitude of the loading platform 51n changes due to cargo handling, the corrected cargo handling plan allows the loading machine 30 to efficiently carry out cargo handling processing.

[0046] The above describes an embodiment of the present invention. However, the present invention is not limited to the above embodiment. For example, in the above embodiment, a forklift is used as an example of a loading / unloading machine. However, the loading / unloading machine may be any of various heavy machinery that transports, loads, and unloads loads. Furthermore, in the above embodiment, an example is shown in which the image acquisition device is installed in a loading yard. However, the image acquisition device may be mounted on the loading / unloading machine. Even in this case, the position and orientation of the loading platform in coordinates based on the loading / unloading machine can be accurately estimated. Alternatively, by sending information about the position and orientation of the loading / unloading machine in coordinates fixed to the loading yard to a control device, the position and orientation of the loading platform in coordinates fixed to the loading yard can be accurately estimated based on images acquired from the loading / unloading machine.

[0047] Furthermore, in the above embodiment, an example was shown in which the control device is installed in a loading dock, but the control device may be mounted on the loading machine or installed in a remote data center such as the cloud. If the control device is remote, image and distance data from the image acquisition device may be transmitted to the control device via communication. Furthermore, loading support information (loading plan and its correction data) from the control device may be transmitted to the loading machine via communication. Furthermore, the control device may be composed of multiple computers (including mobile terminals) that operate in cooperation with each other via communication. In this case, the multiple computers that make up the control device may be distributed to multiple locations, such as the loading dock, a control room, the cloud, and the loading machine. In addition, the details shown in the embodiment may be modified as appropriate without departing from the spirit of the invention. [Explanation of symbols]

[0048] 30 Cargo handling machine 50 Transport Vehicles 51 Cargo area 51n cargo bed 52a~52c Wall 53 Wing 54 Aori E1~E10 Edges 100 Cargo Handling System 110 Image acquisition device 111 Distance Sensor 120 Control device 121 Learning Model 122 Cargo handling control processing program 130 Alarm device L, Lv Outer straight line P, Pv intersection G Loading area c1, c2, c11, c12 Cargo handling plan route

Claims

1. an image capturing device for capturing an image of the loading platform of the transport vehicle; a control device that estimates the position and orientation of the loading platform; Equipped with The control device From the image acquired by the image acquisition device, visible outline lines indicating the outer or inner edges of the walls surrounding the loading platform portion that appear in the image are extracted, and invisible outline lines of the loading platform portion that do not appear in the image or intersections of the invisible outline lines are determined based on the visible outline lines, and the position and orientation of the loading platform are estimated based on the visible outline lines and the invisible outline lines or intersections. Load handling system.

2. the control device extracts the visible outline straight line using a learning model that has been machine-learned using an image of the transport vehicle and an outline straight line of the loading platform of the transport vehicle as training data; The cargo handling system according to claim 1.

3. the image capture device includes a distance sensor for measuring distance; the control device estimates the position and orientation of the loading platform in three-dimensional coordinates based on the extracted visible outline lines, the invisible outline lines or the intersection points, and the distances measured by the distance sensor; The cargo handling system according to claim 1 or 2.

4. The visible outline straight lines extracted by the control device include at least one of an outer or inner edge of a wall surrounding a loading platform of the transport vehicle, an edge of a wing of the transport vehicle, and an edge of a tailgate of the transport vehicle. The cargo handling system according to any one of claims 1 to 3.

5. the control device determines a cargo handling plan to be executed by the cargo handling machine based on the estimated position and attitude of the loading platform. A cargo handling system according to any one of claims 1 to 4.

6. the image acquisition device continues acquiring images of the loading platform portion while the loading machine is performing loading and unloading; The control device continues to extract the visible outline straight line while the cargo handling machine is performing cargo handling, the control device updates the estimated results of the position and attitude of the loading platform unit while the loading machine is performing loading and unloading, based on the continuously extracted visible outline straight lines. A cargo handling system according to any one of claims 1 to 5.

7. the image acquisition device continues acquiring images of the loading platform portion while the loading machine is performing loading and unloading; The control device continues to extract the visible outline straight line while the cargo handling machine is performing cargo handling, the control device updates the estimation results of the position and attitude of the loading platform while the loading machine is performing loading and unloading based on the continuously extracted visible outline straight lines, and corrects the loading and unloading plan based on the updated estimation results. The cargo handling system according to claim 5.

Images