Cargo handling support device, cargo handling support system, and cargo handling equipment

Abstract

To provide a cargo handling support device, cargo handling support system and cargo handling device which suitably support cargo handling.SOLUTION: In a cargo handling support system 200, a cargo handling support device 100 comprises: an image acquisition unit 110 which acquires an image including a pallet P and a cargo H on the pallet P; a storage unit 140 which previously stores 3D-CAD data 144 of the pallet P and the cargo H; and a control unit 120. The control unit 120 collates the image acquired by the image acquisition unit 110 with the 3D-CAD data 144 stored in the storage unit 140 and obtains the gravity center position of the cargo H to the pallet P.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] ,

[0006]

[0001] The present invention relates to a cargo handling support device, a cargo handling support system, and a cargo handling device.

Background Art

[0002] Patent Document 1 discloses a system that acquires an image of a load carried by a forklift and determines that the load has collapsed when the differential amount of the image over time exceeds a threshold value.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above conventional load collapse determination system, even if the load is being transferred stably, there is a risk that the load will be determined to have collapsed just because the position of the load has shifted, and there is also a risk that no load collapse determination will be obtained when the load stability is low but there is no load position shift. In addition, there is a problem that it is difficult to accurately detect the differential amount of the load image over time when a change occurs in the background of the load.

[0005] The present invention has been made in view of such circumstances, and an object thereof is to suitably support cargo handling.

Means for Solving the Problems

[0006] The cargo handling support device according to the present invention includes an image acquisition unit that acquires an image including a pallet and a load on the pallet, a storage unit that stores in advance the shape data of the pallet and the load, a control unit, and is provided with <000003​​Compare the image acquired by the image acquisition unit with the shape data stored in the storage unit, and determine the center of gravity position of the load with respect to the pallet 、 Using a machine learning model trained with training data in which the outline of the load is assigned to an image of the load, the outlines of the pallet and the load are extracted from the image acquired by the image acquisition unit. The outline is compared with the shape data. Furthermore, the cargo handling support device according to the present invention is An image acquisition unit that acquires an image including the pallet and the load on the pallet, A storage unit that pre-stores the shape data of the pallet and the load, Control unit and Equipped with, The control unit, The image acquired by the image acquisition unit is compared with the shape data stored in the storage unit to determine the center of gravity of the load relative to the pallet. Based on the aforementioned center of gravity position, the quality of the loading condition is determined.

Advantages of the Invention

[0007] According to the present invention, it is possible to suitably assist the cargo handling

Brief Description of the Drawings

[0008] [Figure 1] It is a block diagram showing a cargo handling support device and a cargo handling support system according to the present invention [Figure 2] It is a flowchart showing the procedure of the determination process [Figure 3] It is a diagram for explaining the determination process [Figure 4] It is a diagram for explaining the determination process [Figure 5] It is a diagram for explaining the determination process [Figure 6] It is a flowchart showing the procedure of the cargo handling support process [Figure 7] It is a diagram showing an example of an image display on the display unit in the cargo handling support process

Modes for Carrying Out the Invention

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings

[0010] <Cargo handling support device> FIG. 1 is a block diagram showing a cargo handling support device and a cargo handling support system according to the present invention.

[0011] The cargo handling support device 100 of the present embodiment is a device that determines whether the loading state of the load H on the pallet P held and transported by the cargo handling vehicle 30 as a cargo handling device is good or bad. The cargo handling vehicle 30 is a vehicle such as a forklift that performs cargo handling (cargo transportation, loading, unloading, etc.) via the pallet P. The pallet P is a platform on which the load H is stacked and has a structure e (for example, a horizontal hole into which the fork f of a forklift can be inserted) that can be held by the cargo handling vehicle 30. A plurality of loads H can be stacked on the pallet P.

[0012] Whether the loading state is good or bad refers to whether the load H is stably stacked on the pallet P, that is, an index indicating whether there is little or much risk of load collapse or the like when the cargo handling vehicle 30 transports the load H via the pallet P.

[0013] The load H that is the object of the determination of whether the loading state is good or bad is a load H whose center of gravity position M can be obtained from its shape. The shape of the load H may not be one type but a plurality of types. The load H that is the object may be such that an object is placed in a box representing the outer shape of the load H under predetermined conditions, and the shape of the load H and the center of gravity position M are associated with each other. Alternatively, data (for example, the 3D-CAD data 145 described later) indicating the correspondence relationship between the shape of the load H and the center of gravity position M may be given in advance, and the shape of the load H and the center of gravity position M may be associated with each other based on the data.

[0014] Specifically, the cargo handling support device 100 includes an image acquisition unit 110, a control unit 120, and a storage unit 140.

[0015] The image acquisition unit 110 acquires an image including the pallet P and the load H on the pallet P, and transmits the acquired image data to the control unit 120 via wired or wireless connection. The image acquisition unit 110 may be a digital camera having a lens and an image sensor, in which case the image will be an RGB image. Alternatively, the image acquisition unit 110 may be a scanner device that acquires a two-dimensional or three-dimensional shape by scanning light or sound waves and detecting reflected waves. In that case, the image will be a scanner image. The image acquisition unit 110 may include a distance sensor such as an RGB-D sensor, and the acquired image may include distance information. The distance sensor may be any device that can measure the distance between multiple points in the image, such as a LiDAR (Light Detection And Ranging) or a compound eye camera.

[0016] The image acquisition unit 110 may be a first image acquisition unit 110a mounted on the cargo handling vehicle 30, a second image acquisition unit 110b installed at a location separate from the cargo handling vehicle 30, or a combination of these. The image acquisition unit 110 may be configured to acquire images of the load on the pallet P from one direction, or it may be configured to acquire images of the load on the pallet P from multiple directions.

[0017] The control unit 120 includes a CPU (Central Processing Unit) for executing programs and, along with a storage unit 140, constitutes a computer. The control unit 120 calculates the center of gravity position M (see Figure 4) of the load H based on the image acquired by the image acquisition unit 110, and determines the quality of the loading condition based on the calculated center of gravity position M of the load H.

[0018] The memory unit 140 is a memory configured with, for example, RAM (Random Access Memory) and ROM (Read Only Memory), and stores various programs and data, as well as functioning as a work area for the control unit 120. The memory unit 140 pre-stores the program 142 for the determination process and control data 143. The control data 143 includes data indicating the determination area R, which will be described later.

[0019] Furthermore, the memory unit 140 has a learning model 144 and multiple three-dimensional CAD (3D-CAD) data sets 145 pre-stored in it. The learning model 144 extracts the regions (outlines) of palette P and load H from an image. The learning model 144 is pre-created by machine learning using multiple training data sets, each containing multiple sample images with the outlines of palette P and individual loads H within those sample images. The learning model 144 may also be an AI (Artificial Intelligence) with a deep-learned neural network.

[0020] The 3D-CAD data 145 is three-dimensional shape information (design information) of the pallet P and the load H. In this embodiment, multiple 3D-CAD data 145 for all types of pallets P and all types of load H that may be usable at the work site are stored in the storage unit 140. However, each 3D-CAD data 145 only needs to contain three-dimensional shape data for at least one pallet P and load H, and there is no particular formal limit on the number of such data for handling purposes (a single data may contain shape data for all loads H). In addition, the 3D-CAD data 145 includes information such as density necessary for calculating the center of gravity. Alternatively, instead of 3D-CAD data 145, data relating the shape of the load H to its center of gravity M (data that matches the image of the load H to determine its center of gravity M) may be stored. Such data could be, for example, information relating a two-dimensional image of the load H to its center of gravity M, or three-dimensional shape (point cloud) data of the load H measured in advance by a distance sensor.

[0021] <Decision Process> Figure 2 is a flowchart showing the determination process executed by the control unit 120, and Figures 3 to 5 are diagrams illustrating the determination process.

[0022] The control unit 120 performs a determination process based on predetermined start conditions. The start conditions can be set appropriately to indicate when the loading status should be checked, such as just before the loading vehicle 30 starts loading, or during loading (for example, when it passes in front of the second image acquisition unit 110b installed in the loading area).

[0023] As shown in Figure 2, when the determination process is started, the control unit 120 receives the image acquired by the image acquisition unit 110 (step S1). The image acquisition unit 110 photographs the space where the palette P is (likely) located and transmits the acquired image to the control unit 120. The timing of the photography may be based on user operation or on a periodic basis, and may or may not be controlled by the control unit 120. In this embodiment, for example, as shown in Figure 3(a), an image is obtained showing two types of loads H (H1, H2) with different shapes placed on the pallet P.

[0024] Next, the control unit 120 extracts the regions (outlines) of the pallet P and each load H from the image acquired in step S1 (step S2). Specifically, the control unit 120 inputs the image acquired in step S1 into the learning model 144, and extracts the outlines L(L0, L1, L2) of the pallet P and individual loads H from the image, as shown in Figure 3(b). Note that the outline L may be interpreted as the circumtangency or intangency of the load H. The circumtangency refers to the smallest rectangle that circumsects the load H in the image (the rectangle enclosed by a vertical line and a horizontal line in the image), and the intangency refers to the largest rectangle that circumsects the load H in the image (the rectangle enclosed by a vertical line and a horizontal line in the image). When the load H is photographed from the front or directly from the side, the outline of the load H and the circumtangency or intangency of the load H will approximately coincide.

[0025] Next, the control unit 120 takes the outline L extracted in step S2 and the 3D-C of the storage unit 140. Match with AD data 145 (Step S3). Here, the control unit 120 acquires three-dimensional shape data of the pallet P and individual loads H by comparing the outline L of the pallet P and individual loads H with 3D-CAD data 145. If the image acquired in step S1 does not have depth information (when the image acquisition unit 110 is an RGB camera, etc.), two-dimensional contour shape matching (silhouette matching) is performed between the outline L and the 3D-CAD data 145. If the image acquisition unit 110 is an RGB-D sensor, etc. and the image has depth information, the point cloud data contained in the image is matched three-dimensionally with the 3D-CAD data 145.

[0026] Next, as shown in Figure 4(a), the control unit 120 calculates the center of gravity position M of each load H based on the 3D-CAD data 145 of each load H (step S4). At this time, the control unit 120 also calculates (estimates) the posture information of the pallet P and each load H. Then, the control unit 120 determines the center of gravity Mc of the entire load H on the pallet P from the center of gravity M of each load H (step S5). The center of gravity Mc of the entire load H can be calculated from the arrangement of the multiple loads H and the weight ratio of each load H. Alternatively, if the multiple loads H are made of the same material (same density), the overall center of gravity Mc can be calculated from the center of gravity M of each load H. The weight ratio of each load H, or the fact that each load H is made of the same material, only needs to be provided to the control unit 120 in advance.

[0027] Next, the control unit 120 determines whether the loading condition is good or bad based on the calculated center of gravity position Mc of the entire load and the determination area R (step S6). Specifically, as shown in Figure 4(b), the control unit 120 determines whether the center of gravity Mc of the entire load is within the determination area R. If it determines that the center of gravity Mc of the entire load is within the determination area R, the control unit 120 determines that the load H on the pallet P is stable and that the loading condition is good, and transmits this determination result to the support device 210 described later (step S7). On the other hand, if it determines that the center of gravity Mc of the entire load is not within the determination area R, the control unit 120 determines that the load H on the pallet P is unstable and that the loading condition is poor, and transmits this determination result to the support device 210 described later (step S8).

[0028] The determination area R is set in the space above the pallet P so as to be the boundary of the area used to determine whether the load H is stable during handling via the pallet P, based on whether the center of gravity Mc is located inside the area. The shape and size of the determination area R are not particularly limited, but are set based on the orientation estimation result of the pallet P. For example, the determination area R is set in the shape of a pyramid with a rectangular base corresponding to the shape of the top surface of the pallet P and a predetermined height. For example, the base of the determination area R is set so that each side is located inward by a predetermined margin from each side of the top surface of the pallet P. The height of the determination area R is set lower than, for example, the maximum height of the load H. Furthermore, the position of the judgment area R relative to the pallet P is set, for example, with the center of the top surface of the pallet P as the reference position. Then, as shown in Figure 5, if the actual insertion position Pf of the fork f into the fork pocket (structure e) deviates from its reference position (for example, the center in the width direction of the fork pocket), the position of the judgment area R may be adjusted to correspond to this deviation. In other words, the judgment area R may be moved from the reference position in the same way, corresponding to the direction and amount of deviation of the insertion position Pf of the fork f.

[0029] <Setting the area for judgment> The cargo handling support device 100 may further include a setting unit 130 capable of setting a determination area R for the center of gravity position M, as shown in Figure 1. The setting unit 130 includes a display device 131 that outputs a setting display and an input device 132 for specifying the determination area R, allowing an administrator to specify the determination area R via the input device 132 while viewing the display on the display device 131. The display device 131 and input device 132 are not necessarily provided in the cargo handling support device 100, but may be a display device and input device provided in another computer or another mobile terminal. In this case, the setting unit 130 of the cargo handling support device 100 may be software that communicates with the other computer or another mobile terminal to output a setting display and performs input processing to specify the determination area R. The administrator can, for example, change the setting to narrow or widen the determination area R. Alternatively, the administrator can change the setting to raise or lower the determination area R, or change the shape of the determination area R. This configuration allows administrators to change the judgment area R according to various situations, such as situations requiring very high cargo handling stability and situations requiring relatively low cargo handling stability.

[0030] <Cargo handling support system> As shown in Figure 1, the cargo handling support system 200 of this embodiment includes a cargo handling support device 100 and a support device 210 that provides cargo handling support based on the judgment result of the cargo handling support device 100. The support device 210 may include a notification unit 211 that notifies the result of the judgment of whether the cargo handling is good or bad, and a driving control mechanism 212 that controls the operation of the cargo handling vehicle 30 (such as temporary stop control). The notification unit 211 may be mounted on the cargo handling vehicle 30, or it may be located both on the cargo handling vehicle 30 and in the cargo handling area. The notification unit 211 provides notification corresponding to the judgment result of the cargo handling support device 100 by audio output, lamp display, image or text display. The notification unit 211 may include a display unit 211a that displays video. The driving control mechanism 212 is, for example, a mechanism that can temporarily stop or reduce the speed of raising and lowering the forks, or a mechanism that can temporarily stop or reduce the speed of the cargo handling vehicle 30. More specifically, the driving control mechanism 212 is a mechanism capable of preventing the forward movement of the cargo handling vehicle 30 and the insertion of the forks into the pallet P. The driving control mechanism 212 may be mounted on the cargo handling vehicle 30, or, if the cargo handling vehicle 30 is in automatic operation mode, it may be mounted on a device that outputs instructions for automatic operation.

[0031] Figure 6 is a flowchart showing the cargo handling support process performed by the support device. As shown in the judgment process described above (Figure 2), the control unit 120 of the cargo handling support device 100 transmits the judgment result of whether the loading condition is good or bad to the support device 210. The support device 210 receives the judgment result (step S11) and determines the result (step S12). In step S12, if the support device 210 determines that the loading condition is good, it performs an action to determine good (step S13). Specifically, as an action to determine good, the notification unit 211 outputs a lamp of a color indicating good judgment, or outputs characters, images, or sounds indicating good judgment. Furthermore, as an action to determine good, the support device 210 may output an image to the display unit 211a in which an image of the load H on the pallet P, a point image indicating the calculated center of gravity position M, and a pattern representing the judgment area R are superimposed.

[0032] On the other hand, if the support device 210 determines in step S12 that the loading condition is poor, it performs an operation for determining the poor condition (step S14). Specifically, as an operation for determining the poor condition, the notification unit 211 outputs a lamp with a color and flashing action indicating the poor condition, or outputs characters, images, or sounds indicating the poor condition. Furthermore, as an operation for determining the poor condition, the support device 210 may output an image to the display unit 211a in which an image of the load H on the pallet P, a point image indicating the calculated center of gravity position M (including Mc), and a pattern representing the determination area R are superimposed. The image output to the display unit 211a is not particularly limited, but for example, as shown in Figure 7, the load H on the pallet P, its center of gravity position Mc, and the determination area R may be displayed in a three-view drawing. By looking at this image, the loading operator can recognize which load H is poorly arranged. Furthermore, in step S14, the driving control mechanism 212 may perform driving control to temporarily suspend the operation of the cargo handling vehicle 30 or reduce its speed. This allows for checking for any risk of cargo H falling or tipping over before unloading, thereby reducing accidents during cargo handling operations.

[0033] As described above, according to this embodiment, the images of the pallet P and load H acquired by the image acquisition unit 110 and the 3D-CAD data of the pallet P and load H stored in the storage unit 140 are used. By comparing with 145, the center of gravity position M of the load H relative to pallet P can be determined. This allows for a favorable determination of the quality of the loading conditions. Consequently, cargo handling can be appropriately supported based on the determination results.

[0034] Furthermore, according to this embodiment, by evaluating the center of gravity position M(Mc) of the load H on the pallet P in three dimensions, it is possible to suitably determine the three-dimensional tipping of the cargo handling vehicle 30 and the load H, as well as the possibility of tipping. Furthermore, by simply making software changes to the shape data of the pallet P and load H stored in the memory unit 140 and the learning model 144 to match the materials (load H) handled at that work site, it is possible to easily adapt to the cargo handling operations at that work site.

[0035] Furthermore, according to this embodiment, the outlines L of the pallet P and load H are extracted from the actual image using a machine learning model 144, and these outlines L are matched with 3D-CAD data 145. By using the learning model 144 in this way, misrecognition of the outline L of the load H on the pallet P due to the background, etc., can be suppressed, and more accurate calculations can be achieved with less load. Furthermore, by using the learning model 144, even if a relatively inexpensive imaging device is used as the image acquisition unit 110, more accurate calculations can be achieved based on the detection results. Furthermore, by pre-loading potentially usable pallets P and loads H into the learning model 144, the system can effectively handle cases where there are multiple types of pallets P and loads H. It can also effectively handle cases where there are multiple loads H with different shapes.

[0036] Furthermore, according to this embodiment, the calculated center of gravity position M(Mc) of the load is displayed on the display unit 211a, so that the cargo handling worker (for example, the driver of the cargo handling vehicle 30) can immediately recognize the tipping or possibility of tipping of the cargo handling vehicle 30 or the load H.

[0037] <Other> Embodiments of the present invention have been described above. However, the present invention is not limited to the embodiments described above. For example, in the above embodiment, a forklift was used as an example of a cargo handling device, but the cargo handling device may be any heavy machinery used for transporting, loading, and unloading cargo. Furthermore, the cargo handling device may be operated by a human driver or operated automatically. The pallet on which the cargo is loaded is not limited to a pallet for a forklift, but can be any configuration as long as it is a platform for transporting cargo and is transported by heavy machinery. In addition, in the above embodiment, an example was shown in which an image of the cargo H on the pallet P is acquired from the side. However, an image of the cargo H on the pallet P may be acquired from above, and the quality of the loading condition may be determined based on the center of gravity position (center of gravity position in the plane direction) of the upper cargo H. Also, in Figure 1, an example is shown in which the control unit 120, setting unit 130 and storage unit 140 are installed in a location separate from the cargo handling vehicle 30, and a part of the image acquisition unit 110 (first image acquisition unit 110a) and a part of the support device 210 are mounted on the cargo handling vehicle 30. However, some or all of the control unit 120, storage unit 140, and setting unit 130 may be mounted on the cargo handling vehicle 30. In this case, the portion of the control unit 120, storage unit 140, setting unit 130, and image acquisition unit 110 mounted on the cargo handling vehicle 30 and the portion installed at a location other than the cargo handling vehicle 30 can communicate to perform the same processing. Alternatively, some or all of the control unit 120, storage unit 140, and setting unit 130 may be located in a cloud (server device), and the remaining portion, as well as the image acquisition unit 110 and support device 210, may communicate to perform the same processing. Furthermore, the control unit 120, storage unit 140, and setting unit 130 may be divided and mounted on multiple locations, including a cloud, a mobile terminal, and a computer on the cargo handling vehicle 30, and these may communicate with each other to perform the same processing. Furthermore, details shown in the embodiments can be modified as appropriate without departing from the spirit of the invention. [Explanation of Symbols]

[0038] 30 Loading and unloading vehicles 100 Cargo handling support devices 110 Image acquisition unit 110a 1st image acquisition unit 110b Second image acquisition unit 120 Control Unit 130 Setting section 140 Storage section 144 Learning Models 145 3D-CAD data (shape data) 200 Cargo Handling Support System 210 Support equipment 211 Hochi Department 211a Display section f fork H load L Outline M Center of gravity (center of gravity of individual loads) Mc Center of gravity (center of gravity of the entire load) P Palette R judgment area

Claims

1. An image acquisition unit that acquires an image including the pallet and the load on the pallet, A storage unit that pre-stores the shape data of the pallet and the load, Control unit and Equipped with, The control unit, The image acquired by the image acquisition unit is compared with the shape data stored in the storage unit to determine the center of gravity of the load relative to the pallet. Using a machine learning model trained with training data in which the outline of the load is assigned to an image of the load, the outlines of the pallet and the load are extracted from the image acquired by the image acquisition unit. The outline and the shape data are compared. Cargo handling support device.

2. An image acquisition unit that acquires an image including a pallet and the load on the pallet, A storage unit that pre-stores the shape data of the pallet and the load, Control unit and Equipped with, The control unit, The image acquired by the image acquisition unit is compared with the shape data stored in the storage unit to determine the center of gravity of the load relative to the pallet. Based on the aforementioned center of gravity position, the quality of the loading condition is determined. Cargo handling support device.

3. It is equipped with a setting unit that allows setting a region for determining the center of gravity position, The control unit determines whether the loading condition is good or bad based on the calculated center of gravity position and the determination area. The cargo handling support device according to claim 2.

4. The control unit, Determine the individual center of gravity positions of the multiple loads on the pallet, Based on the individual center of gravity positions of the aforementioned multiple loads, the overall center of gravity position of the aforementioned multiple loads is determined. Based on the overall center of gravity position and the determination area, the quality of the loading condition is determined. The cargo handling support device according to claim 3.

5. The aforementioned shape data is three-dimensional CAD data. A cargo handling support device according to claim 1 or claim 2.

6. The image acquisition unit includes a distance sensor and acquires an image containing distance information. A cargo handling support device according to claim 1 or claim 2.

7. A cargo handling support device according to any one of claims 1 to 4, A support device that provides support for cargo handling based on the calculation results of the cargo handling support device, A cargo handling support system equipped with the following features.

8. The support device has a display unit that displays the calculated center of gravity of the load. The cargo handling support system according to claim 7.

9. A cargo handling device that is equipped with at least a part of the image acquisition unit, the control unit, the storage unit, and the support device of the cargo handling support system according to claim 7, or that communicates with at least a part of the image acquisition unit, the control unit, the storage unit, and the support device to perform cargo handling based on the calculation results of the cargo handling support device.

Images