Cargo handling support device, cargo handling vehicle, cargo handling support method, and cargo handling support program

The cargo handling support device enhances automated pallet recognition and storage destination detection, ensuring safe and efficient cargo handling operations by displaying evaluation values for operator confirmation, reducing the risk of accidents and operational costs.

JP2024168050APending Publication Date: 2024-12-05SUMITOMO HEAVY IND LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023084443
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Automated cargo handling vehicles face challenges in accurately recognizing pallets due to varying measurement conditions, leading to potential damage to vehicles or cargo during handling operations.

Method used

A cargo handling support device equipped with a photography unit, detection unit, and display control unit that acquires images, detects cargo handling targets, and displays an evaluation value representing the recognition degree, allowing for safe and automated cargo handling operations.

Benefits of technology

Enables safe automation of cargo handling by ensuring accurate detection and recognition of pallets and storage destinations, reducing the risk of accidents and operational costs, and allowing operators with less experience to perform tasks effectively.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168050000001_ABST
    Figure 2024168050000001_ABST
Patent Text Reader

Abstract

To appropriately automate cargo handling operations.SOLUTION: A forklift 1 is equipped with an imaging unit 24 that acquires images including objects to be handled by a fork 12, and a control unit 27. The control unit 27 detects the cargo to be handled on the basis of the image acquired by the imaging unit 24, and displays an evaluation value indicating the degree of recognition of the cargo to be handled based on the image in the detection of the cargo to be handled in the display unit 23. This allows an operator to check the soundness of the cargo to be handled based on the image by visually checking the evaluation value, and to perform the cargo handling operation safely. Therefore, it is possible to automate the cargo handling work appropriately while ensuring safety.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a cargo handling support device, a cargo handling vehicle, a cargo handling support method, and a cargo handling support program. [Background technology]

[0002] In recent years, cargo handling vehicles such as forklifts that handle cargo are becoming more automated (unmanned). For example, in the technology described in Patent Document 1, a forklift checks its surroundings in the work area and autonomously handles cargo. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-187672 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there are still many technical challenges to overcome in automating loading and unloading vehicles. For example, when automating loading and unloading work by measuring pallets, sensing may not be performed properly depending on the measurement conditions such as the influence of light, and the pallet may not be recognized properly. If loading and unloading work is performed without properly recognizing the pallet, problems such as damage to the vehicle body or the cargo may occur.

[0005] The present invention has been made in consideration of the above circumstances, and has an object to suitably automate loading and unloading operations. [Means for solving the problem]

[0006] The loading and unloading support device according to the present invention comprises: an imaging unit for acquiring an image including a loading target in the loading / unloading work unit; A detection unit that detects the loading target based on the image acquired by the photographing unit; a display control unit that causes a display unit to display an evaluation value representing a recognition degree of the loading object based on the image in the detection of the loading object by the detection unit; Equipped with. Effect of the Invention

[0007] According to the present invention, loading and unloading operations can be suitably automated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a side view of the forklift according to the embodiment during loading / unloading work on a load shelf. [Diagram 2] 1 is a block diagram showing a schematic control configuration of a forklift according to an embodiment of the present invention; [Diagram 3] 10 is a flowchart illustrating a procedure for a cargo collection assistance process according to the embodiment. [Figure 4] 11 is a diagram for explaining a cargo collection assistance process according to an embodiment. FIG. [Diagram 5] FIG. 2 is a side view of the forklift according to the embodiment during loading / unloading (placing) of a load on a load shelf. [Figure 6] 10 is a flowchart showing the procedure of a load placement assistance process according to the embodiment. [Figure 7] 11 is a diagram for explaining the loading assistance process according to the embodiment. FIG. [Figure 8] FIG. 11 is a block diagram showing a schematic control configuration of a forklift and a remote control device according to a modified example of the embodiment. [Figure 9] FIG. 11 is a block diagram showing a schematic control configuration of a forklift and an imaging unit according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

[0010] [Forklift configuration] FIG. 1 is a side view of a forklift 1 according to this embodiment during loading / unloading work on a shelf 40. As shown in FIG. The forklift 1 according to this embodiment is equipped with the cargo handling support device according to the present invention, and thus can suitably perform various cargo handling operations including placing cargo L (pallets 30) on the cargo shelf 40 and removing cargo L from the cargo shelf 40. The forklift 1 is an example of a cargo handling vehicle according to the present invention. Hereinafter, "placing (or removing) a load L (pallet 30)" means placing (or removing) the load L (pallet 30) on (or from) the shelf 42 of the load shelf 40, unless otherwise specified.

[0011] Specifically, the vehicle body 10 of the forklift 1 includes a vehicle body 11, forks 12, a lifting body (lift) 13, a mast 14, and wheels 15. The mast 14 is provided in front of the vehicle body 11, and is driven by a drive source (not shown) to tilt the vehicle body 11 forward and backward. The lifting body 13 is driven by a drive source (not shown) to rise and lower along the mast 14. A pair of left and right forks 12 for holding a load L, a pallet 30, etc. are attached to the lifting body 13. The pair of forks 12 can be tilted and raised and lowered relative to the vehicle body 11 by driving the mast 14 and the lifting body 13. The forks 12 are an example of a loading / unloading unit according to the present invention.

[0012] The pallet 30 is a loading platform on which the load L is placed, and is formed in the shape of a short rectangular plate having two fork pockets 32 on a side surface 33 (see FIG. 4). The pallet 30 is held by the forklift 1 by inserting a pair of forks 12 into the two fork pockets 32.

[0013] FIG. 2 is a block diagram showing a schematic control configuration of the forklift 1. As shown in FIG. As shown in this figure, in addition to the above-mentioned configuration, the forklift 1 is equipped with a drive unit 21, an operation unit 22, a display unit 23, an imaging unit 24, a storage unit 26, and a control unit 27. The cargo handling support device according to the present invention includes at least the imaging unit 24 and the control unit 27.

[0014] The drive unit 21 includes a travel motor, a steering motor, and a loading motor (all not shown), which are various drive sources of the forklift 1. The travel motor drives the drive wheels of the wheels 15. The steering motor rotates (steers) the steering wheels of the wheels 15. The loading motor is a drive source that performs the respective operations of raising and lowering the lifting body 13 and tilting the mast 14.

[0015] The operation unit 22 is an operation means by which an operator (driver) on board performs various operations. The operation unit 22 includes, for example, a handle, pedals, levers, various buttons, etc., and outputs operation signals to the control unit 27 according to the contents of these operations.

[0016] The display unit 23 is, for example, a liquid crystal display, an organic electroluminescence display, or other display, and displays various information based on a display signal input from the control unit 27. The display unit 23 may be a touch panel that also serves as part of the operation unit 22, or may include a speaker capable of displaying (outputting) audio.

[0017] The photographing unit 24 photographs the area in front of the forklift 1, acquires image information, and outputs the acquired information to the control unit 27. The photographing unit 24 in this embodiment is disposed in the lower part of the lifting body 13, in the center in the width direction, and is installed facing forward (see FIG. 1). The type of image acquired by the image capturing unit 24 is not particularly limited, and may be, for example, an RGB image. The image capturing unit 24 may be a scanner device that acquires a two-dimensional or three-dimensional shape by scanning light or sound waves and detecting reflected waves. Alternatively, the image capturing unit 24 may be an RGB-D camera (depth camera), LiDAR (Light Detection And Ranging), or a compound eye camera that can acquire an image including distance information (depth information). The position of the image capturing unit 24 is not particularly limited, and may be, for example, the upper part of the lifting body 13 (a position where the image capturing range does not largely overlap with the load L on the fork 12) or the tip of the fork 12.

[0018] The storage unit 26 is a memory configured by, for example, a random access memory (RAM) or a read only memory (ROM), and stores various programs and data. In the memory unit 26 of this embodiment, in addition to a cargo collection support program 260 and a cargo placement support program 261 for executing the cargo collection support process (see Figure 3) and the cargo placement support process (see Figure 6) described below, a first learning model 262 and a second learning model 263 are pre-stored.

[0019] The first learning model 262 is for detecting the pallet 30 from an image. The first learning model 262 is created in advance by machine learning using, as training data, sample images of a plurality of pallets 30 each having a side surface 33 (see FIG. 4) with a fork pocket 32, and is stored in the storage unit 26. The second learning model 263 is for detecting the shelf boards 42 of the luggage shelf 40 from an image. The second learning model 263 is created in advance by machine learning using, as teacher data, sample images of a plurality of luggage shelves 40 to which the top surfaces (shelf surfaces) 42a or front surfaces 42b (see FIG. 7) of the shelf boards 42 are added, and is stored in the storage unit 26. The first learning model 262 and the second learning model 263 may be an AI (Artificial Intelligence) having a deep learning neural network.

[0020] The control unit 27 is configured with, for example, a CPU (Central Processing Unit) and controls the operation of each part of the forklift 1. Specifically, the control unit 27 operates the drive unit 21 based on the operation content of the operation unit 22, deploys a program pre-stored in the storage unit 26, and executes various processes in cooperation with the deployed program.

[0021] [Pickup Assistance Processing] Next, the operation of the forklift 1 when performing the cargo handling assistance process will be described. FIG. 3 is a flowchart showing the procedure of the item receiving assistance process, and FIG. 4 is a diagram for explaining the item receiving assistance process.

[0022] The goods collection support process is a process for supporting the operation of an operator by automating a part of the goods handling work. Specifically, the goods collection support process is a process in which, when the forklift 1 picks up the goods L (pallet) from the goods shelf 40, the forklift 1 automatically recognizes the pallet 30 and carries out the goods collection. This goods collection support process is carried out by the control unit 27 of the forklift 1 reading and deploying the goods collection support program 260 from the memory unit 26 based on the operation performed by the operator (driver) or the like. In this load-taking operation, the forklift 1 takes out the load L placed on the pallet 30 from the shelf 42 of the load shelf 40. Here, it is assumed that the forklift 1 has traveled up to the front of the load shelf 40.

[0023] As shown in FIG. 3, when the cargo handling assistance process is executed, first, the control unit 27 photographs the cargo shelf 40 in front of the vehicle body with the photographing unit 24 based on the operation of the operator (step S1). At this time, the control unit 27 uses the display unit 23 as a viewfinder to display the image of the photographing unit 24, and acquires image data within the photographing range of the photographing unit 24 at that time based on the operation performed by the operator. While watching the display unit 23, the operator moves the photographing unit 24 by raising and lowering the forks 12, etc., and performs the operation to photograph the load L to be handled and its pallet 30 in a state where it is within the photographing range of the photographing unit 24. The control unit 27 stores the acquired image data in the memory unit 26. As a result, an image (image data) including the load L (pallet 30) to be handled is acquired.

[0024] Next, the control unit 27 detects the palette 30 from the image data acquired in step S1 (step S2). Specifically, the control unit 27 inputs the acquired image data of the shelf 40 into the first learning model 262, and acquires an image portion of the pallet 30 included in the image data. In this embodiment, as shown in Fig. 4(a), the side surface 33 of the pallet 30 including the two fork pockets 32 is detected. The detection method for the palette 30 is not particularly limited as long as it uses image data (including point cloud data, etc.), and may be detection using, for example, a sliding window or deep learning inference.

[0025] Next, the control unit 27 selects the pallet 30 to be handled based on the operation of the operator (step S3). Here, when multiple pallets 30 are detected in step S2, the control unit 27 causes the display unit 23 to display the multiple pallets 30 as candidates (for loading and unloading). Then, the control unit 27 selects one of the displayed (candidate) pallets 30 as the loading and unloading target based on the operation of the operator. At this time, the control unit 27 causes the display unit 23 to display the image used in step S2, the pallet display unit 23a, and the number display unit 23b. The pallet display unit 23a is a portion that indicates the detected pallet 30 (side surface 33) on the image used in step S2. The number display unit 23b is a portion that displays a number obtained by numbering the pallet display unit 23a on the image used in step S2, and is displayed adjacent to the corresponding pallet display unit 23a. In addition, if only one pallet 30 is detected in step S2, the control unit 27 may check, based on the operator's operation, whether the single pallet 30 is the loading target, or may omit step S3. Furthermore, if the pallet 30 to be loaded has not been detected, the control unit 27 may return to step S1 and retake the photographing, or may interrupt the process, based on the operation of the operator.

[0026] Next, the control unit 27 starts the loading / unloading operation (unloading operation) of unloading the pallet 30, which is the loading target selected in step S3, from the shelf 40 (step S4). Specifically, in this pick-up operation, first, the control unit 27 extracts the fork pockets 32 from the latest image of the pallet 30 and identifies their positions. Then, the control unit 27 controls the drive unit 21 to insert the forks 12 into the fork pockets 32. The loading and unloading operation may be started automatically after the selection of the pallet 30 in step S3, or may be started after a start operation by the operator is received.

[0027] Next, the control unit 27 acquires image data of the pallet 30 by the photographing unit 24 during the loading operation, and detects the pallet 30 to be loaded based on the acquired image (step S5). Here, the pallet 30 is photographed and detected in the same manner as in steps S1 and S2 described above. Note that immediately after the start of the work in step S4, the process in step S5 may be omitted by using the data in steps S1 and S2.

[0028] Next, the control unit 27 calculates an evaluation value representing the degree of recognition of the palette 30 based on the image data in the detection of the palette 30, and causes the display unit 23 to display the evaluation value (step S6). As this evaluation value, for example, a reliability score of the recognition result, an acquired point cloud amount, etc. can be used. The reliability score is an index indicating the probability that the target object (here, the pallet 30) is included in the extracted rectangle. The point cloud amount is a point cloud amount acquired for recognizing the pallet 30, for example, a point cloud amount within a range recognized as the side surface 33. The control unit 27 calculates (acquires) the evaluation value, and causes the display unit 23 to display a time series of the evaluation values, for example, as shown in FIG. 4(b). Here, the control unit 27 may display information related to the recognition of the pallet 30 and the work status (condition) together with the evaluation value on the display unit 23. The information related to the recognition of the pallet 30 is, for example, the type and size of the pallet. The work status may be, for example, waiting for a start command, in automatic operation, interrupted, completed, etc. In this manner, during automatic operation of the forklift 1, information such as the time series of the evaluation values ​​and the work status are displayed on the display unit 23. Therefore, the operator can easily recognize information such as the evaluation values ​​and the work status by looking at the display unit 23.

[0029] Next, the control unit 27 determines whether or not the evaluation value is normal (step S7). Here, the control unit 27 determines that the evaluation value calculated in step S6 is abnormal if the evaluation value fluctuates significantly by more than a predetermined range during the execution of the pick-up operation, or if it falls below a predetermined threshold value that has been set in advance. This determines the soundness of the image-based detection of the pallet 30, for example, whether the amount of point cloud is insufficient, whether the point cloud for extracting the side surface 33 of the pallet 30 is properly connected as one surface, etc. If the evaluation value fluctuates significantly, there is a possibility that the recognition result has been different from the time of the original judgment. If the evaluation value falls below the threshold value, there is a possibility that the measurement for the autonomous driving is inaccurate. Then, when it is determined that the evaluation value is normal (step S7; Yes), the control unit 27 shifts the process to step S9 described later.

[0030] On the other hand, when it is determined in step S7 that the evaluation value is not normal (step S7; No), the control unit 27 stops the operation of the forklift 1 (vehicle body 10) and notifies the operator of the occurrence of an abnormality (step S8). In this case, the manner of notification is not particularly limited, and a warning may be displayed on the display unit 23, or a warning sound may be output from a speaker (not shown). Then, the control unit 27 determines whether to stop (stop) the automatic operation or to continue it based on the operation of the operator (step S8a). If the automatic operation is to be stopped, the goods collection support process is ended. In addition, the operator may be able to intervene in the work at any time during the loading operation, for example by operating the operation unit 22, and interrupt (stop) or continue the automatic operation.

[0031] Next, the control unit 27 determines whether or not the loading operation started in step S4 is completed (step S9). Then, when it is determined that the unloading operation is not completed (step S9; No), the control unit 27 shifts the process to the above-mentioned step S5, and repeats the processes of steps S5 to S9. As a result, while the vehicle body 10 is approaching the load L (pallet 30), the evaluation value indicating the degree of recognition of the pallet 30 is successively updated, and the evaluation value is displayed on the display unit 23 and its success or failure is determined in real time. Therefore, loading and unloading work can be safely performed while constantly checking the soundness of the pallet 30 detection based on the image by determining the success or failure of the evaluation value and visually confirming it by the operator.

[0032] In step S9, if it is determined that the cargo unloading operation is completed (step S9; Yes), that is, if it is determined that the forks 12 have been inserted into the fork pockets 32 of the pallet 30, the control unit 27 executes other tasks to be automatically processed (step S10). For example, the control unit 27 raises the forks 12 inserted into the fork pockets 32 to lift the pallet 30 and the load L from the shelf board 42, and then moves the vehicle body 10 backward to a predetermined position. Thereafter, the control unit 27 ends the load collection assistance process. At this time, the control unit 27 may cause the display unit 23 to display the work status of "Complete". If no other tasks to be automatically processed are specifically set, step S10 may be omitted.

[0033] [Cargo placement assistance processing] Next, the operation of the forklift 1 when performing the load placement assistance process will be described. FIG. 5 is a side view of the forklift 1 during loading / unloading (placing) the goods on the shelf 40, FIG. 6 is a flow chart showing the procedure of the goods placement support process, and FIG. 7 is a diagram for explaining the goods placement support process.

[0034] The goods placement support process is a process for supporting the operation of the operator by automating a part of the cargo handling work. Specifically, the goods placement support process is a process in which, when the forklift 1 places the goods L (pallet) on the goods shelf 40, the forklift 1 automatically recognizes the goods shelf 40 and executes the goods placement (shelf placement). This goods placement support process is executed by the control unit 27 of the forklift 1 reading and developing the goods placement support program 261 from the memory unit 26 based on the operation performed by the operator (driver) and the like. In this load placement, the forklift 1 places the load L (pallet 30) held by the forks 12 on the shelf 42 of the load shelf 40. Here, as shown in Fig. 5, the forklift 1 is assumed to have traveled to the front of the load placement destination P (see Fig. 7) of the load shelf 40 where the load L (pallet 30) is to be placed, while still holding the load L (pallet 30) with the forks 12. The "load placement destination P" refers to a region of the top surface 42a of a given shelf 42 within a given range (for example, a range of a given width x a given depth corresponding to the size of the pallet 30) in the front of the vehicle body, and may include information such as the height or the number of levels of the shelf 42 (which level).

[0035] As shown in FIG. 6, when the luggage placement assistance process is executed, first, the control unit 27 photographs the luggage shelf 40 in front of the vehicle body with the photographing unit 24 based on the operation of the operator (step T1). At this time, the control unit 27 uses the display unit 23 as a viewfinder to display the image of the photographing unit 24, and acquires image data within the photographing range of the photographing unit 24 at that time based on the operation performed by the operator. While watching the display unit 23, the operator moves the photographing unit 24 by raising and lowering the forks 12, etc., and performs the operation of photographing with the area of ​​the upper surface 42a of the shelf board 42 including the placement destination P of the loading target within the photographing range of the photographing unit 24, as shown in Fig. 7. The control unit 27 stores the acquired image data in the memory unit 26. As a result, an image (image data) including the placement destination P of the loading target is acquired.

[0036] Next, the control unit 27 detects the shelf board 42 (loading destination P) from the image data acquired in step T1 (step T2). Specifically, the control unit 27 inputs the acquired image data of the shelf 40 into the second learning model 263, and acquires an image portion of the upper surface 42a of the shelf board 42 included in the image data. In this embodiment, of the shelf board 42, (a predetermined range of) the substantially horizontal upper surface 42a is detected as the placement destination P. The method for detecting the loading destination P is not particularly limited as long as it uses image data (including point cloud data, etc.). For example, depth data may be expanded into a three-dimensional point cloud and a horizontal plane may be extracted.

[0037] Next, the control unit 27 selects a placement destination P for the cargo to be handled based on the operation of the operator (step T3). Here, in step T2, when the upper surfaces 42a (of a specified range) of multiple shelves 42 are detected as candidates for the loading destination P, the control unit 27 selects one of them as the loading destination P based on the operator's operation. Furthermore, if only one top surface 42a is detected in step T2, the control unit 27 may confirm, based on the operator's operation, whether that one top surface 42a is the loading destination P of the loading object, or may omit step T3. If the placement destination P (upper surface 42a) of the loading target has not been detected, the control unit 27 may return to step T1 and retake the photograph, or may interrupt the process, based on the operator's operation.

[0038] Next, the control unit 27 starts the loading / unloading operation (load placement operation) of placing the held load L (pallet 30) at the load placement destination P of the loading target selected in step T3 (step T4). Specifically, in this load placement operation, the control unit 27 controls the drive unit 21 to raise and lower the forks 12 while moving the vehicle body 10 forward, and place the load L (pallet 30) held by the forks 12 at the load placement destination P. The loading and unloading operation may be started automatically after the loading destination P is selected in step T3, or may be started after a start operation by an operator is received.

[0039] Next, the control unit 27 acquires image data of the shelf 40 by the photographing unit 24 during the placement operation, and detects the placement destination P of the object to be loaded based on the image (step T5). Here, similarly to steps T1 and T2 described above, the shelf 40 is photographed and the placement destination P is detected. Note that immediately after the start of the work in step T4, the processing in step T5 may be omitted by using the data in steps T1 and T2 described above.

[0040] Next, the control unit 27 calculates an evaluation value representing the degree of recognition of the placement destination P based on the image data in the detection of the placement destination P, and causes the display unit 23 to display the evaluation value (step T6). As the evaluation value, for example, the flatness of the upper surface 42a, the acquired point cloud amount, etc. can be used. The point cloud amount is the amount of points acquired for recognizing the upper surface 42a. The control unit 27 calculates (acquires) the evaluation value, and causes the display unit 23 to display the time series of the evaluation value (see FIG. 4(b)). Here, the control unit 27 may display information related to the recognition of the placement destination P and the work status (state) on the display unit 23 together with the evaluation value. The information related to the recognition of the placement destination P is, for example, the size and flatness of the upper surface 42a. The work status is, for example, waiting for a start command, in automatic operation, interrupted, completed, etc. In this manner, during automatic operation of the forklift 1, information such as the time series of the evaluation values ​​and the work status are displayed on the display unit 23. Therefore, the operator can easily recognize information such as the evaluation values ​​and the work status by looking at the display unit 23.

[0041] Next, the control unit 27 judges whether or not the evaluation value is normal (step T7). Here, the control unit 27 judges that the evaluation value calculated in step T6 is abnormal if the evaluation value during the placement operation fluctuates by more than a predetermined amount or falls below a predetermined threshold value set in advance. This judges the soundness of the image-based detection of the placement destination P, for example, by checking whether the point cloud amount is sufficient, whether the point cloud for extracting the upper surface 42a of the shelf board 42 is properly connected as one surface, etc. If the evaluation value fluctuates significantly, there is a possibility that the recognition result has been different from the time of the judgment in the first place. If the evaluation value falls below the threshold value, there is a possibility that the measurement for the autonomous driving is not accurate. If it is determined that the evaluation value is normal (step T7; Yes), the control unit 27 advances the process to step T9, which will be described later.

[0042] On the other hand, when it is determined in step T7 that the evaluation value is not normal (step T7; No), the control unit 27 stops the operation of the forklift 1 (vehicle body 10) and notifies the operator of the occurrence of an abnormality (step T8). In this case, the manner of notification is not particularly limited, and a warning may be displayed on the display unit 23, or a warning sound may be output from a speaker (not shown). Then, the control unit 27 determines whether to stop (stop) or continue the automatic operation based on the operation of the operator (step T8a). If the automatic operation is to be stopped, the load placement assistance process is ended. In addition, the operator may be able to intervene in the work at any time during the loading operation, for example by operating the operation unit 22, and interrupt (stop) or continue the automatic operation.

[0043] Next, the control unit 27 determines whether or not the loading operation started in step T4 is completed (step T9). Then, when it is determined that the loading operation is not completed (step T9; No), the control section 27 shifts the process to the above-mentioned step T5, and repeats the processes of steps T5 to T9. As a result, while the vehicle body 10 is approaching the loading destination P, the evaluation value indicating the degree of recognition of the loading destination P is successively updated, and the evaluation value is displayed on the display unit 23 and its success or failure is determined in real time. Therefore, loading and unloading operations can be safely performed while constantly checking the soundness of the detection of the pallet 30 based on the image by determining the success or failure of the evaluation value and visually confirming it by the operator.

[0044] In step T9, if it is determined that the loading operation is completed (step T9; Yes), that is, if it is determined that the pallet 30 has been placed at the loading destination P, the control unit 27 executes other operations to be automatically processed (step T10). For example, the control unit 27 moves the vehicle body 10 backward to remove the forks 12 from the pallet 30 (fork pockets 32), and then lowers the forks 12. Thereafter, the control unit 27 ends the load placement assistance process. At this time, the control unit 27 may cause the display unit 23 to display the operation status of "Complete." If no other tasks to be automatically processed are specifically set, step T10 may be omitted.

[0045] [Technical effect of the present embodiment] As described above, according to this embodiment, an image (image data) including the loading target (pallet 30, loading destination P, etc.) of the fork 12 is acquired by the photographing unit 24, and the loading target is detected based on this image. Then, an evaluation value indicating the degree of recognition of the loading target based on the image in the detection of the loading target is displayed on the display unit 23. This allows the operator to visually check the evaluation value and safely carry out loading and unloading work while checking the soundness of the image-based loading and unloading object detection. Therefore, loading and unloading work can be suitably automated while ensuring safety. As a result, even operators with little experience can work safely. In addition, by having an operator monitor the automatic operation, the need for safety devices that are implemented during automation can be reduced, and the cost of such devices can be reduced.

[0046] Furthermore, according to this embodiment, the control unit 27 stops the operation of the forklift 1 when it is determined that the evaluation value is not normal. This makes it possible to prevent accidents caused by errors in detecting the objects being loaded.

[0047] Furthermore, according to this embodiment, the loading and unloading objects detected by the detection unit are displayed as candidates on the display unit 23, and one of the candidates for the loading and unloading objects displayed on the display unit 23 is selected based on the operation of the operator. This allows the operator to easily select the loading target.

[0048] Furthermore, according to this embodiment, while the forklift 1 is performing a loading / unloading operation on a loading object, image acquisition, detection of the loading object, and display of the evaluation value are sequentially performed. This allows loading and unloading operations to be carried out safely while checking the accuracy of image-based detection of loading objects in real time during the operation.

[0049] Furthermore, according to the present embodiment, the start, continuation, and interruption of the loading and unloading operation of the forklift 1 on the loading target are executed based on the operation of the operator. This allows the operator to decide whether to execute or halt automatic operation, ensuring safety more reliably. In addition, the operator can monitor the status of automatic operation by visually checking the evaluation value, making it easier for him to make decisions about starting, continuing, or halting loading and unloading operations.

[0050] [Variation 1] In the above embodiment, the operator rides on the forklift 1. However, the operator may remotely control (remotely pilot) the forklift 1. Specifically, as shown in Fig. 8, an operator may remotely operate the forklift 1 by using a remote control device 50 capable of communicating with the forklift 1 through a communication network NW, for example. The remote control device 50 includes an operation unit 52 through which the operator performs various operations, a display unit 53 that displays various information including images from the forklift 1, a storage unit 56, and a control unit 57. The operation unit 52 and the display unit 53 function at least in the same manner as the operation unit 22 and the display unit 23 in the above embodiment. The degree of sharing of processing between the forklift 1 and the remote control device 50 is not particularly limited. However, it goes without saying that data required for the processing (for example, a learning model for detecting a loading object and various programs, etc.) must be stored in advance in the storage unit 56 of the remote control device 50. The present invention can be suitably applied to remote-controlled forklifts because it can suitably automate loading and unloading operations that require relatively high operational skills, such as loading and unloading.

[0051] [Variation 2] In the above embodiment, the photographing unit 24 for capturing images of the loading and unloading target is mounted on the forklift 1. However, the photographing unit may be installed on the environment (facility) side. Specifically, as shown in Fig. 9, an image including the loading target may be acquired by, for example, an image capturing unit 64 installed around the load shelf 40. In this case, the image capturing unit 64 captures, for example, the load shelf 40 from a diagonally upward front direction, and transmits the acquired image (image data) to the forklift 1 via the communication network NW. The device type of the image capturing unit 64 and the type of image to be acquired may be the same as those of the image capturing unit 24 in the above embodiment.

[0052] [others] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments (including modified examples). For example, in the goods receiving assistance process, the pallet 30 is detected by the first learning model 262, but the goods L placed on the pallet 30 may also be detected at the same time. This is useful when the goods L are fixed-shaped goods, etc.

[0053] In addition, in the load placement assistance process of the above embodiment, the top surface 42a of the shelf 42 is detected as the load placement destination P, but the front surface 42b (see FIG. 7) may be detected instead of the top surface 42a. The operator can visually confirm that the top of the shelf 42 (top surface 42a) is open. In this case, the second learning model 263 detects the front surface 42b of the load shelf 40 from the image.

[0054] In the above embodiment, the loading and unloading work has been described by taking and placing an item on the shelf 40. However, the loading and unloading work according to the present invention is not limited to being done on a shelf, and includes, for example, taking and placing an item on the floor (laying flat), etc., as long as the loading object (item, pallet, or loading destination) can be detected from an image.

[0055] Furthermore, the loading vehicle according to the present invention is not limited to a forklift as long as it has a loading / unloading section and performs loading / unloading operations on a loading target, but may include, for example, a transport vehicle. In addition, the details shown in the above embodiment can be modified as appropriate without departing from the spirit of the invention. [Explanation of symbols]

[0056] 1. Forklift (cargo handling vehicle) 10. Body 12 Fork (loading and unloading section) 22 Control section 23 Display section 23a Palette display 23b Number display section 24 Photography Department 26 Memory section 27 Control unit (detection unit, display control unit, determination unit, operation control unit, second display control unit, selection unit) 30 pallets (for handling) 32 Fork Pocket 33 Side 40 Luggage Shelf 42 Shelf 42a Top side 42b front 50 Remote Control Device 52 Operation section 53 Display section 64 Filming Department 260 Cargo Assistance Program 261 Loading Assistance Program 262 First Learning Model 263 Second Learning Model L load P Destination (loading target)

Claims

1. an imaging unit that captures an image including an object to be handled in the loading and unloading unit; a detection unit that detects the object to be handled based on the image acquired by the photographing unit; a display control unit that causes a display unit to display an evaluation value representing a degree of recognition of the object based on the image in the detection of the object by the detection unit; A loading and unloading support device.

2. a determination unit that determines whether the evaluation value is normal; an operation control unit that stops operation of the cargo handling vehicle having the cargo handling unit when the determination unit determines that the evaluation value is not normal; Equipped with The cargo handling support device according to claim 1.

3. a second display control unit that causes the display unit to display the object detected by the detection unit as a candidate object; a selection unit that selects one of the candidates for the cargo handling object displayed on the display unit based on an operation by an operator; Equipped with The cargo handling support device according to claim 1.

4. The acquisition of the image by the photographing unit, the detection of the loading object by the detection unit, and the display of the evaluation value by the display control unit are sequentially performed when a loading vehicle having the loading operation unit is performing loading and unloading work on the loading object. The cargo handling support device according to claim 1.

5. an operation control unit that controls the operation of the cargo handling vehicle having the cargo handling unit; The operation control unit starts, continues, and interrupts the loading and unloading operation of the loading and unloading vehicle on the loading target based on an operation by an operator. The cargo handling support device according to claim 1.

6. The cargo handling support device according to any one of claims 1 to 5, The loading and unloading unit; A loading vehicle equipped with:

7. The control unit an imaging step of acquiring an image including the object to be handled in the loading and unloading section; a detection step of detecting the object to be handled based on the image acquired in the photographing step; a display control step of displaying an evaluation value representing a degree of recognition of the object based on the image in the detection step of the object on a display unit; A method for supporting cargo handling.

8. Computer, an imaging unit for acquiring an image including an object to be handled in the loading and unloading unit; a detection unit that detects the object to be handled based on the image acquired by the photographing unit; a display control unit that causes a display unit to display an evaluation value representing a degree of recognition of the object based on the image in the detection of the object by the detection unit; A cargo handling assistance program that functions as a

Citation Information

Patent Citations

  • Monitoring system and monitoring method for automatic forklift

    JP2021187672A