Cargo handling condition determination device, driving assistance system, and driving assistance program
The cargo handling state determination device uses imaging and reference axis detection to prevent cargo from falling by adjusting vehicle speed and turning speed based on cargo stability, addressing the issue of unstable loads in forklifts.
Patent Information
- Application Number
- JP2021020907
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-12
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2041-02-12
AI Technical Summary
Accidents occur frequently in cargo handling vehicles like forklifts due to unstable cargo loads, as workers often lack time to visually check and adjust the stability of the cargo before transportation, and existing technologies fail to prevent such accidents effectively.
A cargo handling state determination device and system that uses an imaging device to capture images of the cargo and pallet, detects the outer coordinates of the cargo, sets a reference axis based on the vehicle and pallet coordinates, and determines if the cargo is unevenly loaded, restricting the vehicle's speed and turning speed if instability is detected.
Prevents cargo from falling during transportation by accurately determining cargo stability and adjusting vehicle operations to ensure safe handling.
Smart Images

Figure 0007578499000001 
Figure 0007578499000002 
Figure 0007578499000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a load handling state determination device, a driving assistance system, and a driving assistance program. [Background technology]
[0002] Techniques for assisting the operation of a forklift have been proposed (see, for example, Patent Document 1). Patent Document 1 shows a technique for preventing the positional relationship between a moving body and an obstacle from becoming unclear. The safety device disclosed in Patent Document 1 is mounted on a forklift and includes two imaging devices (stereo cameras) and an image processing device. The imaging devices are positioned so that the uppermost side of the vertical angle of view is above the horizontal plane. The image processing device calculates and stores the position of the obstacle from an image captured by the imaging devices. When the imaging device no longer captures an image of the obstacle, the image processing device estimates the position of the obstacle based on the stored position information.
[0003] Also, a technology has been proposed for grasping how objects, such as cardboard boxes loaded on a pallet, are loaded when a robot performs a depalletizing process for unloading the objects from the pallet (see, for example, Patent Document 2). The image processing device disclosed in Patent Document 2 is connected to a controller of a depalletizing robot, and includes a projection means, an imaging means, and a control means. The projection means projects a lattice pattern onto a plurality of workpieces loaded on the pallet. The imaging means images the projected lattice pattern. The control means creates information related to the workpieces based on first image data obtained by imaging the lattice pattern projected in a state where a pallet loaded with workpieces is not present, and second image data obtained by imaging the lattice pattern projected in a state where a pallet loaded with workpieces is present. The control means can create information related to the height of the workpieces based on the distance between the intersections of the lattice pattern in each image data. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-89636 A [Patent Document 2] JP 2003-97918 A Summary of the Invention [Problem to be solved by the invention]
[0005] Incidentally, at work sites where loads are transported using forklifts, such as within a factory site or in a warehouse, accidents have occurred in which loads being transported by the forklift are dropped and damaged.
[0006] In actual work of this kind, the goods placed on the pallet are lifted from below by the forklift's claws and transported as is. The frequency of loads falling during transport by forklift largely depends on the stability of the loads placed on the pallets. Therefore, in order to prevent loads from falling during transport, it is generally desirable to visually check the stability of the load before transporting it. Ideally, loads to be transported by forklift should be stacked so that the center of gravity is located in the center of the pallet, but in busy work sites, it is difficult to stack loads carefully and pay sufficient attention to time-consuming visual inspections.
[0007] On the other hand, if a worker notices that a package is unstable before or during transportation by a forklift, it is desirable to correct the package state each time. However, in busy work sites, workers may not have the time to carry out the time-consuming task of correcting the package state, so they may think, "It will be fine without correction," and continue working without correcting the package state.
[0008] According to the technology described in the above Patent Document 1, it is possible to avoid collision with an obstacle and prevent accidents in which cargo falls due to collision, but it is not possible to prevent accidents in which cargo falls that are not caused by collision. According to the technology described in the above Patent Document 2, when the cargo loaded on the pallet is in a different loading pattern from the loading pattern transmitted from the host computer, information on the deviation can be created. However, when the image processing device in Patent Document 2 is applied to the operation support of a forklift, if information on the reference loading pattern of the cargo is not obtained, the loading state of the cargo cannot be grasped and the cargo cannot be prevented from falling. Moreover, it is practically difficult to capture two types of lattice pattern projection images with and without cargo and pallets after creating information on a stable loading pattern every time the cargo loaded on the pallet is transported by the forklift.
[0009] The present invention has been made in consideration of the above-mentioned circumstances, and an object of the present invention is to prevent accidents in which luggage falls from a loading vehicle due to unstable luggage transportation. [Means for solving the problem]
[0010] In order to achieve the above-mentioned object, a load handling state determination device, a driving assistance system, and a driving assistance program according to the present invention are characterized by the following (1) to (10). (1) An image acquisition unit that is attached to a cargo handling vehicle that transports a platform on which luggage is placed, and that acquires an image captured by an imaging device that captures an image of an area including a part of the cargo handling vehicle, the luggage, and the platform; a baggage coordinate detection unit that detects an outline coordinate of the baggage in the image; a determination unit that determines whether or not the luggage is positioned offset with respect to a reference axis that divides the image in half in a width direction, according to a position of an outline coordinate of the luggage with respect to the reference axis. Cargo handling status determination device. (2) A reference axis setting unit is further provided for setting a straight line extending in a vertical direction in the image, which divides the image in half in a width direction, as the reference axis. The loading state determination device according to (1) above. (3) A reference axis setting unit is further provided for setting the reference axis based on an outline coordinate of a part of the loading vehicle and an outline coordinate of the loading platform in the image. The loading state determination device according to (1) above. (4) A part of the loading vehicle is a claw protruding from the vehicle body, the reference axis setting unit sets, in the image, a straight line passing through a center point of each intersection between a side extending in a width direction of the placement table and the claws as the reference axis; The loading state determination device according to (3) above. (5) The image acquisition unit acquires a first image and a second image captured after capturing the first image, the reference axis setting unit recognizes two first intersections that are the respective intersections in the first image, recognizes two second intersections that are the respective intersections in the second image, synthesizes each movement vector from the first intersections to the second intersections, calculates the synthesized vector as a movement vector of the center point, and sets a straight line including the movement vector of the center point as the reference axis. The loading state determination device according to (4) above. (6) The image acquisition unit acquires a first image and a second image captured after capturing the first image, The determination unit is Recognizing first feature points, which are two feature points spaced apart in a width direction of the luggage in the first image, recognizing second feature points, which are two feature points spaced apart in a width direction of the luggage in the second image, and calculating luggage movement vectors, which are movement vectors from the first feature points to the second feature points; determining whether the luggage is positioned offset with respect to the reference axis according to at least one of a distance and an angle of the luggage movement vector with respect to the reference axis; A cargo handling state determination device according to any one of (1) to (5) above. (7) The image acquisition unit captures an image of the area at any time during the process of inserting the nail into the placement table. A loading status determination device according to (5) or (6) above. (8) A limiting unit is provided that limits at least one of a moving speed and a turning speed of the cargo handling vehicle when the determining unit determines that the baggage is positioned offset with respect to the reference axis. The cargo handling state determination device according to any one of (1) to (7) above. (9) A communication terminal having the cargo handling state determination device according to any one of (1) to (8) above, An in-vehicle device having the imaging device, Driving assistance system. (10) A driving assistance program for assisting the driving of a loading vehicle that transports a loading platform on which a load is placed, comprising: acquiring an image from an imaging device attached to the cargo handling vehicle, the imaging device imaging an area including a part of the cargo handling vehicle, the baggage, and the platform; detecting an outline coordinate of the baggage in the image; determining whether the luggage is positioned offset with respect to a reference axis according to a position of an outer coordinate of the luggage with respect to the reference axis set in the image; A driving assistance program that executes the following:
[0011] According to the loading status determination device having the configuration of (1), the driving assistance system having the configuration of (9), and the driving assistance program having the configuration of (10), it is determined that the luggage is placed biased with respect to the reference axis, depending on the position of the outer periphery coordinates of the luggage with respect to the reference axis that divides the image in half in the width direction. For example, when the positions of the outer periphery coordinates of a plurality of luggage are separated in one direction from the reference axis, it is possible to detect that the luggage is placed biased in the width direction on the loading platform. Therefore, when the loading posture is unstable, the vehicle speed and turning speed are limited, and it is possible to prevent the luggage from collapsing or the luggage from falling.
[0012] According to the loading status determination device having the configuration of (2) above, a straight line extending in the vertical direction that divides the image in half in the width direction is used as the reference axis, so that it is possible to detect whether the luggage is placed unevenly in the width direction on the loading platform depending on the position of the luggage's outline coordinates relative to this reference axis.
[0013] According to the loading status determination device having the configuration of (3) above, the reference axis is set based on the outline coordinates of a portion of the loading vehicle and the outline coordinates of the loading platform, so that the imbalance of the luggage can be determined with high accuracy according to the actual imaging conditions.
[0014] According to the cargo handling state determination device configured as in (4) above, by using a straight line passing through the center point of each intersection point between the claw and one side of the platform as a reference axis, it is possible to accurately determine the imbalance of the cargo in the width direction of the platform based on the center position of the claw that holds the platform. Therefore, even if the camera is not attached to the center of the vehicle in the lateral direction, it is possible to accurately determine whether or not the cargo is imbalanced in the width direction of the platform.
[0015] According to the cargo handling state determination device having the above configuration (5), by setting a straight line including the movement vector of the center point as the reference axis, it is possible to accurately determine whether or not the cargo is biased in the width direction.
[0016] According to the cargo handling status determination device having the configuration of (6) above, for example, if the cargo movement vectors on the left and right sides of the reference axis are asymmetric, it can accurately determine whether or not there is cargo imbalance in the width direction.
[0017] According to the cargo handling state determination device configured as in (7) above, the determination unit calculates, for example, a movement vector based on an image captured at any point during the process of inserting the claw into the platform, and determines whether the left and right sides are even with respect to the reference axis. Therefore, the stability of the cargo state before the lift operation can be accurately determined.
[0018] According to the loading status determination device having the configuration of (8) above, when it is detected that the luggage is unevenly placed, the moving speed and turning speed of the loading vehicle are limited, thereby reducing the probability of the luggage falling during transportation, which helps prevent falling accidents. Effect of the Invention
[0019] According to the present invention, it is possible to prevent accidents in which luggage falls from a loading vehicle due to unstable luggage transportation.
[0020] The present invention has been briefly described above. Furthermore, the details of the present invention will be further clarified by reading the following description of the embodiment of the present invention (hereinafter, referred to as "embodiment") with reference to the accompanying drawings. [Brief description of the drawings]
[0021] [Figure 1] FIG. 1 is a block diagram showing a hardware configuration of a driving assistance system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a side view showing an example of the appearance of a forklift truck and a load during operation. [Diagram 3] FIG. 3 is a side view showing an example of the relationship between the luggage photographing camera and the forklift. [Figure 4] FIG. 4 is a diagram showing an example of an image captured by the baggage photographing camera. [Diagram 5] FIG. 5 is a flowchart showing an example of a cargo handling state determination operation in the vehicle-mounted device shown in FIG. [Figure 6] FIG. 6 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 7] FIG. 7 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 8] FIG. 8 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 9] FIG. 9 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 10] FIG. 10 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 11] FIG. 11 is a diagram showing an example of an image captured by the baggage photographing camera. [Figure 12] FIG. 12 is a diagram showing an example of an image captured by another baggage photographing camera. [Figure 13] FIG. 13 is a diagram showing an example of an image captured by another baggage photographing camera. [Figure 14] FIG. 14 is a block diagram showing an example of a functional configuration for realizing a driving assistance system according to an embodiment of the present invention. [Figure 15] FIG. 15 is a diagram for explaining another example of how to obtain the reference axis. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0022] Specific embodiments of the present invention will be described below with reference to the accompanying drawings.
[0023] <System configuration> The hardware configuration of a driving assistance system according to an embodiment of the present invention is shown in Fig. 1. The driving assistance system shown in Fig. 1 is mainly used for driving assistance when operating a cargo handling vehicle such as a forklift to transport cargo.
[0024] The driving assistance system shown in Fig. 1 includes an on-board device 10 that is used while being mounted on each cargo handling vehicle, and an office PC 30 that is installed in a specified office to manage each cargo handling vehicle, workers, work contents, etc. Although details will be described later, the on-board device 10 of this embodiment has, as its main function, a driving assistance function that is useful for preventing baggage from falling during baggage transport work.
[0025] The vehicle-mounted device 10 and the office PC 30 do not have to be connected via a network, in which case the vehicle-mounted device 10 is configured to read a memory card 65 that holds data recorded by the vehicle-mounted device. The vehicle-mounted device 10 may also include the functions of a general vehicle-mounted device, such as a drive recorder.
[0026] The office PC 30 is composed of a general-purpose computer device installed in an office, and manages the operation status of the loading and unloading vehicle, the amount of cargo transported, etc. In the example of Fig. 1, an access point 80 is installed to relay data communication between the vehicle-mounted device 10 and the office PC 30. Wireless communication between the access point 80 and the vehicle-mounted device 10 may be performed via a mobile communication network (mobile line network) such as LTE (Long Term Evolution) / 5G (5th Generation), or via a wireless LAN (Local Area Network).
[0027] The vehicle-mounted device 10 includes various interfaces (I / F) 12A, 12B, 13, 14, 16, 19, and 29 to enable input or output of various signals. The speed I / F 12A has a function for inputting a vehicle speed pulse signal outputted from a vehicle speed sensor 51 mounted on the vehicle side. The engine speed I / F 12B has a function for inputting an engine speed pulse signal outputted from the vehicle side. The external input I / F 13 is used for inputting various external signals.
[0028] The sensor input I / F 14 is used to input signals from various sensors. In the example of Fig. 1, a G sensor 28 and a gyro sensor 52 are connected to the sensor input I / F 14. The G sensor 28 detects the magnitude of acceleration in various directions applied to the vehicle on which the vehicle-mounted device 10 is mounted. The gyro sensor 52 detects the angular velocity of the vehicle on which the vehicle-mounted device 10 is mounted, and can output a signal indicating a change in the turning speed and direction of the vehicle in a horizontal plane.
[0029] The analog input I / F 29 is used to input various analog signals. The camera I / F 16 has a function for connecting the vehicle-mounted camera 23. That is, the camera I / F 16 has a function for taking in a video signal output by the vehicle-mounted camera 23 and converting it into predetermined digital image data suitable for computer processing.
[0030] The audio I / F 19 has a function of generating a predetermined audio signal that can be used for audio alerts, etc.
[0031] The control unit 11, which realizes the main functions of the vehicle-mounted device 10, is composed of electronic circuits mainly including a processor of a microcomputer (CPU). This microcomputer realizes the control functions of the vehicle-mounted device 10, which will be described later, by executing programs previously stored in the non-volatile memory 26A or the like.
[0032] The above-mentioned interfaces 12A, 12B, 13, 14, 16, and 29 are connected to the input of the control unit 11. In addition, a speaker 20 is connected to the output of the control unit 11 via an audio I / F 19.
[0033] In addition, the GPS receiving unit 15, the recording unit 17, the display unit 27, the power supply unit 25, the communication unit 24, the non-volatile memory 26A, the volatile memory 26B, the card I / F 18, the RTC unit 21, the switch input unit 22, and the signal output unit 53 are connected to the control unit 11.
[0034] The GPS receiver 15 receives radio waves from multiple GPS (Global Positioning System) satellites via an antenna 15a. Based on the multiple reception signals received by the GPS receiver 15, information indicating the current position can be calculated and obtained. Note that a beacon can be used instead of the GPS to grasp the information on the current position.
[0035] The recording unit 17 is used to automatically record image data of images output by the vehicle-mounted camera 23, for example, and hold the data for a certain period of time. The display unit 27 can be used to display visible information such as characters necessary for operating the vehicle-mounted device 10, and information calling attention to driving operations so that the driver can visually recognize the information.
[0036] The power supply unit 25 generates stable power supply based on power supply supplied from the vehicle side, and supplies the generated power supply to each circuit in the vehicle-mounted device 10 including the control unit 11 . The communication unit 24 provides a wireless communication function for data communication between the vehicle-mounted device 10 and the access point 80 .
[0037] The non-volatile memory 26A is made up of a semiconductor memory, and previously stores programs executable by the microcomputer of the control unit 11, various constant data required for control, tables, and the like.
[0038] The volatile memory 26B is used to temporarily store data generated by the control unit 11 during processing. A memory card 65 carried by the driver is removably connected to the card I / F 18. The control unit 11 can read data from the memory card 65 attached to the card I / F 18, and can also write various data generated by the control unit 11 to the memory card 65 via the card I / F 18.
[0039] The RTC (real time clock) unit 21 is configured with an integrated circuit having a clock function. That is, the RTC unit 21 can generate information on the current time and grasp the elapsed time.
[0040] The switch input unit 22 is used to input signals representing the states of various switches necessary for operating the vehicle-mounted device 10 . The signal output unit 53 can output driving limit signals SGA and SGB that can be used to limit the driving of a cargo handling vehicle from the vehicle-mounted device 10. The driving limit signal SGA is expected to be used, for example, to limit the traveling speed of the cargo handling vehicle. The driving limit signal SGB is expected to be used, for example, to limit the turning speed of the cargo handling vehicle.
[0041] On the other hand, the office PC 30 is a PC that runs on a general-purpose operating system. The office PC 30 can be used as an operation management device for grasping the operation status of a loading vehicle and the transport amount of luggage, etc. The office PC 30 has a control unit (CPU) 31, a communication unit 32, a display unit 33, a memory unit 34, a card I / F 35, an operation unit 36, an output unit 37, an audio I / F 38, and an external I / F 48.
[0042] The control unit 31 comprehensively controls each unit of the office PC 30. The communication unit 32 is capable of communicating with the vehicle-mounted device 10 via an access point 80. The display unit 33 can display various information that can be used for managing the operation of each cargo handling vehicle. The storage unit 34 can acquire and manage data generated by the on-board device 10 mounted on each cargo handling vehicle.
[0043] A memory card 65 is attached to the card I / F 35 so as to be freely insertable and removable. The card I / F 35 is used to input various data recorded in the vehicle-mounted device 10 from the memory card 65. The operation unit 36 has a keyboard, mouse, etc., and accepts operations by the administrator of the office PC 30. The output unit 37 outputs various data. A microphone 41 and a speaker 42 are connected to the audio I / F 38. The administrator can also make audio calls using the microphone 41 and the speaker 42.
[0044] External storage devices (not shown), such as an operation data database (DB) and a hazard map database (DB), can be connected to the external I / F 48. The hazard map DB can hold data indicating specific locations where cargo handling vehicles such as forklifts are prone to accidents.
[0045] <Examples of cargo handling vehicles> An example of the appearance of a forklift in operation and a load is shown in Figure 2. In Figure 2, the left side represents the forward direction of the forklift, and the right side represents the backward direction.
[0046] As shown in Fig. 2, the forklift 90 has a plurality of (two in this embodiment) claws 91 (parts of a cargo handling vehicle) that protrude long from the vehicle body forward of the driver's seat, and a backrest 92. The claws 91 and the backrest 92 are supported by the mast 93 in a state in which they can be raised and lowered in the vertical direction along the mast 93, and are attached to the vehicle body. In addition, the forklift 90 can drive the positions of the claws 91 and the backrest 92 in the vertical direction by driving a predetermined lifting mechanism. Furthermore, the forklift 90 is provided with a drive mechanism that changes the inclination angle of the mast 93 that supports the claws 91 and the backrest 92, and the tilt angles of the claws 91 and the backrest 92 can be adjusted.
[0047] Meanwhile, various cargo 100 to be transported by the forklift 90 are generally stored in a state in which they are placed on a pallet 110, which is a platform arranged on the ground 98. Therefore, when actually transporting the cargo 100, the forklift 90 advances toward the pallet 110 with the claws 91 lowered as shown in FIG. 2, so that the claws 91 pass through the inside of the pallet 110. When the forklift 90 lifts the claws 91 upward in this state, the cargo 100 placed on the pallet 110 can be lifted together with the pallet 110. Then, by moving the forklift 90 with the cargo 100 and the pallet 110 lifted, the cargo 100 and the pallet 110 can be transported.
[0048] Incidentally, in work sites where forklifts are used to transport loads, such as within a factory or a warehouse, accidents frequently occur in which loads being transported by the forklift are dropped and damaged.
[0049] In actual work of this kind, the cargo 100 placed on the pallet 110 as shown in FIG. 2 is lifted from below together with the pallet 110 by the fork 91 of the forklift 90, and the cargo 100 is transported in this state.
[0050] The frequency of loads falling during transport by forklift largely depends on the stability of the loads placed on the pallets. Therefore, in order to prevent loads from falling during transport, it is generally desirable to visually check the stability of the load before transporting it.
[0051] Ideally, loads to be transported by forklift should be stacked so that the center of gravity is located in the center of the pallet, but in busy work sites, it is difficult to stack loads carefully and pay sufficient attention to time-consuming visual inspections.
[0052] On the other hand, if a worker notices that a package is unstable before or during transportation by a forklift, it is desirable to correct the package state each time. However, in busy work sites, workers may not have the time to carry out the time-consuming task of correcting the package state, so they may think, "It will be fine without correction," and continue working without correcting the package state.
[0053] 1, the office PC 30 that manages the operation of the vehicle-mounted device 10 and the forklift 90 has a driving support function that is useful for preventing the load 100 from falling during transportation by the forklift 90. In this embodiment, an on-vehicle camera 23 attached to the body of the forklift 90 is used to obtain information required for the driving support.
[0054] <Installation position and shooting range of the in-vehicle camera> FIG. 3 shows an example of the relationship between the vehicle-mounted camera 23 for photographing luggage and a forklift 90.
[0055] In the example shown in Fig. 3, the vehicle-mounted camera 23 is installed on the upper part of the mast 93, and the shooting direction is adjusted so that it faces a direction inclined downward at about 45 degrees from the horizontal direction. In other words, since the area of the shooting range 23A shown in Fig. 3 becomes the shooting target, the claws 91 of the forklift 90, the pallet 110 being transported, and the baggage 100 can be shot by the vehicle-mounted camera 23. It is desirable that the shooting range 23A of the vehicle-mounted camera 23 be a wide range of, for example, about 120 degrees so that substantially the entire area of the baggage 100 and the pallet 110 can be shot simultaneously.
[0056] <Example of captured image> An example of an image captured by the on-board camera 23 installed on the forklift 90 to capture an image of the luggage 100 is shown in FIG.
[0057] In the example of Fig. 4, a ground area 201 corresponding to the ground surface 98 and a claw pattern 202 corresponding to the claws 91 of a forklift 90 are included in a camera image 200 captured by the vehicle-mounted camera 23. The example of Fig. 4 illustrates a situation in which the baggage 100 and the pallet 110 are not present within the capturing range 23A of the vehicle-mounted camera 23. In the example of FIG. 4, a FOE (Focus Of Expansion) point (vanishing point) 203 exists at a position close to the top center of the screen of the camera image 200.
[0058] <Load handling status determination operation> An example of the cargo handling state determination operation in the vehicle-mounted device 10 shown in Fig. 1 is shown in Fig. 5. Also, examples of images captured by the vehicle-mounted camera 23 that captures images of luggage are shown in Figs.
[0059] The control unit 11 of the vehicle-mounted device 10 executes a predetermined program on the non-volatile memory 26A to perform the load-handling state determination operation shown in Fig. 5. When the forklift 90 starts to operate, the control unit 11 determines whether the claw 91 is included in the shooting range 23A of the vehicle-mounted camera 23, that is, whether the claw 91 is captured in the video (image) captured by the vehicle-mounted camera 23 (step S11). If the claw 91 is captured, a camera image 200A as shown in Fig. 6 is obtained, and the control unit 11 proceeds from step S11 to step S12.
[0060] 6, in addition to the ground area 201, a claw pattern 202 appears. Therefore, for example, the control unit 11 processes and recognizes the contents of the camera image 200A, so that the position of the claw 91 of the forklift 90 can be grasped (step S12).
[0061] When the forklift 90 approaches the pallet 110 on which the luggage 100 is loaded, a camera image 200B such as that shown in Fig. 7 is obtained. Therefore, for example, the control unit 11 performs image processing and recognizes the contents of the camera image 200B, whereby the control unit 11 can detect that the forklift 90 has approached the luggage 100 loaded on the pallet 110 (YES in step S13), and the process proceeds to step S14.
[0062] In addition to the ground area 201 and the claw pattern 202, a pallet pattern 210 corresponding to the pallet 110 and a baggage pattern 220 corresponding to the baggage 100 appear in the camera image 200B shown in FIG.
[0063] Therefore, the control unit 11 can recognize the pallet pattern 210 and the luggage pattern 220 from the camera image 200B. In step S14, the control unit 11 starts detecting the coordinates of the lower side of the front side of the pallet 110 and the feature point coordinates of the luggage 100. That is, the control unit 11 detects and grasps the lower side B from the feature point coordinates of the corners of the lowermost side (lower side in FIG. 7) of the contour of the pallet pattern 210. In addition, the control unit 11 detects the coordinates of the outermost corners (two points with the longest distance in the left-right direction in FIG. 7) of the contour of the luggage pattern 220 to grasp the feature points of the luggage 100. The camera image 200B shown in FIG. 7 is updated successively, and the position and size of each pattern in the camera image 200B change with the movement of the forklift 90, and each feature point also changes, so the detection of the lower side B and the detection of the feature point coordinates of the luggage 100 are repeatedly executed by the control unit 11.
[0064] The control unit 11 detects whether the claw 91 has been inserted into the pallet 110 based on the bottom side B of the pallet pattern 210 and the feature point coordinates of the luggage pattern 220 detected in step S14 (step S15). That is, the control unit 11 detects that the claw 91 has been inserted into the pallet 110 when the claw pattern 202 matches the coordinates of the bottom side B of the pallet pattern 210.
[0065] In the next step S16, the control unit 11 starts measuring the movement vectors Vc1, Vc2 of two feature points spaced apart in the width direction of the package 100 and the movement vector Vp of the pallet 110. The measurement of the movement vectors Vc1, Vc2, and Vp is repeatedly performed by the control unit 11.
[0066] For example, when the forklift 90 moves forward from the state in which the camera image 200C shown in Fig. 8 is obtained, the forklift 90 approaches the baggage 100 and the pallet 110, and the coordinates of each feature point in the camera image 200C move over time. Specifically, each feature point moves in a radial direction from the position of the FOE point 203 toward its periphery. The amount of movement of each feature point in the camera image 200C changes depending on the distance from the vehicle-mounted camera 23 and the traveling speed of the forklift 90. A vector indicating the direction and amount of movement of each feature point is a movement vector.
[0067] An example of calculation of the movement vector of the pallet 110 will be described. The intersections A1 and A2 between the inner edges (the edges of the left and right claw patterns 202 that are close to each other) of the left and right claw patterns 202 and the bottom side B of the pallet pattern 210 at the time when the control unit 11 detects that the claw 91 has been inserted into the pallet 110 are used as references. The control unit 11 calculates vectors Va1 and Va2 from the intersections A1 and A2, which are references, to the intersections between the inner edges of the left and right claw patterns 202 and the bottom side B at the time when the part of the claw pattern 202 corresponding to the tip of the claw 91 is covered by the pallet pattern 210 and cannot be seen. The control unit 11 regards the midpoint of the intersections A1 and A2 as the center point P of the pallet pattern 210 corresponding to the left-right center of the pallet 110, and calculates a vector Vp obtained by combining the vectors Va1 and Va2 as the movement vector of the center point P, i.e., the movement vector of the pallet 110. The tip of the claw 91 is inserted into the inside of the pallet 110 when the forklift 90 performs a lifting operation.
[0068] 8, the control unit 11 can calculate the movement vector Vp of the center point P in the pallet pattern 210, and the movement vectors Vc1, Vc2 of each feature point in the baggage pattern 220. Each of the movement vectors Vc1, Vc2, and Vp changes as the camera image 200C is updated.
[0069] In practice, the movement vectors Vc1, Vc2, and Vp are obtained by calculating the amount and direction of movement between the coordinates of the feature points of the luggage pattern 220 and the center point P of the pallet pattern 210 (hereinafter also referred to as "feature points, etc.") in a first image taken at a certain time t1 and the coordinates in a second image taken at a time t2 after a certain period of time has passed.
[0070] The control unit 11 calculates and stores each of the movement vectors Vc1, Vc2, and Vp that change successively until the forklift 90 starts lifting the pallet 110 on which the package 100 is loaded (step S16).
[0071] Furthermore, based on the calculation result in step S16, the control unit 11 judges whether the movement vectors Vc1 and Vc2 of the luggage 100 are equal on the left and right sides with respect to the movement vector Vp of the pallet 110 (step S17). That is, the control unit 11 sets a straight line including the movement vector Vp of the pallet 110, that is, a straight line connecting the center point P and the FOE point 203 (a straight line shown by a dashed line in FIG. 9) as a reference axis J. Then, the control unit 11 judges whether there is an abnormality such as the luggage 100 being unevenly loaded according to the positions of the characteristic points of the luggage pattern 220 and the distances and angles of the movement vectors Vc1 and Vc2 of the luggage 100 relative to this reference axis J. Note that the reference axis is not limited to the straight line connecting the center point P and the FOE point 203, but may be another straight line passing through the center point P. For example, a perpendicular line drawn from the center point P to the side of the pallet pattern 210, or a straight line obtained by calculating the movement vector of the center point P from consecutive images and extending the movement vector, etc. may be used. Although the accuracy is reduced, a straight line extending in the vertical direction that divides the image into two equal parts to the left and right may be used as the reference axis. Note that the camera image 200D in Fig. 9 corresponds to the camera image 200C in Fig. 8 in which the baggage pattern 220 and the pallet pattern 210 are omitted.
[0072] When the control unit 11 determines in step S17 that the movement vectors Vc1, Vc2 of the luggage 100 are even on the left and right, it determines that the luggage 100 is loaded in a balanced manner on the pallet 110 (stable luggage) (step S18). The camera image 200E in FIG. 10 shows an example in which it has been determined that the luggage is stable. In the camera image 200E, the movement vectors Vc1, Vc2 of the luggage 100 are arranged symmetrically with respect to the movement vector Vp. Note that the control unit 11 may determine that the movement vectors Vc1, Vc2 of the luggage 100 are even on the left and right when the distances and angles of the movement vectors Vc1, Vc2 of the luggage 100 relative to the reference axis J are not significantly different from each other.
[0073] On the other hand, when the control unit 11 determines in step S17 that the movement vectors Vc1, Vc2 of the luggage 100 are not uniform on the left and right, it detects an abnormality in which the luggage 100 is unevenly loaded on the pallet 110 (step S19). The camera image 200F in FIG. 11 shows an example in which the luggage 100 is determined to be unevenly loaded. In the camera image 200F, the movement vectors Vc1, Vc2 of the luggage 100 are both located on one side of a divided area divided into two parts on the left and right by the movement vector Vp of the pallet 110, and neither of the movement vectors Vc1, Vc2 is located on the other side. In other words, the movement vectors Vc1, Vc2 of the luggage 100 are located away in one direction from the reference axis J including the movement vector Vp of the pallet 110.
[0074] Camera image 200F shown in Fig. 11 shows a situation in which luggage pattern 220 representing luggage 100 is unevenly placed on pallet pattern 210. A situation such as camera image 200F in Fig. 11 means that the state of luggage 100 on pallet 110 is very unstable. In other words, luggage 100 is likely to fall off pallet 110 during transportation. Therefore, in the situation in Fig. 11, it is necessary to alert the driver and support safe driving in order to prevent luggage 100 from falling.
[0075] Therefore, when the control unit 11 detects an abnormality in the load imbalance in step S19, it restricts the travel speed and / or the turning speed of the forklift 90 (step S20). Specifically, when the travel speed or turning speed is high, the control unit 11 restricts the driving operation of the forklift 90 by outputting a message such as a voice, calling the driver's attention using a screen display, or controlling the driving restriction signal SGA or SGB. When the driving restriction signal SGA or SGB is used, for example, the travel speed of the forklift 90 is restricted so as not to increase, or the brakes are automatically applied. Also, the turning speed of the forklift 90 is restricted so as not to increase. This restriction can prevent accidents in which the load falls.
[0076] In this way, the control unit 11 sets the reference axis J (a straight line including the movement vector Vp of the pallet 110) based on the claws 91 of the forklift 90 and the outer coordinates of the pallet 110. The control unit 11 then detects an abnormality in the uneven distribution of the cargo based on at least one of the positions of the outer coordinates (characteristic points) of the cargo 100 relative to the reference axis J and the distances and angles of the movement vectors Vc1 and Vc2. For example, when all of the outer coordinates of the cargo 100 are located at positions spaced apart in one direction relative to the reference axis J, it is determined that the cargo 100 is unevenly distributed in the width direction of the pallet 110. The vehicle-mounted device 10 can detect that the cargo 100 is unevenly distributed on the pallet 110 in the width direction, and therefore can limit the vehicle speed and turning speed in an unstable state to prevent cargo from falling over.
[0077] Furthermore, the vehicle-mounted device 10 can accurately determine whether or not the load is offset from the center in the width direction of the forklift 90 by setting a straight line connecting the FOE point 203 and the center point P of each of the intersection points A1, A2 between the left and right claw patterns 202 and one side (lower side B) of the pallet pattern 210 as a reference axis J. Therefore, even if the vehicle-mounted camera 23 is not attached to the center in the width direction of the forklift 90, it can accurately determine whether or not the load is offset in the width direction of the forklift 90.
[0078] <Other examples of captured images> 12 and 13 respectively show camera images 200G and 200H when the on-board camera 23 is attached at a position shifted from the center in the width direction of the forklift 90. Fig. 12 and Fig. 13 show example images when the angle of view of the on-board camera 23 is about 180°. The on-board camera 23 captures images of the claws 91, luggage 100, pallet 110, and the upper part of the backrest 92, and the camera images 200G and 200H include the claw pattern 202, luggage pattern 220, pallet pattern 210, and backrest pattern 204.
[0079] In the camera images 200G and 200H, the control unit 11 calculates vectors Va1 and Va2 from the intersections A1 and A2 between the bottom side of the pallet pattern 210 and the left and right claw patterns 202, as in the case shown in FIG. 8, and calculates the movement vector Vp of the center point P. The control unit 11 also calculates the movement vectors Vc1 and Vc2 of the luggage 100, and determines whether the movement vectors Vc1 and Vc2 of the luggage 100 are equal on the left and right sides based on the movement vector Vp. In the camera image 200G, the movement vectors Vc1 and Vc2 of the luggage 100 are arranged symmetrically based on the movement vector Vp, and the luggage is determined to be stable. On the other hand, in the camera image 200H, the distances and angles of the movement vectors Vc1 and Vc2 of the luggage 100 relative to the straight line (reference axis J) including the movement vector Vp are different (not symmetrical). That is, since an abnormality in the uneven distribution of luggage is detected in the camera image 200H, the vehicle-mounted device 10 restricts the traveling speed and the turning speed.
[0080] In this way, even if the vehicle-mounted camera 23 is not attached to the center of the width of the forklift 90, the vehicle-mounted device 10 can detect an abnormality in which the luggage 100 on the pallet 110 is unevenly loaded, and prevent the luggage from falling over by limiting the travel speed, etc. Even if the luggage 100 (pallet 110) is not placed on the ground, the vehicle-mounted device 10 can detect an abnormality in luggage unevenness by determining whether the movement vectors Vc1, Vc2 of the luggage 100 are equal on the left and right sides based on the above-mentioned movement vector Vp.
[0081] The vehicle-mounted device 10 also calculates the movement vectors Vp, Vc1, and Vc2 of the center point P and the characteristic points of the luggage 100 in an image captured when the insertion of the claw 91 into the pallet 110 begins and an image captured when the tip of the claw 91 is inserted into the pallet 110. Furthermore, the vehicle-mounted device 10 judges whether the movement vectors Vc1 and Vc2 are equal on the left and right with respect to this movement vector Vp. Therefore, the vehicle-mounted device 10 can accurately judge the stability of the package immediately before the lift operation is performed.
[0082] According to the prior invention by the present inventors, it was possible to determine whether or not a load is biased toward the back of a load platform based on an image of the load placed on the load platform captured by an imaging device mounted on a loading vehicle (forklift) (see Patent Application No. 2020-073046). In contrast, according to the present embodiment, as described above, it is possible to determine whether or not the load is biased toward the left or right of the load platform. Therefore, by combining the technology of the prior invention with the technology of the present embodiment, the vehicle-mounted device 10 can determine whether or not there is an abnormality in that the load is biased toward the back or left or right of the load platform.
[0083] <System functional configuration example> Fig. 14 shows an example of a functional configuration required to realize each feature of the driving assistance system shown in Fig. 1. Each function shown in Fig. 14 can be realized by the control unit 11 or the computer of the office PC 30 executing a program prepared in advance, or can be realized by using dedicated electronic circuit hardware.
[0084] 14, the vehicle-mounted device 10 includes the functions of an image processing unit 302, an image storage unit 303, a claw image recognition unit 304a, an approach detection unit 304b, a claw insertion detection unit 304c, a pallet image recognition unit 305, a luggage image recognition unit 306, a pallet side detection unit 307, and a luggage outer coordinate detection unit 308. The vehicle-mounted device 10 also includes the functions of a pallet movement vector calculation unit 311, a luggage movement vector calculation unit 312, a luggage imbalance abnormality determination unit 314, an alarm notification unit 316, a traveling speed monitoring unit 317, a turning speed monitoring unit 318, and a vehicle operation restriction unit 319. The vehicle-mounted device 10 also includes the functions of a luggage volume calculation unit 321, a luggage volume recording unit 322, a time information management unit 323, a data storage unit 324, and an alarm recording unit 325.
[0085] Some of the functions shown in FIG. 14 may be located on the office PC 30 side. In that case, the entire system can be made to function properly by data communication between the vehicle-mounted device 10 and the office PC 30, or by data transfer using the memory card 65.
[0086] The image processing unit 302 inputs the vehicle-mounted camera image 301 output from the vehicle-mounted camera 23, and generates image data suitable for data processing by the control unit 11. The image storage unit 303 can temporarily store the image data output from the image processing unit 302.
[0087] The nail image recognition unit 304a retrieves image data of the latest image frame from the output of the image processing unit 302, and simultaneously reads and retrieves image data of a previous image frame captured a predetermined time ago (e.g., one second ago) from the image storage unit 303. The nail image recognition unit 304a then performs processing for recognizing a pattern corresponding to the nail 91 for each of the two image frames captured at different times. The approach detection unit 304b detects that the forklift 90 has approached the baggage 100 based on the image data in the latest image frame. The nail insertion detection unit 304c detects that the nail 91 has been inserted into the pallet 110 when it detects an intersection between the nail 91 and the pallet 110 based on image data in the latest image frame and image data in past image frames.
[0088] The palette image recognition unit 305 takes in image data for the latest image frame from the output of the image processing unit 302, and simultaneously reads and takes in image data for a previous image frame captured a predetermined time ago (e.g., one second ago) from the image storage unit 303. The palette image recognition unit 305 then performs processing to recognize a pattern corresponding to the palette 110 for each of the two image frames captured at different times.
[0089] The luggage image recognition unit 306 retrieves image data of the latest image frame from the output of the image processing unit 302, and simultaneously reads and retrieves image data of a previous image frame captured a predetermined time ago (e.g., one second ago) from the image storage unit 303. Then, the luggage image recognition unit 306 performs processing to recognize a pattern corresponding to the luggage 100 for each of the two image frames captured at different times.
[0090] The palette side detection unit 307 can detect, for each image frame, a plurality of coordinates of each corner that represents the bottom side of the outer contour of the pattern of the palette 110 recognized by the palette image recognition unit 305 .
[0091] The baggage outline coordinate detection unit 308 can detect, for each image frame, a plurality of coordinates of each corner that represents the outline of the pattern of the baggage 100 recognized by the baggage image recognition unit 306 .
[0092] The pallet movement vector calculation unit 311 calculates a movement vector at the midpoint of the intersection between the claws 91 detected by the claw insertion detection unit 304c and the pallet 110. In other words, the pallet movement vector calculation unit 311 calculates a movement vector as the amount and direction of movement between the coordinates of the intersections A1, A2 in an image frame and the coordinates of the intersections A1, A2 in an image frame a certain time earlier. The pallet movement vector calculation unit 311 then combines these to calculate the movement vector Vp of the center point P.
[0093] The baggage movement vector calculation unit 312 calculates a movement vector for each feature point of the baggage 100 detected by the baggage outline coordinate detection unit 308. That is, the baggage movement vector calculation unit 312 calculates movement vectors Vc1 and Vc2 as the amount and direction of movement between the coordinates of a feature point in a certain image frame and the coordinates of the feature point in an image frame a certain time earlier.
[0094] The luggage imbalance abnormality determination unit 314 has a function for determining whether or not there is an abnormality regarding the imbalance of luggage positions. That is, the luggage imbalance abnormality determination unit 314 inputs the movement vector Vp of the pallet 110 from the pallet movement vector calculation unit 311. The luggage imbalance abnormality determination unit 314 inputs the movement vector Vp of the pallet 110 from the pallet movement vector calculation unit 311, and inputs the movement vectors Vc1, Vc2 of the luggage 100 from the luggage movement vector calculation unit 312. Then, the luggage imbalance abnormality determination unit 314 determines whether or not there is an abnormality by comparing whether or not the movement vectors Vc1, Vc2 of the luggage 100 are equal on the left and right sides based on the movement vector Vp of the pallet 110 as in step S17 of FIG. 5.
[0095] When the luggage imbalance abnormality determination unit 314 detects an abnormality, the alarm notification unit 316 generates an alarm signal SG0 indicating the occurrence of an abnormality.
[0096] In response to this warning signal SG0, for example, the audio I / F 19 generates an audio signal and outputs an audio announcement for alerting the driver from the speaker 20. Also, a message alerting the driver is output on the screen of the display unit 27.
[0097] The traveling speed monitoring unit 317 monitors the signal output from the vehicle speed sensor 51 to grasp the traveling speed of the forklift 90 and monitor whether the speed is too high. The turning speed monitoring unit 318 monitors the angular velocity based on the signal output from the gyro sensor 52, and monitors whether the turning speed of the forklift 90 is excessive.
[0098] When the load imbalance abnormality determination unit 314 detects an abnormality, the vehicle operation restriction unit 319 generates operation restriction signals SGA, SGB for restricting the running speed and turning speed of the forklift 90. The vehicle operation restriction unit 319 generates the operation restriction signals SGA, SGB using the outputs of the running speed monitoring unit 317 and the turning speed monitoring unit 318. These operation restriction signals SGA, SGB are input to the forklift 90 side.
[0099] Meanwhile, the luggage volume calculation unit 321 calculates and grasps the volume of luggage 100 actually transported by the forklift 90 equipped with the vehicle-mounted device 10 in each transport work. That is, the coordinates of each characteristic point of the pattern outline representing each luggage 100 to be transported are obtained as the output of the luggage outline coordinate detection unit 308, so the shape and size of each luggage can be estimated by calculation from the coordinates of each characteristic point. In addition, since the mounting position of the vehicle-mounted camera 23 is fixed and the shooting range 23A is also fixed, it is possible to grasp the position, distance, and dimensions between the coordinates of each characteristic point from each coordinate, and the volume of the luggage can be calculated.
[0100] The package volume recording unit 322 sequentially stores information on the volume of each package calculated by the package volume calculation unit 321 in the data storage unit 324 together with information on the current time output by the time information management unit 323, for example at regular time intervals.
[0101] The time information management unit 323 obtains information on the current time from the output of the RTC unit 21. The alarm recording unit 325 stores information about the alarm generated by the alarm notification unit 316 in the data storage unit 324 together with information about the current time output by the time information management unit 323 .
[0102] The volume information stored in the data storage unit 324 is associated with the time when the work was performed. Therefore, by analyzing the data stored in the data storage unit 324, it is possible to accurately grasp the total volume of the cargo 100 actually transported by the operation of the corresponding forklift 90. In addition, the number of times an alarm was generated and the time when it was generated can also be grasped.
[0103] The office PC 30 can obtain the data stored in the data memory unit 324 of the vehicle-mounted device 10 mounted on each forklift 90 via wireless data communication or the like and manage it in real time, making it easy to manage the overall progress of work, etc.
[0104] It should be noted that each processing step shown in FIG. 5 may be executed by either the control unit 11 on the vehicle-mounted device 10 or the control unit 31 of the office PC 30.
[0105] In the above embodiment, the midpoint of each intersection A1, A2 between each inner edge of the left and right nail patterns 202 and the bottom side B of the pallet pattern 210 is regarded as the center point P, and the straight line including the movement vector Vp of the center point P is regarded as the reference axis J, but the method of obtaining the reference axis is not limited to this. For example, as shown in FIG. 15, the midpoint of each intersection A1-1, A2-1 between each outer edge of the left and right nail patterns 202 and the bottom side B of the pallet pattern 210 may be regarded as the center point P. Then, the vector obtained by synthesizing vectors Va1-1, Va2-1 starting from each intersection A1-1, A2-1 may be regarded as the movement vector Vp of the center point P, and the straight line including this movement vector Vp may be regarded as the reference axis J. By using the outer edge of the nail pattern 202, even when there is only one nail 91, the reference axis J can be obtained from two or more consecutive images without depending on the number of nails 91.
[0106] Here, the features of the load handling state determination device, the driving assistance system, and the driving assistance program according to the above-described embodiments of the present invention will be briefly summarized and listed in the following [1] to
[10] . [1] An image acquisition unit (image processing unit 302, S11) is attached to a loading vehicle (forklift 90) that transports a platform (pallet 110) on which a load (100) is placed, and acquires an image captured by an imaging device (vehicle-mounted camera 23) that captures an image of an area including a part (claw 91) of the loading vehicle, the load, and the platform; A baggage coordinate detection unit (baggage outer coordinate detection unit 308, S14) for detecting the outer coordinates of the baggage in the image; and a determination unit (baggage imbalance abnormality determination unit 314, S17) that determines whether or not the baggage is placed biased with respect to a reference axis (J) that divides the image in half in the width direction, according to a position of the outer coordinates of the baggage with respect to the reference axis. Cargo handling status determination device (control unit 11 of vehicle-mounted device 10). [2] A reference axis setting unit is further provided for setting a straight line extending in a vertical direction in the image, which divides the image in half in a width direction, as the reference axis. The loading status determination device described in [1] above. [3] A reference axis setting unit is further provided for setting the reference axis based on the outline coordinates of a part of the loading vehicle and the outline coordinates of the loading platform in the image. The loading status determination device described in [1] above. [4] A part of the loading vehicle is a claw (91) protruding from the vehicle body, the reference axis setting unit sets, in the image, a straight line passing through a center point (P) of each intersection point (A1, A2) between one side extending in a width direction of the placement table and the claws as the reference axis; The loading status determination device described in [3] above. [5] The image acquisition unit acquires a first image and a second image captured after capturing the first image; the reference axis setting unit recognizes two first intersections (A1, A2) that are the respective intersections in the first image, recognizes two second intersections that are the respective intersections in the second image, synthesizes each movement vector (Va1, Va2) from the first intersections to the second intersections, calculates the synthesized vector as a movement vector (Vp) of the center point, and sets a straight line including the movement vector of the center point as the reference axis; The loading status determination device described in [4] above. [6] The image acquisition unit acquires a first image and a second image captured after capturing the first image; The determination unit is Recognizing first feature points, which are two feature points spaced apart in a width direction of the luggage in the first image, recognizing second feature points, which are two feature points spaced apart in a width direction of the luggage in the second image, and calculating luggage movement vectors (Vc1, Vc2), which are movement vectors from the first feature points to the second feature points; determining whether the luggage is positioned offset with respect to the reference axis according to at least one of a distance and an angle of the luggage movement vector with respect to the reference axis; A cargo handling status determination device according to any one of [1] to [5] above. [7] The image acquisition unit captures an image of the area at any time during the process of inserting the nail into the placement table. A cargo handling status determination device according to the above [5] or [6]. [8] A limiting unit (S20, vehicle operation limiting unit 319) that limits at least one of the moving speed and the turning speed of the loading vehicle when the determining unit determines that the luggage is positioned offset relative to the reference axis. The cargo handling state determination device according to any one of the above [1] to [4]. [9] A communication terminal (office PC 30) having the cargo handling status determination device according to any one of [1] to [8] above; and an in-vehicle device (10) having the imaging device. Driving assistance system.
[10] A driving assistance program for assisting the driving of a loading vehicle that transports a platform on which luggage is placed, comprising: A step of acquiring an image from an imaging device attached to the cargo handling vehicle, the imaging device imaging an area including a part (claw 91) of the cargo handling vehicle, the baggage, and the platform (image processing unit 302); A step (S14) of detecting an outline coordinate of the baggage in the image; a step (S17) of determining whether or not the luggage is positioned offset with respect to a reference axis set in the image according to a position of the contour coordinates of the luggage with respect to the reference axis; A driving assistance program that executes the following: [Explanation of symbols]
[0107] 10 Onboard equipment 11 Control section 12A, 12B, 13, 14, 16, 18, 19, 29 Interface (I / F) 15 GPS receiver 17 Recording Section 20 Speaker 21 RTC section 22 Switch input section 23 In-vehicle camera 23A Shooting range 24 Communications Department 25 Power supply section 26A non-volatile memory 26B Volatile memory 27 Display section 28 G Sensor 30 Office PC 31 Control Unit 32 Communications Department 33 Display section 34 Storage section 35,38,48 Interface 36 Operation section 37 Output section 41. Mike 42 Speaker 51 Vehicle speed sensor 52 Gyro sensor 53 Signal output section 65 Memory Card 80 Access Points 90 Forklift 91 Nails 92 Backrest 93 Mast 98 Ground 100 Baggage 110 Palettes 200,200A,200B,200C,200D,200E,200F,200G,200H Camera images 201 Ground area 202 Nail Pattern 203 FOE points 210 Palette Pattern B Bottom 220 Luggage Pattern J reference axis
Claims
1. an image acquisition unit that is attached to a cargo handling vehicle that transports a platform on which luggage is placed, and that acquires an image captured by an imaging device that captures an image of an area including a part of the cargo handling vehicle, the luggage, and the platform; a baggage coordinate detection unit that detects an outline coordinate of the baggage in the image; a determination unit that determines whether or not the package is positioned offset with respect to a reference axis that divides the image in half in a width direction, according to a position of an outline coordinate of the package with respect to the reference axis. Cargo handling status determination device.
2. a reference axis setting unit that sets a straight line that divides the image in half in a width direction and extends in a vertical direction in the image as the reference axis; The loading state determination device according to claim 1.
3. a reference axis setting unit that sets the reference axis based on an outline coordinate of a part of the loading vehicle and an outline coordinate of the loading platform in the image; The loading state determination device according to claim 1.
4. A part of the loading vehicle is a claw protruding from a vehicle body, the reference axis setting unit sets, in the image, a straight line passing through a center point of each intersection between a side extending in a width direction of the placement table and the claws as the reference axis; The loading state determination device according to claim 3.
5. The image acquisition unit acquires a first image and a second image captured after capturing the first image, the reference axis setting unit recognizes two first intersections that are the respective intersections in the first image, recognizes two second intersections that are the respective intersections in the second image, synthesizes each movement vector from the first intersections to the second intersections, calculates the synthesized vector as a movement vector of the center point, and sets a straight line including the movement vector of the center point as the reference axis. The loading state determination device according to claim 4.
6. The image acquisition unit acquires a first image and a second image captured after capturing the first image, and the determination unit Recognizing first feature points, which are two feature points spaced apart in a width direction of the luggage in the first image, recognizing second feature points, which are two feature points spaced apart in a width direction of the luggage in the second image, and calculating luggage movement vectors, which are movement vectors from the first feature points to the second feature points; determining whether the luggage is positioned offset with respect to the reference axis according to at least one of a distance and an angle of the luggage movement vector with respect to the reference axis; The loading state determination device according to any one of claims 1 to 5.
7. A part of the loading vehicle is a claw protruding from a vehicle body, the image acquisition unit captures an image of the region at any time during the process of inserting the nail into the placement table. The loading state determination device according to claim 5 or 6.
8. and a limiting unit that limits at least one of a moving speed and a turning speed of the cargo handling vehicle when the determining unit determines that the luggage is disposed offset with respect to the reference axis. The loading state determination device according to any one of claims 1 to 7.
9. A communication terminal having the cargo handling state determination device according to any one of claims 1 to 8; An in-vehicle device having the imaging device, Driving assistance system.
10. A driving assistance program for assisting the driving of a loading vehicle that transports a platform on which luggage is placed, comprising: acquiring an image from an imaging device attached to the cargo handling vehicle, the imaging device imaging an area including a part of the cargo handling vehicle, the baggage, and the platform; detecting an outline coordinate of the baggage in the image; determining whether the luggage is positioned offset with respect to a reference axis according to a position of an outer coordinate of the luggage with respect to the reference axis set in the image; A driving assistance program that executes the following:
Citation Information
Patent Citations
Program for processing image using image processing device, or computer, and recording medium storing the same
JP2003097918A
Forklift
JP2013230903A
Fork lift
JP2016210586A
On-vehicle device, cargo-handling machine, control circuit, control method, and program
JP2018158779A
Image processing apparatus, camera deviation determination system, and image processing method
JP2018160732A
