Forklift operation support system, forklift operation support method, and program

Abstract

To provide a forklift operation support system that provides operation support images to forklift operators according to the vehicle status, without reducing work efficiency and improving visibility. [Solution] A forklift operation support system 10 comprises a forklift 101 that can communicate with each other, and a display device 201 that displays operation support images to assist in the operation of the forklift 101. The forklift 101 comprises two or more cameras 105 that capture images of the area around the forklift 101, a determination unit 111 that acquires status information of the forklift 101 and determines the vehicle status, a determination unit 112 that determines the image configuration of the operation support image based on the determined vehicle status of the forklift 101, and an image processing unit 113 that integrates two or more images to generate operation support image data based on the determined screen configuration. The display device 201 receives the generated operation support image data and displays the operation support image.

Description

Technical Field

[0001] The present invention relates to a forklift operation support system for assisting in the operation of a forklift, a forklift operation support method, and a program.

Background Art

[0002] In a warehouse, the arrangement of goods changes constantly for temporary storage and shipping operations. Therefore, in forklift operations in the warehouse (manned forklift operations), even at the same work site, the blind spots change depending on the situation of the day and time, and situations that are difficult to visually recognize from the driver's seat occur. Also, in high-place cargo handling operations, the upper shelf surface cannot be seen, and for example, the insertion state of the claws (forks) cannot be accurately known. Therefore, the driver (hereinafter also referred to as an operator) has to operate by estimating based on experience and intuition from the view visible from the driver's seat. That is, it is necessary to perform necessary confirmation work and judgment with little information visible from the driver's seat, and the stress on the driver is great. For this reason, there is concern about a labor shortage in the future.

[0003] Recently, with the development of wireless communication systems, it has become possible to provide high-quality wireless networks such as high capacity and low latency, and remote applications that require high communication quality such as telemedicine services and remote robot control, which have been difficult to achieve until now, are being studied. Also in the logistics industry, in response to this trend, remote forklifts aimed at eliminating labor shortages through multi-site support and applying to harsh environments such as refrigerated and frozen warehouses are being studied.

[0004] As technologies for improving the visibility of an operator during remote operation or when riding in the driver's seat of a forklift, Patent Documents 1 and 2 are disclosed. Patent Document 1 discloses an invention that ensures a sufficient field of view while suppressing the consumption of communication resources by manually switching between a running system camera image and a cargo handling system camera image by an operator. Also, Patent Document 2 discloses an invention that improves visibility by superimposing and displaying a mark on the front surface of a pallet that is the cargo handling target.

Prior Art Documents

[0005] [Patent Document 1] Patent No. 6959532 [Patent Document 2] Japanese Patent Publication No. 2021-66539 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] In Patent Document 1, multiple camera images are manually switched, allowing for flexible control of the field of view according to the vehicle status of the forklift, such as driving and loading. However, this method requires time for switching decisions and issuing switching instructions, resulting in reduced work efficiency. In Patent Document 2, while visibility can be improved by automatically recognizing pallets and drawing marks using image recognition, it cannot recognize the vehicle status of the forklift, making it difficult to control the switching of visual information according to the vehicle status.

[0007] Therefore, the present invention aims to provide a forklift operation support system, a forklift operation support method, and a program that can provide the forklift operator with images corresponding to the vehicle status, without reducing work efficiency and with improved visibility, whether the forklift is operated by riding on the actual machine or by remote control. [Means for solving the problem]

[0008] A representative example of the invention disclosed in this application is as follows: That is, a forklift operation support system comprising forklifts capable of communicating with each other, and a display device that displays operation support images to assist in the operation of the forklifts, The aforementioned forklift is Two or more cameras that capture images of the area around the forklift, A determination unit that acquires status information of the forklift and determines the vehicle status, A determination unit that determines the image configuration of the operation support image based on the determined vehicle state of the forklift, An image processing unit that generates the operation support image data by integrating two or more of the images based on the determined screen configuration, Equipped with, The display device is characterized by receiving the generated operation support image data and displaying the operation support image. [Effects of the Invention]

[0009] According to the present invention, whether the forklift is operated while riding on the actual machine or remotely, it is possible to provide the forklift operator with operation support images that correspond to the vehicle status, without reducing work efficiency and with improved visibility. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic perspective view of a forklift operation support system according to the first embodiment of the present invention. [Figure 2] This figure shows an example of camera position and camera field of view according to the first embodiment of the present invention. [Figure 3] This is a block diagram showing the software configuration of a forklift operation support system according to the first embodiment of the present invention. [Figure 4] This diagram shows the hardware configuration of the operation support control unit included in a forklift according to the first embodiment of the present invention. [Figure 5] This is a block diagram showing an example of the hardware configuration of a display device according to the first embodiment of the present invention. [Figure 6] This is a flowchart showing the image processing performed by the operation support control unit according to the first embodiment of the present invention based on the vehicle state. [Figure 7] These are examples of image configurations for each vehicle condition. [Figure 8]It is a flowchart showing the vehicle state determination process in the determination unit according to the first embodiment of the present invention. [Figure 9] It is a diagram showing an example of a vehicle state transition model used by the determination unit according to the first embodiment of the present invention to manage the vehicle state. [Figure 10] It is a diagram showing an example of a screen of a display unit on which an operation support image in a traveling state according to the first embodiment of the present invention is displayed. [Figure 11] It is a block diagram showing the software configuration of a forklift operation support system according to the second embodiment of the present invention. [Figure 12] It is a diagram showing an example of an analysis result display screen output by the communication monitoring unit according to the second embodiment of the present invention. [Figure 13] It is a flowchart showing the communication analysis process by the communication monitoring unit according to the second embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0011] Embodiments of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments shown below. These examples are merely illustrative, and the present invention can be implemented in various modified and improved forms based on the knowledge of those skilled in the art.

[0012] In this specification and the drawings, components with the same reference numerals indicate the same components as each other. Also, when it is not necessary to separately explain components, they are described without suffixes (for example, communication unit 107), and when it is necessary to separately explain components, they are described with suffixes (for example, communication unit 107-A). Furthermore, an image means a still image or a moving image captured or displayed on a screen, and image data means a still image or a moving image processed or transmitted by the system.

[0013] In this specification, a forklift means a self-propelled material handling vehicle equipped with forks and a mast for raising and lowering the forks, and the forks are jigs for holding the object to be handled, and may be attachment jigs other than claws (hereinafter also simply referred to as "attachments").

[0014] <First Embodiment of the Invention> Figure 1 is a schematic diagram of the forklift operation support system 10 in this embodiment. The forklift operation support system 10 integrates two or more images captured by cameras installed on the forklift to generate operation support image data corresponding to the vehicle status of the forklift, provides the generated operation support image data to a communication-connected display device, and displays the operation support image on the display device.

[0015] As shown in Figure 1, the forklift operation support system 10 includes a forklift 101 equipped with multiple cameras 105 and a display device 201. For example, the multiple cameras 105 consist of seven cameras 105 attached externally to the vehicle of the forklift 101. In this embodiment, each of the seven cameras 105 consists of a rear right camera 105-1, a rear left camera 105-2, a rear camera 105-3, a front right camera 105-4, a front left camera 105-5, a lower front camera 105-6, and a lower camera 105-7.

[0016] Furthermore, the configuration of the multiple cameras 105 is not limited to that shown in Figure 1. The front right camera 105-4 and the front left camera 105-5 may be either one or the other, and the number of cameras is not limited to seven, but can be any number of two or more. In addition, the cameras 105 do not have to be mounted on the forklift 10, but may be mounted on a warehouse or a drone, or in any other position, as long as they can capture images of the area around the forklift.

[0017] Figure 2 shows an example of the field of view of camera 105. Each camera is not limited to a 2D camera; it may also be a 3D camera (e.g., a stereo camera), and the images captured by the cameras may be either two-dimensional or three-dimensional. Furthermore, each camera 105 may use a 2D camera and a three-dimensional sensor such as LiDAR to achieve the same image as a 3D camera.

[0018] Figure 3 is a block diagram showing the software configuration of the forklift operation support system 10 in this embodiment. The forklift 101 receives operation commands from the operator via a control stick in the driver's seat or a remote control device, and performs driving or cargo handling operations. The forklift 101 includes a drive system 102, a sensor 103, a camera 105, an operation support control unit 106, a determination unit 111, a decision unit 112, an image processing unit 113, a storage unit 114, and a communication unit 107-A.

[0019] The display device 201 comprises a display unit 211 and a communication unit 107-B, and is a suitable display device such as a tablet, which receives and displays operation support image data generated by the forklift 101. The communication means between the display device 201 and the forklift 101 may be wired or wireless.

[0020] The operator on board the forklift 101 operates the forklift 101 by operating the control stick (not shown) or the like while looking at the operation support images displayed on the display unit 211, for parts that are difficult to see from the driver's seat. In this case, the display device 201 is installed in a position that is easy for the operator to see. In addition, the operator remotely controlling the forklift 101 operates the forklift 101 remotely by operating remote control devices (not shown), such as a remote control stick or remote controller, while looking at the operation support images displayed on the display unit 211.

[0021] The drive system 102 includes a travel device that moves in response to a travel command from the operator, a cargo handling device that performs cargo handling operations in response to a cargo handling command from the operator, and a drive control unit that controls the travel device and the cargo handling device. Specifically, the drive system 102 controls the forks, wheels, mast, turn signals, etc., based on operation commands input to a control stick (not shown) or remote control device (not shown) provided on the forklift 101. Sensor 103 is a sensor for acquiring information about the forklift 101 and the work site environment, and can be optionally attached. Examples include load sensors, contact sensors, distance sensors, acceleration sensors, light sensors, temperature sensors, and three-dimensional sensors.

[0022] Camera 105 acquires images of the area around the forklift. Multiple images captured by multiple cameras 105 are integrated by the image processing unit 113 (described later) and converted into operation support image data. The operation support image data is then encoded and transmitted to the display device 201. The encoded operation support image data is decoded by the display device 201 and displayed on the display unit 211 as an operation support image.

[0023] The determination unit 111 collects state information, which is information about each part of the forklift, and determines the vehicle state, which represents the operating state of the forklift, based on the collected state information. Here, the state information is obtained from the drive system 102 and sensors 103, and includes machine speed, reach-out amount, ON / OFF status of the load sensor (depth / up / down), mast angle, fork height, fork left / right position, steering angle, battery (or fuel) level, operating mode, etc.

[0024] In this embodiment, there are four vehicle states: driving state, loaded state, loading / unloading driving state, and unloading state. The first driving state refers to the forklift driving without holding a load (load handling object 701). The second driving state refers to the forklift driving with a load. The third loading state refers to the forklift in the process of loading a load. The last unloading state refers to the forklift in the process of unloading a load. The vehicle status is determined from the ON / OFF status of the load sensors (depth / up / down), reach-out amount, mast angle, etc. The detailed determination process will be described later in Figures 8 and 9.

[0025] The determination unit 112 retrieves the screen configurations pre-set for each vehicle state from the storage unit 114 based on the vehicle state acquired by the judgment unit 111, and determines the screen configuration of the operation support image to be output to the display unit 211. The determination unit 112 also retrieves the presence or absence of image processing and the processing content pre-set for each vehicle state and camera from the storage unit 114 based on the vehicle state acquired by the judgment unit 111, and determines the presence or absence of image processing and the processing content (for example, image enlargement / reduction) of the image acquired by camera 105.

[0026] The image processing unit 113 processes the image that the determination unit 112 has determined to require processing, based on the processing details also determined by the determination unit 112, to generate processed image data. Furthermore, the image processing unit 113 integrates processed image data for processed images and unprocessed images based on the screen configuration determined by the determination unit 112 to generate operation support image data. At this time, the image processing unit 113 integrates multiple images and / or processed image data to generate fewer operation support image data than the number of images and / or processed image data, preferably one image. Detailed processing will be described later in Figure 7. Furthermore, the image processing unit 113 encodes the generated operation support image data.

[0027] The display device 201 includes a forklift communication unit 107-B and a display unit 211. The display unit 211 decodes the operation support image data received from the forklift 101 via the forklift communication unit 107-B and displays the operation support image.

[0028] Figure 4 shows the hardware configuration of the operation support control unit 106 provided in the forklift 101. The CPU 301 is an arithmetic unit that executes programs stored in the memory 302. The various functions of the unit are realized by the CPU 301 executing various programs. Note that some of the processing performed by the CPU 301 through program execution may be performed by other arithmetic units (such as ASICs: Application Specific Integrated Circuits, FPGAs: Field Programmable Gate Arrays, etc.).

[0029] Memory 302 includes non-volatile memory elements such as ROM (Read Only Memory) and volatile memory elements such as RAM (Random Access Memory). ROM stores immutable programs (such as BIOS: Basic Input Output System). RAM is a high-speed, volatile memory element like DRAM (Dynamic Random Access Memory) and temporarily stores programs executed by the CPU 301 and data used during program execution.

[0030] The auxiliary storage device 303 is a high-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive) or flash memory (SSD: Solid State Drive). The auxiliary storage device 303 also stores data used by the CPU 301 during program execution, as well as the programs executed by the CPU 301. In other words, programs are read from the auxiliary storage device 303, loaded into memory 302, and executed by the CPU 301, thereby realizing the various functions of the device.

[0031] The communication interface 304 is a network interface device that controls communication with other devices according to a predetermined protocol. The communication interface 304 is used to acquire vehicle information from the sensor 103 attached to the forklift, and can also provide a wired or wireless communication environment when transmitting data to the display device 201. The input / output I / F305 is an interface device to which devices requiring input / output of electrical signals, such as the drive system 102, sensor 103, and camera 105, are connected, and which performs data acquisition and control using analog or digital electrical signals.

[0032] The input / output interface 305 and sensor 103 are connected by a wire, and sensor values ​​are acquired using SPI (Serial Peripheral Interface), I2C (registered trademark), RS-232C (Recommended Standard-232C), Modbus (registered trademark), etc. The input / output interface 305 and camera 105 are connected by a wire, and image data is acquired using USB®, HDMI®, Ethernet®, etc.

[0033] The GPU306 is a processing unit specialized for image processing. It works in conjunction with the CPU301 to perform some image processing tasks at high speed.

[0034] Figure 5 is a block diagram showing the hardware configuration of the display device 201 in this embodiment. The display device 201 is a computer consisting of a CPU 301, memory 302, auxiliary storage device 303, communication interface 304, and display 307. The CPU 301, memory 302, auxiliary storage device 303, communication interface 304, and display 307 are connected by a common transmission path, which is a bus. Detailed explanations of components that overlap with those in Figure 4 are omitted.

[0035] The display 307 is a device that displays operation support images and may consist of multiple displays. Since the operation support image data received from the forklift 101 is encoded, the operation support image data received from the image processing unit 113 via the communication I / F 304 is decoded by a processing unit such as the CPU 301 and displayed as an operation support image. The display device 201 may be connected to an external computing device that performs decoding processing of operation support image data. The display device 201 may also be equipped with an internal or external remote control device for the operator to input operation commands in order to remotely control the forklift 101. If the remote control device is provided externally, the remote control device may communicate directly with the forklift 101, or if it is directly connected to the forklift 101, the display device 201 may communicate with the forklift 101 via the remote control device.

[0036] Figure 6 is a flowchart illustrating the operation support image data generation process performed by the operation support control unit 106 in this embodiment based on the vehicle status. This process is continuously and repeatedly executed at predetermined fixed intervals according to the frame rate of the camera 105, etc.

[0037] Step S1001: The image processing unit 113 acquires an image from the camera 105. Note that images may be acquired from only some of the connected cameras 105, rather than all of them. Step S1002: The determination unit 111 determines the vehicle status, such as the driving status and the loading status, in response to the image acquired in step S1001. The determination unit 111 may also determine the vehicle status at predetermined intervals. In this case, the decision unit 112, described later, determines the screen configuration of the operation support image using the vehicle status most recently acquired by the image processing unit 113. The vehicle status determination process in the determination unit 111 will be described later in Figures 8 and 9.

[0038] Step S1003: The determination unit 112 determines the screen configuration of the operation support image based on the vehicle state determined by the judgment unit 111 in step S1002. More specifically, the determination unit 112 retrieves the image configuration associated with the vehicle state determined by the judgment unit 111 in step S1002 from the storage unit 114, which has pre-stored image configurations associated with vehicle states, and determines it as the screen configuration of the operation support image.

[0039] Figure 7 shows an example of an image configuration associated with the vehicle status. In this embodiment, the display unit 211 is composed of two displays, so an example of an image configuration when operation support images are displayed on two screens is shown. Note that the image configuration shown in Figure 7 is just one example of a desirable configuration, and the screen configuration (screen arrangement and / or components) can be arbitrarily set by the system administrator or operator.

[0040] Figure 7(a) shows the image configuration of the operation support images (hereinafter also referred to as the driving image configuration) when the forklift 101 is driving without holding a load. The driving image configuration consists of images that need to be checked when driving, and comprises a first driving image 401-C1 composed of images acquired by front cameras 105-5, 105-4, and 105-7 that capture the front side in the direction of travel, and a second driving image 402-C1 composed of images acquired by rear cameras 105-3, 105-2, and 105-1 that capture the rear side in the direction of travel.

[0041] The configuration of the first driving image 401-C1 consists of images from camera 105-5 (hereinafter also referred to as image 5) and camera 105-4 (hereinafter also referred to as image 4) placed side by side at the top, and an image from camera 105-7 (hereinafter also referred to as image 7) placed at the bottom.

[0042] Furthermore, the second driving image 402-C1 is configured such that the image from camera 105-3 (hereinafter also referred to as image 3) is placed on the left half, and the image from camera 105-2 (hereinafter also referred to as image 2) and the image from camera 105-1 (hereinafter also referred to as image 1) are placed side by side on the right half. Furthermore, if the image processing unit 113 processes the image into processed image data, the processed image data will be placed in place of the original image in the running image configuration and in each of the image configurations described later.

[0043] Next, Figure 7(b) shows the image configuration of the operation support images (hereinafter also referred to as the cargo handling driving image configuration) of the forklift 101 in the cargo handling driving state when loaded with cargo. The cargo handling driving image configuration consists of images that need to be checked during cargo handling driving, and comprises a first cargo handling driving image 401-C3 composed of images acquired by front cameras 105-5, 105-4, and 105-6 that capture images of the front side in the direction of travel, and a second cargo handling driving image 402-C3 composed of images acquired by rear cameras 105-3, 105-2, and 105-1 that capture images of the rear side in the direction of travel.

[0044] The first cargo handling driving image 401-C3 is composed of images 5 and 4 placed side by side at the top, and the image from camera 105-6 (hereinafter also referred to as image 6) at the bottom. The reason why the first cargo handling driving image 401-C3 differs from the first driving image 401-C1 in that image 6 is a component instead of image 7 is that, since the cargo handling driving condition involves driving while holding the load, image 7 mainly shows only the load, while image 6 shows not only the load but also the area around the load. Furthermore, the second loading / unloading travel image 402-C3 has the same image and layout configuration as the second travel image 402-C1.

[0045] Next, Figure 7(c) shows the image configuration of the operation support image of the loading state during the loading operation of the forklift 101 (hereinafter also referred to as the loading image configuration). The loading image configuration consists of images that need to be checked during loading, and comprises a first loading image 401-C2 composed of images acquired by the front cameras 105-5, 105-4, and 105-7 that capture the front side in the direction of travel, and a second loading image 402-C2 composed of images acquired by the rear cameras 105-2 and 105-1 that capture the rear side in the direction of travel, and the front lower left camera 105-6 that captures the front lower side from an oblique angle in the direction of travel.

[0046] The first loading image 401-C2 has the same image and layout configuration as the first driving image 401-C1 described above. The second loading image 402-C2 is configured in the same way as the second driving image 401-C1 described above, but with image 6 replaced by image 3 in the left half. The reason why image 6 is placed in place of image 3 in the second loading image 402-C2 is that the vehicle rarely moves in reverse when loading, while image 6 shows the area around the cargo that needs to be checked.

[0047] Finally, Figure 7(d) shows the image configuration of the operation support images (hereinafter also referred to as the unloading image configuration) of the forklift 101 during unloading work. The unloading image configuration consists of a first unloading image 401-C4 composed of images acquired by the front cameras 105-5, 105-4, and 105-6 that capture the front side in the direction of travel, and a second unloading image 402-C4 composed of images acquired by the rear cameras 105-2 and 105-1 that capture the rear side in the direction of travel, and the front lower center camera 105-7 in the direction of travel.

[0048] The first unloading image 401-C4 has the same image and layout configuration as the first loading / unloading driving image 401-C1 described above. The second unloading image 402-C4 is configured in the same way as the second driving image 402-C1 described above, but with image 7 placed in the left half instead of image 3. The reason why image 7 is placed in place of image 3 in the second unloading image 402-C4 is that reversing is infrequent during unloading, while image 7 shows the area below the load that needs to be checked.

[0049] Returning to Figure 6, Step S1004: Based on the vehicle state determined in step S1002, the determination unit 112 determines whether or not each image, which is a component of the operation support image, has been processed and what the processing details are. Specifically, the determination unit 112 retrieves the processing details from the storage unit 114, which stores in advance the vehicle state and / or the presence or absence of processing and processing details associated with the camera, for images that are associated with the vehicle state determined in step S1002 and have been processed, and then determines the processing details. The processing options include removing unwanted reflections from parts of the image, enlarging areas for detailed viewing or shrinking areas for overall viewing, and mirroring the image to reverse its horizontal orientation.

[0050] Step S1005: The image processing unit 113 processes the image whose processing content was determined in step S1004 according to the processing content to generate processed image data. Step S1006: The image processing unit 113 integrates the processed image data from the images processed in step S1005, and the unprocessed images from the images not processed in step S1005, based on the screen configuration determined in step S1003, to generate operation support image data. Step S1007: The image processing unit 113 analyzes the measurement information and images acquired from the drive system 102 and the sensor 103, generates additional information based on the analysis results, and adds it to the operation support image data generated in step S1006. The additional information to be generated may be determined according to the vehicle status. The additional information to be added to the operation support image data is set arbitrarily by the operator before starting forklift operation, etc., but after starting operation, the operator can switch the display of the added additional information ON / OFF.

[0051] The additional information generated by the image processing unit 113 can be broadly divided into aircraft-related additional information concerning the aircraft's status and warning-related additional information that notifies the operator of a warning. Additional aircraft information includes aircraft speed, battery (or fuel) level, fork side shift, fork height, reach-out OK, fork insertion OK, load loading OK, tilt OK, etc. Additional warning information includes notifications about obstacles or people near forklift 101, and notifications about the height of the forklift's forks.

[0052] First, let's explain the additional aircraft information with an example. For example, the image processing unit 113 acquires the vehicle speed from the sensor 103, generates a digital display or speedometer icon indicating the acquired vehicle speed as additional vehicle information, and superimposes it onto the operation support image data. The image processing unit 113 may also generate additional vehicle information regarding vehicle speed only when the vehicle is in motion or during cargo handling. Furthermore, the image processing unit 113 acquires the remaining battery (or fuel) level from the sensor 103, generates a digital display showing the acquired battery (or fuel) level, a battery (or fuel) level icon, etc., as additional aircraft information, and superimposes it onto the operation support image data.

[0053] Furthermore, the image processing unit 113 acquires the left-right position of the claw from the sensor 103, calculates the amount of side shift of the claw based on the acquired left-right position, generates a digital display or gauge icon showing the calculated side shift amount as additional aircraft information, and superimposes it onto the operation support image data. The position of superimposition is arbitrary, and the same applies to other icons. Furthermore, the image processing unit 113 acquires the height of the claw (mast) from the sensor 103, generates a digital display or gauge icon indicating the height of the claw (mast) as additional aircraft information, and superimposes it onto the operation support image data.

[0054] Furthermore, the image processing unit 113 acquires the reach-out amount from the sensor 103, and if the acquired reach-out amount is greater than or equal to a predetermined value, it generates an icon indicating reach-out as additional aircraft information and superimposes it on the operation support image data. The image processing unit 113 may also generate a chromatic icon, such as blue or red, indicating reach-out when the reach-out amount is greater than or equal to a predetermined value, and an achromatic icon, such as gray, indicating reach-out when the reach-out amount is less than a predetermined value, as additional aircraft information. The image processing unit 113 may also generate additional aircraft information regarding reach-out only when the vehicle state is loading or unloading. Furthermore, the image processing unit 113 obtains the ON / OFF status of the load sensor (depth) from the sensor 103, and if it is "ON", it generates an icon indicating claw insertion as additional machine information and superimposes it on the operation support image data. The image processing unit 113 may also generate a colored icon indicating claw insertion when the load sensor (depth) is "ON", and an achromatic icon indicating claw insertion when it is "OFF" as additional machine information. The image processing unit 113 may also generate additional machine information regarding claw insertion only when the vehicle state is loading or unloading.

[0055] Furthermore, the image processing unit 113 obtains the ON / OFF status of the load sensor (depth / up / down) from the sensor 103, and if it is "ON", it generates an icon indicating loading as additional machine information and superimposes it on the operation support image data. The image processing unit 113 may also generate a colored loading icon as additional machine information when the load sensor (depth / up / down) is "ON", and a colorless loading icon when it is "OFF". The image processing unit 113 may also generate additional machine information related to loading only when the vehicle status is loading or unloading. Note that the ON / OFF status of the load sensor (depth / up / down) is "ON" when both the load sensor (depth) and the load sensor (up / down) are "ON", and "OFF" otherwise. Furthermore, the image processing unit 113 acquires the tilt angle from the sensor 103, and if the acquired tilt angle is greater than or equal to a predetermined value, it generates an icon indicating tilt as additional aircraft information and superimposes it on the operation support image data. The image processing unit 113 may also generate a chromatic icon indicating tilt when the acquired tilt angle is greater than or equal to a predetermined value, and an achromatic icon indicating tilt when the acquired tilt angle is less than the predetermined value as additional aircraft information. The image processing unit 113 may also generate additional aircraft information related to tilt only when the vehicle state is loading or unloading.

[0056] Next, we will explain additional warning information with an example. The image processing unit 113 uses deep learning based on images captured by the camera to detect obstacles and / or people in the surrounding area and in the direction of travel. The image processing unit 113 then generates warning information to alert the detected obstacles and / or people, such as a warning icon with an exclamation mark inside a yellow triangle, or a border around the obstacles and / or people, and superimposes it onto the operation support image data. Note that the additional information is not limited to warning icons and borders superimposed on the operation support image data; it may also include information that instructs the operation support image data to flash or information that instructs the output of a warning sound. The image processing unit 113 may generate warning information regarding obstacles and / or people only when the vehicle is in motion or during cargo handling.

[0057] The image processing unit 113 issues a warning about the height of the forklift attachment or mast if the height deviates from a predetermined value. Specifically, it obtains the height of the forks (mast) from the sensor 103, and if the obtained height of the forks (mast) is above the upper limit, it issues a warning about the height of the forks, for example, by generating a warning icon with "STOP" or "Caution: Ceiling" written on it as additional warning information. The image processing unit 113 also obtains the height of the forks (mast) from the sensor 103, and if the obtained height of the forks (mast) is below the lower limit, it issues a warning about the height of the forks, for example, by generating a warning icon with "STOP" written on it as additional warning information. The image processing unit 113 may not only display warning icons, but also display information instructing the flashing of operation support images or information instructing the output of warning sounds. Furthermore, the upper and lower limits of the claw height are pre-set by the operator and stored in the memory unit 114. The image processing unit 113 may also generate additional warning information regarding the claw height only when the vehicle is loading or unloading.

[0058] Returning to Figure 6, Step S1008: The image processing unit 113 performs encoding processing on the operation support image data to which additional information was added in step S1007. This makes it possible to reduce the amount of communication resources used when transmitting images. Step S1009: The image processing unit 113 transmits the operation support image data after the encoding process in step S1008 to the display unit 211.

[0059] Figure 8 is a flowchart illustrating the process for determining the vehicle status in the determination unit 111 in this embodiment. The determination unit 111 starts processing at predetermined intervals or in response to requests from the image processing unit 113. In the case of predetermined intervals, it is desirable that processing be performed at sufficiently short intervals so that the latest vehicle status can always be determined.

[0060] Step S1101: The determination unit 111 acquires status information from the drive system 102 and the sensor 103. The status information acquired in this step includes at least the ON / OFF status of the load from the load sensor (depth / up / down) and the mast angle from the instrument, and in the case of a reach forklift, the reach-out amount is also included.

[0061] Step S1102: The determination unit 111 determines the vehicle state based on the state information obtained in step S1101 and the vehicle state transition model. The vehicle state transition model is an information model consisting of vehicle states and transition conditions between vehicle states. An example of the vehicle state transition model in this embodiment is shown in Figure 9.

[0062] In this embodiment, the vehicle state transition model comprises four states: driving state C1, loading state C2, loading / unloading driving state C3, and unloading state C4. The initial state is driving state C1. Each state can transition according to predetermined transition conditions, as indicated by the arrows.

[0063] The determination unit 111 determines whether the condition for transitioning from the current vehicle state to the next vehicle state is met by the state information obtained in step S1101, and if it is met, it determines that the vehicle has transitioned to the next vehicle state.

[0064] For example, in the case of a reach forklift, if the current state is driving state C1, the vehicle state transitions to loading state C2 when the forks reach out. That is, if the current state is driving state C1, the determination unit 111 determines that the vehicle state has transitioned to the next state, loading state, if the reach-out amount of the state information acquired in step S1102 is greater than 0. If the reach-out amount of the state information acquired in step S1101 is 0 or less, the vehicle remains in the current driving state C1.

[0065] In addition to the configuration shown in Figure 9, other vehicle states such as emergency stop, high-speed driving, low-speed driving, and human detection may also be managed. The initial state does not have to be driving state C1. The arrows representing transitionable states may be connected to multiple vehicle states, such as an arrow further connected from the loaded state C2 towards the driving state C1.

[0066] Figure 10 is an example of the screen of the display unit showing operation support images during driving in this embodiment. The operation support image data shown in the figure consists of a first driving image 401-C1 consisting of forward images 4, 5, and 7, and a second driving image 402-C1 consisting of rear images 1, 2, and 3, based on the driving image configuration.

[0067] In the driving image 401-C1, icons I1 are superimposed, indicating reach-out OK, fork insertion OK, load loading OK, and tilt OK in gray; in other words, icons meaning no reach-out, no fork insertion, no load loading, and no tilt. Additionally, a scale icon I2 indicating the amount of side shift is superimposed on the driving image 401-C1. Furthermore, since a person is visible in image 4 of the driving image 401-C1, a frame I3 surrounding the person and a warning icon I4 with an exclamation mark inside a triangle are superimposed. A warning sound may also be added to the driving image 401-C1 and outputted. In the driving image 402-C1, a battery (or fuel) level icon I5, indicating the remaining battery (or fuel) level, and a speedometer icon I6, indicating the vehicle's speed, are superimposed. The superimposed positions of the icons can be set arbitrarily, but they may also be determined according to the vehicle's status. Thus, the operation support screen data has a screen configuration that corresponds to the vehicle status, and the operator's visibility is enhanced by the overlaying of various icons and the addition of sound.

[0068] As described above, the forklift operation support system in this embodiment integrates two or more images captured by a camera mounted on the forklift according to the vehicle status, generates operation support image data with a smaller data capacity than the two or more original image data, encodes the generated operation support image data, and transmits it to a display device. This reduces the consumption of communication resources compared to transmitting two or more captured image data to the display device, preventing a shortage of communication resources, and allows for simultaneous viewing of multiple images, thus providing operation support images that make it easier for the forklift operator to operate without reducing work efficiency.

[0069] Furthermore, the operation support image data is configured such that the image composition, whether or not the images of the constituent elements are processed, and / or the content of the image processing are determined according to the vehicle condition, making it easier for the operator to check different operational checkpoints depending on the vehicle condition. In addition, the operation support image data can be supplemented with additional vehicle information (e.g., reach-out) that the operator cannot confirm by visual inspection or operation support image data alone, as well as warning information that notifies the operator of hazards that could lead to accidents if overlooked. These features further improve the visibility of the operation support image data. Therefore, whether operating a forklift by riding on the actual machine or by remotely controlling it, it becomes possible to provide the forklift operator with operation support image data that corresponds to the vehicle's condition, without reducing work efficiency and improving visibility.

[0070] <Second Embodiment> The second embodiment includes a function to monitor whether operation support image data, generated by integrating two or more images according to the vehicle status, is being transmitted correctly to the display device. This function makes it easy for the operator to determine whether to stop the operation due to a system failure or a serious deterioration in communication quality, or to continue the operation due to a temporary problem, when remotely operating a forklift in a warehouse environment or other environment where wireless communication quality fluctuates frequently, if the display unit 211 displays an operation support image with some data missing. Note that the same reference numerals are used for the same configurations and processes as in the first embodiment, and their descriptions are omitted.

[0071] Figure 11 is a block diagram showing an example of the software configuration in the forklift operation support system 20 in this embodiment. In this embodiment, the operation support control unit 126 of the forklift 121 is equipped with a communication unit 127-A instead of a communication unit 107-A. The display device 202 is equipped with a communication unit 127-B instead of a communication unit 107-B, and further includes a communication monitoring unit 212.

[0072] Communication units 127-A and 127-B measure the communication quality between communication unit 127-A and communication unit 127-B, respectively. Communication quality includes delay, throughput, traffic volume, packet loss rate, and radio signal strength, and is measured by methods such as monitoring IP (Internet Protocol) packets passing through communication I / F 304, and sending and receiving measurement IP (Internet Protocol) packets containing timestamps and sequence numbers between communication unit 127-A and communication unit 127-B.

[0073] The communication monitoring unit 212 collects the communication quality measured by the communication units 127-A and 127-B, analyzes the collected communication quality, and notifies the operator of the analysis results. The analysis results are notified to the operator in a visualized form, as shown in Figure 12. In addition, the communication monitoring unit 212 may notify the operator of the collected communication quality all at once so that the operator can easily understand the situation in detail. Alternatively, the analysis results may be notified by sound instead of visualized information.

[0074] Figure 13 is a flowchart of the communication analysis process performed by the communication monitoring unit 212 in this embodiment. The communication monitoring unit 212 repeatedly performs this process at predetermined intervals and sequentially provides the operator with the communication analysis results regarding whether the operation support image data is being transmitted correctly.

[0075] Step S1201: The communication monitoring unit 212 acquires the vehicle status from the determination unit 111 via the communication unit 127-A and the communication unit 127-B. Step S1202: The communication quality is acquired from the communication units 127 (127-A, 127-B) of both the operation support control unit 126 and the display device 202. Alternatively, the measurement results of the communication quality may be collected in advance on one of the communication units 127-B, etc., and the results may be acquired.

[0076] Step S1203: The communication monitoring unit 212 performs a communication analysis process on the communication quality acquired in step S1202 based on the vehicle status acquired in step S1201, and acquires the communication analysis results. The acquired analysis results are shown in Figure 12 above, and are transmitted to the display device 202 and may or may not be displayed on the display unit 211.

[0077] Step S1204: The communication monitoring unit 212 determines, based on the communication analysis results obtained in step S1203, whether the communication quality is showing an abnormal value, that is, whether a system failure or a serious deterioration in communication quality has occurred. For example, it stores the communication analysis results under normal conditions in advance, and if the error between the communication analysis results obtained in step S1203 and the communication analysis results under normal conditions becomes large, it determines that there is a system failure or a serious deterioration in communication quality.

[0078] The abnormal communication quality values ​​used for detection are created in advance for each vehicle state. For example, in driving state M1201, an abnormality is detected if the traffic volume falls outside 10% of 10Mbps, and in loaded state M1202, an abnormality is detected if the delay is greater than 100ms or the packet loss rate is greater than 0.01%. Note that abnormal communication quality detection may also be performed using statistical methods such as machine learning or neural networks. If the communication monitoring unit 212 determines that the communication quality is showing an abnormal value, it proceeds to step S1205. On the other hand, if it determines that the communication quality is not showing an abnormal value, it terminates the process.

[0079] Step S1205: The communication monitoring unit 212 transmits a message to the display device 202 indicating that the communication quality is abnormal, and notifies the operator by displaying it on the display unit 211, etc.

[0080] As described above, the forklift operation support system in this embodiment monitors and analyzes communication between the forklift and the display device, and can notify the operator whether a system failure or a serious deterioration in communication quality has occurred when distortion of the operation support image occurs. This makes it easier for the operator to decide whether to continue or stop the operation, thereby preventing a decrease in safety and unnecessary work stoppages.

[0081] Furthermore, since the operation support image data transmitted to the display device 202 differs depending on the vehicle status, determining whether a system failure or a serious deterioration in communication quality has occurred can be performed according to the vehicle status, thereby improving the accuracy of the determination. In addition, the operator can understand the current communication quality by being presented with the communication analysis results. [Explanation of Symbols]

[0082] 10, 20…Remote control system, 101, 121…Forklift, 201, 202…Display device, 103…Sensor, 105…Camera, 111…Determination unit, 112…Decision unit, 113…Image processing unit, 107, 127…Communication unit, 106, 126…Operation support control unit, 211…Display unit, 212…Communication monitoring unit

Claims

1. A forklift operation support system comprising forklifts that can communicate with each other, and a display device that displays operation support images to assist in the operation of the forklifts, The aforementioned forklift is Two or more cameras that capture images of the area around the forklift, A determination unit that acquires status information of the forklift and determines the vehicle status, A determination unit that determines the image configuration of the operation support image based on the determined vehicle state of the forklift, An image processing unit that generates the operation support image data by integrating two or more of the images based on the determined screen configuration, Equipped with, The display device is a forklift operation support system that receives the generated operation support image data and displays the operation support image.

2. The forklift is equipped with sensors that acquire state information regarding the information of each part of the forklift. The forklift operation support system according to claim 1, characterized in that the vehicle state is a driving state, a cargo handling driving state, a cargo loading state, and a cargo unloading state.

3. The vehicle state transitions in the following order: driving state, loading state, loading / unloading driving state, and unloading state, and the unloading state returns to the driving state. The transition from the aforementioned driving state to the aforementioned loading state occurs when the reach-out amount of the aforementioned state information meets predetermined conditions. The transition from the loading state to the cargo handling / driving state, and from the cargo handling / driving state to the unloading state, occurs when the mast angle of the state information satisfies predetermined conditions. The forklift operation support system according to claim 2, characterized in that the transition from the unloading state to the driving state occurs when the reach-out amount and the value of the load sensor for the state information meet predetermined conditions.

4. The determination unit determines whether or not to process the image and the content of the processing based on the determined vehicle condition of the forklift. The forklift operation support system according to claim 1, wherein the image processing unit integrates the processed image data obtained by processing the image determined to have been processed based on the determined processing content, and the image determined not to have been processed, based on the determined screen configuration, to generate the operation support image data.

5. The forklift operation support system according to claim 1, wherein the image processing unit adds machine additional information, which is information regarding the state of the forklift machine, to the operation support image data.

6. The forklift operation support system according to claim 1, wherein the image processing unit adds warning information that warns that there is an obstacle in the direction of travel of the forklift and / or that the height of the forklift attachment or mast has deviated from a predetermined value.

7. The image processing unit encodes the generated operation support image data and transmits it to the display device via the communication unit. The forklift operation support system according to claim 1, wherein the display device receives the encoded operation support image data, decodes it, and displays it.

8. The aforementioned display device The forklift operation support system according to claim 1, further comprising a communication monitoring unit that measures the communication quality between the forklift and the display device, analyzes whether the operation support image data is being transmitted normally based on the measured communication quality and the vehicle status, and outputs the analysis result.

9. A method to be performed by a forklift operation support system comprising forklifts that can communicate with each other, and a display device that displays operation support images to assist in the operation of the forklifts, The steps include: two or more cameras equipped on the forklift capture images of the area around the forklift; The steps include: acquiring status information of the forklift and determining the vehicle status; The steps include determining the image configuration of the operation support image based on the determined vehicle status of the forklift, Based on the determined screen configuration, the steps include: integrating two or more images to generate the operation support image data; The display device receives the generated operation support image data and displays the operation support image, A method that includes this.

10. A forklift operation support system comprising forklifts capable of communicating with each other, and a display device that displays operation support images to assist in the operation of the forklifts, A determination unit that acquires status information of the forklift and determines the vehicle status, A determination unit determines the image configuration of the operation support image based on the determined vehicle state of the forklift. An image processing unit generates the operation support image data by integrating two or more images of the area around the forklift, captured by two or more cameras provided on the forklift, based on the determined screen configuration. To make it function as, A program that receives the generated operation support image data and displays the operation support image on the display device.

Images