Forklift operation assistance system, forklift operation assistance method, and program

Abstract

[Problem] To provide a forklift operation assistance system that provides an operation assistance image corresponding to a vehicle state to a forklift operator with improved visibility and without a decrease in work efficiency. [Solution] A forklift operation assistance system 10 comprises forklifts 101 capable of mutual communication, and a display device 201 for displaying an operation assistance image for assisting the operation of the forklifts 101. The forklifts 101 each comprise two or more cameras 105 that capture images around the forklift 101; a determination unit 111 that acquires state information of the forklift 101 and determines a vehicle state; a decision unit 112 that decides an image configuration of the operation assistance image on the basis of the determined vehicle state of the forklift 101; and an image processing unit 113 that generates operation assistance image data by integrating two or more images on the basis of the decided screen configuration. The display device 201 receives the generated operation assistance image data and displays the operation assistance image.

Description

Forklift Operation Support System, Forklift Operation Support Method, and Program

[0001] The present invention relates to a forklift operation support system for supporting the operation of a forklift, a forklift operation support method, and a program.

[0002] In a warehouse, since the arrangement of goods changes constantly during temporary storage and shipping operations, etc., in forklift work in the warehouse (manned forklift work), even at the same work site, the blind spots change depending on the situation of the day and time, and it is difficult to visually recognize the working environment from the driver's seat. Also, in high-place cargo handling work, the upper shelf surface cannot be seen, and for example, the insertion state of the claws (forks) cannot be accurately known. Therefore, the driver (hereinafter also referred to as an operator) has to estimate the situation of the working environment based on the scenery visible from the driver's seat and operate based on experience and intuition. That is, the driver needs to perform necessary confirmation work and judgment with little information visible from the driver's seat, and the stress on the driver is great. For this reason, there is a concern about a shortage of labor force in the future.

[0003] Also, recently, with the development of wireless communication systems, it has become possible to provide high-quality wireless networks such as high capacity and low latency, and remote applications that require high communication quality such as telemedicine services and remote robot control, which have been difficult to implement until now, are being studied. The logistics industry is also following this trend, and remote forklifts are being studied for the purpose of resolving labor shortages through multi-site support and for application to harsh environments such as refrigerated and frozen warehouses.

[0004] As technologies for improving the visibility of an operator during remote operation or when riding in the driver's seat of a forklift, Patent Documents 1 and 2 are disclosed. Patent Document 1 discloses an invention in which an operator manually switches between a running system camera image and a cargo handling system camera image to ensure a sufficient field of view while suppressing consumption of communication resources. Also, Patent Document 2 discloses an invention in which visibility is improved by superimposing and displaying a mark on the front surface of a pallet that is the cargo handling target.

[0005] Japanese Patent No. 6959532, Japanese Unexamined Patent Application Publication No., 2021-66539

[0006] In Patent Document 1, multiple camera images are manually switched, allowing for flexible control of the field of view according to the vehicle status of the forklift, such as driving and loading. However, this method requires time for switching decisions and issuing switching instructions, resulting in reduced work efficiency. In Patent Document 2, while visibility can be improved by automatically recognizing pallets and drawing marks using image recognition, it cannot recognize the vehicle status of the forklift, making it difficult to control the switching of visual information according to the vehicle status.

[0007] Therefore, the present invention aims to provide a forklift operation support system, a forklift operation support method, and a program that can provide the forklift operator with images corresponding to the vehicle status, without reducing work efficiency and with improved visibility, whether the forklift is operated by riding on the actual machine or by remote control.

[0008] A representative example of the invention disclosed in this application is as follows: a forklift operation support system comprising a forklift capable of communicating with each other and a display device that displays operation support images to assist in the operation of the forklift, wherein the forklift comprises two or more cameras that capture images of the area around the forklift, a determination unit that acquires state information of the forklift and determines the vehicle state, a determination unit that determines the image configuration of the operation support image based on the determined vehicle state of the forklift, and an image processing unit that integrates the two or more images based on the determined screen configuration to generate the operation support image data, and the display device receives the generated operation support image data and displays the operation support image.

[0009] According to the present invention, whether the forklift is operated while riding on the actual machine or remotely, it is possible to provide the forklift operator with operation support images that correspond to the vehicle status, without reducing work efficiency and with improved visibility.

[0010] This is a schematic perspective view of a forklift operation support system according to the first embodiment of the present invention. This figure shows an example of camera position and camera field of view according to the first embodiment of the present invention. This is a block diagram showing the software configuration of a forklift operation support system according to the first embodiment of the present invention. This figure shows the hardware configuration of an operation support control unit provided in a forklift according to the first embodiment of the present invention. This is a block diagram showing an example of the hardware configuration of a display device according to the first embodiment of the present invention. This is a flowchart showing image processing performed by the operation support control unit according to the first embodiment of the present invention based on the vehicle state. This is an example of image configuration for each vehicle state. This is a flowchart showing the vehicle state determination process in the determination unit according to the first embodiment of the present invention. This figure shows an example of a vehicle state transition model used by the determination unit according to the first embodiment of the present invention to manage the vehicle state. This figure shows an example of the screen of a display unit showing an operation support image in the driving state according to the first embodiment of the present invention. This is a block diagram showing the software configuration of a forklift operation support system according to the second embodiment of the present invention. This figure shows an example of an analysis result display screen output by a communication monitoring unit according to the second embodiment of the present invention. This is a flowchart showing the communication analysis process by a communication monitoring unit according to the second embodiment of the present invention.

[0011] Embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the embodiments shown below. These examples are merely illustrative, and the present invention can be implemented in various modified and improved forms based on the knowledge of those skilled in the art.

[0012] In this specification and in the drawings, components with the same reference numeral shall indicate the same component. Furthermore, if it is not necessary to explain a component separately, it shall be described without a subscript (e.g., communication unit 107), and if it is necessary to explain a component separately, it shall be described with a subscript (e.g., communication unit 107-A). In addition, "image" means a still image or video captured or displayed on a screen, and "image data" means a still image or video processed by or transmitted by the system.

[0013] In this specification, a forklift means a self-propelled material handling vehicle equipped with forks and a mast for raising and lowering the forks, and the forks are fixtures for holding the object to be handled, and may be attachment fixtures other than claws (hereinafter also simply referred to as "attachments").

[0014] <First Embodiment of the Invention> Figure 1 is a schematic diagram of the forklift operation support system 10 in this embodiment. The forklift operation support system 10 is a system that integrates two or more images captured by a camera installed on the forklift to generate operation support image data corresponding to the vehicle state of the forklift, provides the generated operation support image data to a communication-connected display device, and displays the operation support image on the display device.

[0015] As shown in Figure 1, the forklift operation support system 10 includes a forklift 101 equipped with multiple cameras 105 and a display device 201. For example, the multiple cameras 105 consist of seven cameras 105 attached externally to the vehicle of the forklift 101. In this embodiment, each of the seven cameras 105 consists of a rear right camera 105-1, a rear left camera 105-2, a rear camera 105-3, a front right camera 105-4, a front left camera 105-5, a lower front camera 105-6, and a lower camera 105-7.

[0016] Furthermore, the configuration of the multiple cameras 105 is not limited to that shown in Figure 1. The front right camera 105-4 and the front left camera 105-5 may be either one or the other, and the number of cameras is not limited to seven, but can be any number of two or more. In addition, the cameras 105 do not have to be mounted on the forklift 10, but may be mounted on a warehouse or a drone, or in any other position, as long as they can capture images of the area around the forklift.

[0017] Figure 2 shows an example of the field of view of camera 105. Each camera is not limited to a 2D camera; it may also be a 3D camera (for example, a stereo camera), and the image captured by the camera may be either a two-dimensional or three-dimensional image. Furthermore, each camera 105 may use a 2D camera and a three-dimensional sensor such as LiDAR to achieve the same image as a 3D camera.

[0018] Figure 3 is a block diagram showing the software configuration of the forklift operation support system 10 in this embodiment. The forklift 101 receives operation commands from the operator via a control stick in the driver's seat or a remote control device, and performs driving or cargo handling operations. The forklift 101 includes a drive system 102, a sensor 103, a camera 105, an operation support control unit 106, a determination unit 111, a decision unit 112, an image processing unit 113, a storage unit 114, and a communication unit 107-A.

[0019] The display device 201 comprises a display unit 211 and a communication unit 107-B, and is a suitable display device such as a tablet, which receives and displays operation support image data generated by the forklift 101. The communication means between the display device 201 and the forklift 101 may be wired or wireless.

[0020] The operator on board the forklift 101 operates the forklift 101 by operating the control stick (not shown) or the like while looking at the operation support images displayed on the display unit 211 for parts that are difficult to see from the driver's seat. In this case, the display device 201 is installed in a position that is easy for the operator to see. In addition, the operator remotely controlling the forklift 101 operates the forklift 101 remotely by operating a remote control device (not shown), such as a remote control stick or remote controller, while looking at the operation support images displayed on the display unit 211.

[0021] The drive system 102 includes a travel device that moves in response to a travel operation from the operator, a cargo handling device that performs cargo handling operations in response to a cargo handling operation from the operator, and a drive control unit that controls the travel device and the cargo handling device. Specifically, the drive system 102 controls the forks, wheels, mast, turn signals, etc., based on operation commands input to a control stick (not shown) or remote control device (not shown) provided on the forklift 101. The sensors 103 are sensors that can be optionally attached to acquire information about the forklift 101 and the work site environment, and include load sensors, contact sensors, distance sensors, acceleration sensors, light sensors, temperature sensors, and three-dimensional sensors.

[0022] Camera 105 acquires images of the area around the forklift. Multiple images captured by multiple cameras 105 are integrated by the image processing unit 113 (described later) and converted into operation support image data. The operation support image data is then encoded and transmitted to the display device 201. The encoded operation support image data is decoded by the display device 201 and displayed on the display unit 211 as an operation support image.

[0023] The determination unit 111 collects state information, which is information about each part of the forklift, and determines the vehicle state, which represents the operating state of the forklift, based on the collected state information. Here, the state information is acquired by the determination unit 111 from the drive system 102 and sensors 103, and includes machine speed, reach-out amount, ON / OFF status of the load sensor (depth / up / down), mast angle, fork height, fork left / right position, steering angle, battery (or fuel) level, operating mode, etc.

[0024] In this embodiment, there are four vehicle states: driving state, loading state, cargo handling driving state, and unloading state. The first, driving state, indicates that the forklift is driving without holding a load (cargo handling object 701). The second, cargo handling driving state, indicates that the forklift is driving with a load. The third, loading state, indicates that the forklift is in the process of loading a load. The last, unloading state, indicates that the forklift is in the process of unloading a load. Which of the above four vehicle states the vehicle is in is determined from the ON / OFF status of the load sensor (depth / up / down), reach-out amount, mast angle, etc. Detailed determination processes will be described later in Figures 8 and 9.

[0025] The determination unit 112 retrieves a screen configuration pre-set for each vehicle state from the storage unit 114 based on the vehicle state acquired by the determination unit 111, and determines the screen configuration of the operation support image to be output to the display unit 211. The determination unit 112 also retrieves the presence or absence of image processing and the processing content pre-set for each vehicle state and camera from the storage unit 114 based on the vehicle state acquired by the determination unit 111, and determines whether or not to process the image acquired by the camera 105 and the processing content (for example, image enlargement / reduction).

[0026] The image processing unit 113 processes the images that the determination unit 112 has determined to be processed, based on the processing details also determined by the determination unit 112, to generate processed image data. The image processing unit 113 also integrates the processed image data for processed images and the unprocessed images for unprocessed images, based on the screen configuration determined by the determination unit 112, to generate operation support image data. At this time, the image processing unit 113 integrates multiple images and / or processed image data to generate fewer operation support image data than the number of images and / or processed image data, preferably one. Detailed processing will be described later in Figure 7. Furthermore, the image processing unit 113 encodes the generated operation support image data.

[0027] The display device 201 includes a forklift communication unit 107-B and a display unit 211. The display unit 211 decodes operation support image data received from the forklift 101 via the forklift communication unit 107-B and displays the operation support image.

[0028] Figure 4 shows the hardware configuration of the operation support control unit 106 provided in the forklift 101. The CPU 301 is an arithmetic unit that executes programs stored in the memory 302. The various functions of the device are realized by the CPU 301 executing various programs. Note that some of the processing performed by the CPU 301 when executing programs may be executed by other arithmetic units (hardware such as ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, etc.).

[0029] The memory 302 includes a non-volatile memory element called ROM (Read Only Memory) and a volatile memory element called RAM (Random Access Memory). The ROM stores immutable programs (such as BIOS: Basic Input Output System). The RAM is a high-speed, volatile memory element such as DRAM (Dynamic Random Access Memory) and temporarily stores programs executed by the CPU 301 and data used during program execution.

[0030] The auxiliary storage device 303 is a high-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive) or flash memory (SSD: Solid State Drive). The auxiliary storage device 303 also stores data used by the CPU 301 when executing programs, and the programs executed by the CPU 301. In other words, programs are read from the auxiliary storage device 303, loaded into memory 302, and executed by the CPU 301 to realize the various functions of the device.

[0031] The communication interface 304 is a network interface device that controls communication with other devices according to a predetermined protocol. The communication interface 304 is used to acquire vehicle information from the sensor 103 attached to the forklift, and can also provide a wired or wireless communication environment when transmitting data to the display device 201. The input / output interface 305 is an interface device to which devices requiring input and output of electrical signals, such as the drive system 102, sensor 103, and camera 105, are connected, and which performs data acquisition and control using analog or digital electrical signals.

[0032] The input / output interface 305 and sensor 103 are connected by a wire, and sensor values ​​are acquired using SPI (Serial Peripheral Interface), I2C (registered trademark), RS-232C (Recommended Standard-232C), Modbus (registered trademark), etc. The input / output interface 305 and camera 105 are connected by a wire, and image data is acquired using USB (registered trademark), HDMI (registered trademark), Ethernet (registered trademark), etc.

[0033] The GPU 306 is a computing unit specialized for image processing. It works in conjunction with the CPU 301 to perform some image processing tasks at high speed.

[0034] Figure 5 is a block diagram showing the hardware configuration of the display device 201 in this embodiment. The display device 201 is a computer composed of a CPU 301, memory 302, auxiliary storage device 303, communication interface 304, and display 307. The CPU 301, memory 302, auxiliary storage device 303, communication interface 304, and display 307 are connected by a common transmission path, which is a bus. Detailed explanations of components that overlap with those in Figure 4 are omitted.

[0035] The display 307 is a device that displays operation support images and may consist of multiple displays. Since the operation support image data received from the forklift 101 is encoded, the operation support image data received from the image processing unit 113 via the communication I / F 304 is decoded by a processing unit such as the CPU 301, and the display 307 displays the operation support image. The display device 201 may be connected to an external processing unit that performs decoding of the operation support image data. The display device 201 may also be equipped with an internal or external remote control device for the operator to input operation commands in order to remotely control the forklift 101. If an external remote control device is provided, the remote control device may communicate directly with the forklift 101, or if it is directly connected to the forklift 101, the display device 201 may communicate with the forklift 101 via the remote control device.

[0036] Figure 6 is a flowchart illustrating the operation support image data generation process performed by the operation support control unit 106 in this embodiment based on the vehicle status. This process is continuously and repeatedly executed at predetermined fixed intervals according to the frame rate of the camera 105, etc.

[0037] Step S1001: The image processing unit 113 acquires an image from the camera 105. Note that the image processing unit 113 may acquire images from only some of the connected cameras 105, rather than all of them. Step S1002: The determination unit 111 determines the vehicle status, such as the driving status or the loading status, in response to the image acquired in step S1001. Note that the determination unit 111 may determine the vehicle status at predetermined intervals. In this case, the decision unit 112, which will be described later, determines the screen configuration of the operation support image using the vehicle status most recently acquired by the image processing unit 113. The vehicle status determination process in the determination unit 111 will be described later in Figures 8 and 9.

[0038] Step S1003: The determination unit 112 determines the screen configuration of the operation support image based on the vehicle state determined by the determination unit 111 in step S1002. Specifically, the determination unit 112 retrieves the image configuration associated with the vehicle state determined by the determination unit 111 in step S1002 from the storage unit 114, which has pre-stored image configurations associated with vehicle states, and determines it as the screen configuration of the operation support image.

[0039] Figure 7 shows an example of an image configuration associated with the vehicle status. In this embodiment, the display unit 211 is composed of two displays, so an example of an image configuration when two operation support images are displayed on two screens is shown. Note that the image configuration shown in Figure 7 is just one example of a desirable configuration, and the screen configuration (screen arrangement and / or components) can be arbitrarily set by the system administrator or operator.

[0040] Figure 7(a) shows the image configuration of the operation support image (hereinafter also referred to as the driving image configuration) when the forklift 101 is driving without holding a load. The driving image configuration consists of images that need to be checked when driving, and comprises a first driving image 401-C1 configured by the image processing unit 113 using images acquired by the front cameras 105-5, 105-4, and 105-7 that capture images of the front in the direction of travel, and a second driving image 402-C1 configured by the image processing unit 113 using images acquired by the rear cameras 105-3, 105-2, and 105-1 that capture images of the rear in the direction of travel.

[0041] The configuration of the first driving image 401-C1 is such that the image from camera 105-5 (hereinafter also referred to as image 5) and the image from camera 105-4 (hereinafter also referred to as image 4) are placed side by side at the top, and the image from camera 105-7 (hereinafter also referred to as image 7) is placed at the bottom.

[0042] Furthermore, the second driving image 402-C1 is configured such that the image from camera 105-3 (hereinafter also referred to as image 3) is placed on the left half, and the image from camera 105-2 (hereinafter also referred to as image 2) and the image from camera 105-1 (hereinafter also referred to as image 1) are placed side by side on the right half. When the image is processed into processed image data by the image processing unit 113, the processed image data is placed in place of the image in the driving image configuration and each of the image configurations described later.

[0043] Next, Figure 7(b) shows the image configuration of the operation support image (hereinafter also referred to as the cargo handling driving image configuration) of the forklift 101 in a cargo handling driving state with a load. The cargo handling driving image configuration consists of images that need to be checked during cargo handling driving, and comprises a first cargo handling driving image 401-C3 configured by the image processing unit 113 using images acquired by front cameras 105-5, 105-4, and 105-6 that capture the front side in the direction of travel, and a second cargo handling driving image 402-C3 configured by the image processing unit 113 using images acquired by rear cameras 105-3, 105-2, and 105-1 that capture the rear side in the direction of travel.

[0044] The configuration of the first cargo handling driving image 401-C3 is such that images 5 and 4 are arranged side by side horizontally at the upper part, and an image of camera 105-6 (hereinafter also referred to as image 6) is arranged at the lower part. Different from the first driving image 401-C1, the first cargo handling driving image 401-C3 uses image 6 as a component instead of image 7 because the cargo handling driving state is driving while holding a load. In image 7, mainly only the load is shown, but in image 6, not only the load but also the vicinity of the load is shown. Also, the second cargo handling driving image 402-C3 has the same image and arrangement configuration as the second driving image 402-C1.

[0045] Next, FIG. 7(c) shows the image configuration of the operation support image in the loading state during the operation of the forklift 101 loading a load (hereinafter also referred to as the loading image configuration). The loading image configuration includes the first loading image 401-C2 configured by the image processing unit 113 from images obtained by the front cameras 105-5, 105-4, 105-7 that image the front side in the traveling direction, which are images necessary for confirmation during loading, and the second loading image 402-C2 composed of images obtained by the rear cameras 105-2, 105-1 that image the rear side in the traveling direction and the front lower left camera 105-6 that obliquely images the front lower side in the traveling direction.

[0046] The first loading image 401-C2 has the same image and arrangement configuration as the first driving image 401-C1 described above. The second loading image 402-C2 has a configuration in which in the second driving image 401-C1 described above, image 6 is arranged instead of image 3 in the left half. In the second loading image 402-C2, image 6 is arranged instead of image 3 because during loading, reverse movement is rare, and image 6 shows the vicinity of the load that needs to be confirmed.

[0047] Finally, FIG. 7(d) shows the image configuration of the operation support image in the unloading state during the unloading operation of the forklift 101 (hereinafter also referred to as the unloading-time image configuration). The unloading-time image configuration includes, as constituent elements, images that need to be confirmed during unloading, and the first unloading-time image 401-C4 configured by the image processing unit 113 from images acquired by the front cameras 105-5, 105-4, and 105-6 that image the front side in the traveling direction, and the second unloading-time image 402-C4 configured by the image processing unit 113 from images acquired by the rear cameras 105-2, 105-1 that image the rear side in the traveling direction, and the camera 105-7 at the front lower center in the traveling direction.

[0048] The first unloading-time image 401-C4 has the same image and arrangement configuration as the first cargo handling traveling-time image 401-C1 described above. The second unloading-time image 402-C4 has a configuration in which in the second traveling-time image 402-C1 described above, the left half is replaced with image 7 instead of image 3. In the second unloading-time image 402-C4, image 7 is arranged instead of image 3 because it is rare to reverse during unloading, and the vicinity of the load to be unloaded that needs to be confirmed is shown in image 7.

[0049] Returning to FIG. 6, Step S1004: The determination unit 112 determines the presence or absence and the content of processing of each image that is a constituent element of the operation support image based on the vehicle state determined in Step S1002. Specifically, the determination unit 112 obtains the processing content from the storage unit 114 that stores in advance the presence or absence and the content of processing associated with the vehicle state and / or the camera, and determines the processing content for the images associated with the vehicle state determined in Step S1002 and having processing. The processing content includes partial elimination of the image such as elimination of unnecessary reflections, enlargement of the location to be displayed in detail or reduction of the location to be displayed as a whole, mirror image processing for reversing the left and right of the image, etc.

[0050] Step S1005: The image processing unit 113 processes the image whose processing content was determined in step S1004 according to the processing content to generate processed image data. Step S1006: The image processing unit 113 integrates the processed image data of the image processed in step S1005, and the images that were not processed in step S1005, based on the screen configuration determined in step S1003, to generate operation support image data. Step S1007: The image processing unit 113 generates additional information based on the analysis results obtained by analyzing measurement information and images acquired from the drive system 102 and sensors 103, and adds it to the operation support image data generated in step S1006. The additional information to be generated may be determined according to the vehicle status. In addition, the additional information that the image processing unit 113 adds to the operation support image data can be arbitrarily set by the operator before the start of forklift operation, etc., but after the start of operation, the operator can switch the display of the added additional information ON / OFF.

[0051] The additional information generated by the image processing unit 113 can be broadly divided into two categories: additional machine information relating to the machine's status and additional warning information that notifies the operator of warnings. Additional machine information includes machine speed, battery (or fuel) level, fork side shift, fork height, reach out OK, fork insertion OK, load loading OK, tilt OK, etc. Additional warning information includes warning notifications for obstacles or people near the forklift 101, and warning notifications for the height of the forklift's forks, etc.

[0052] First, let's explain the additional aircraft information with an example. For example, the image processing unit 113 acquires the aircraft speed from the sensor 103, generates a digital display or speedometer icon showing the acquired aircraft speed as additional aircraft information, and superimposes it onto the operation support image data. The image processing unit 113 may generate additional aircraft information regarding aircraft speed only when the vehicle is in motion or during cargo handling. In addition, the image processing unit 113 acquires the battery (or fuel) level from the sensor 103, generates a digital display or battery (or fuel) level icon showing the acquired battery (or fuel) level as additional aircraft information, and superimposes it onto the operation support image data.

[0053] Furthermore, the image processing unit 113 acquires the left-right position of the claw from the sensor 103, calculates the amount of side shift of the claw based on the acquired left-right position of the claw, generates a digital display or gauge icon indicating the calculated side shift amount as additional aircraft information, and superimposes it onto the operation support image data. The position where the image processing unit 113 superimposes this additional aircraft information is arbitrary, and the same applies to other icons. In addition, the image processing unit 113 acquires the height of the claw (mast) from the sensor 103, generates a digital display or gauge icon indicating the height of the claw (mast) as additional aircraft information, and superimposes it onto the operation support image data.

[0054] Furthermore, the image processing unit 113 acquires the reach-out amount from the sensor 103, and if the acquired reach-out amount is greater than or equal to a predetermined value, it generates an icon indicating reach-out as additional machine information and superimposes it on the operation support image data. The image processing unit 113 may generate a chromatic icon, such as blue or red, indicating reach-out when the reach-out amount is greater than or equal to a predetermined value, and an achromatic icon, such as gray, indicating reach-out when the reach-out amount is less than a predetermined value, as additional machine information. The image processing unit 113 may generate additional machine information regarding reach-out only when the vehicle state is loading or unloading. In addition, the image processing unit 113 acquires the ON / OFF status of the load sensor (depth) from the sensor 103, and if it is "ON", it generates an icon indicating claw insertion as additional machine information and superimposes it on the operation support image data. The image processing unit 113 may generate a chromatic icon indicating claw insertion when the load sensor (depth) is "ON", and an achromatic icon indicating claw insertion when it is "OFF", as additional machine information. The image processing unit 113 may generate additional machine information related to claw insertion only when the vehicle state is loading or unloading.

[0055] Furthermore, the image processing unit 113 obtains the ON / OFF status of the load sensor (depth / up / down) from the sensor 103, and if it is "ON", it generates an icon indicating loading as additional machine information and superimposes it on the operation support image data. The image processing unit 113 may also generate a colored loading icon as additional machine information when the load sensor (depth / up / down) is "ON", and a colorless loading icon when it is "OFF". The image processing unit 113 may also generate additional machine information related to loading only when the vehicle state is loading or unloading. Note that the ON / OFF status of the load sensor (depth / up / down) is "ON" when both the load sensor (depth) and the load sensor (up / down) are "ON", and "OFF" otherwise. In addition, the image processing unit 113 obtains the tilt angle from the sensor 103, and if the obtained tilt angle is greater than or equal to a predetermined value, it generates an icon indicating tilt as additional machine information and superimposes it on the operation support image data. Furthermore, the image processing unit 113 may generate a colored icon indicating tilt when the acquired tilt angle is greater than or equal to a predetermined value, and an achromatic icon indicating tilt when the acquired tilt angle is less than the predetermined value, as additional aircraft information. The image processing unit 113 may also generate additional aircraft information regarding tilt only when the vehicle state is loading or unloading.

[0056] Next, the warning information will be explained with an example. The image processing unit 113 uses deep learning or the like based on the image captured by the camera to detect obstacles and / or people in the surrounding area and in the direction of travel. The image processing unit 113 then generates warning information to draw attention to the detected obstacles and / or people, such as a warning icon with an exclamation mark inside a yellow triangle, or a border around the obstacles and / or people, and superimposes it onto the operation support image data. Note that the aircraft-related information is not limited to the warning icon and border superimposed on the operation support image data; it may also include information that instructs the operation support image data to flash or information that instructs the output of a warning sound. The image processing unit 113 may generate warning information regarding obstacles and / or people only when the vehicle is in motion or during cargo handling.

[0057] The image processing unit 113 issues a warning about the height of the forklift attachment or mast if the height deviates from a predetermined value. Specifically, it obtains the height of the forks (mast) from the sensor 103, and if the obtained height of the forks (mast) is above the upper limit, it issues a warning about the height of the forks, for example, by generating a warning icon with "STOP" or "Caution: Ceiling" written on it as additional warning information. The image processing unit 113 also obtains the height of the forks (mast) from the sensor 103, and if the obtained height of the forks (mast) is below the lower limit, it issues a warning about the height of the forks, for example, by generating a warning icon with "STOP" written on it as additional warning information. The image processing unit 113 may also issue information that instructs the flashing of an operation support image or the output of a warning sound, rather than just a warning icon. The upper and lower limits of the fork height are set in advance by the operator or the like and stored in the storage unit 114. The image processing unit 113 may generate additional warning information regarding the height of the claws only when the vehicle is loading or unloading.

[0058] Returning to Figure 6, Step S1008: The image processing unit 113 performs encoding on the operation support image data to which additional information was added in step S1007. This makes it possible to reduce the amount of communication resources used when transmitting images. Step S1009: The image processing unit 113 transmits the operation support image data after the encoding process in step S1008 to the display unit 211.

[0059] Figure 8 is a flowchart showing the process for determining the vehicle state in the determination unit 111 in this embodiment. The determination unit 111 starts processing at predetermined intervals or in response to requests from the image processing unit 113. In the case of predetermined intervals, it is desirable that processing be performed at sufficiently short intervals so that the determination unit 111 can always determine the latest vehicle state.

[0060] Step S1101: The determination unit 111 acquires status information from the drive system 102 and the sensor 103. The status information acquired in this step includes at least the ON / OFF status of the load from the load sensor (depth / up / down) and the mast angle from the instrument, and in the case of a reach forklift, the reach-out amount is also included.

[0061] Step S1102: The determination unit 111 determines the vehicle state based on the state information obtained in step S1101 and the vehicle state transition model. The vehicle state transition model is an information model consisting of vehicle states and transition conditions between vehicle states. An example of the vehicle state transition model in this embodiment is shown in Figure 9.

[0062] In this embodiment, the vehicle state transition model comprises four states: driving state C1, loading state C2, loading / unloading driving state C3, and unloading state C4. The initial state is driving state C1. Each state can transition according to predetermined transition conditions, as indicated by the arrows.

[0063] The determination unit 111 determines whether the condition for transitioning from the current vehicle state to the next vehicle state is met by the state information obtained in step S1101, and if it is met, it determines that the vehicle has transitioned to the next vehicle state.

[0064] For example, in the case of a reach forklift, if the current state is driving state C1, the vehicle state transitions to loading state C2 when the forks reach out. That is, if the current state is driving state C1, the determination unit 111 determines that the vehicle state has transitioned to the next state, loading state, if the reach-out amount of the state information acquired in step S1102 is greater than 0. If the reach-out amount of the state information acquired in step S1101 is 0 or less, the vehicle remains in the current driving state C1.

[0065] In addition to the configuration shown in Figure 9, the determination unit 111 may also manage other vehicle states such as emergency stop state, high-speed driving state, low-speed driving state, and person detection state. The initial state does not have to be driving state C1. The arrows representing the transitionable states may be connected to multiple vehicle states, such as an arrow further connected from the loading state C2 towards the driving state C1.

[0066] Figure 10 is an example of the screen of the display unit showing operation support images during driving in this embodiment. The operation support image data shown in the figure consists of a first driving image 401-C1 consisting of forward images 4, 5, and 7, and a second driving image 402-C1 consisting of rear images 1, 2, and 3, based on the driving image configuration.

[0067] In the driving image 401-C1, icons I1 are superimposed, indicating reach-out OK, fork insertion OK, cargo loading OK, and tilt OK in gray; in other words, icons meaning no reach-out, no fork insertion, no cargo loading, and no tilt. Also, in the driving image 401-C1, a scale icon I2 indicating the amount of side shift is superimposed. Furthermore, since a person is visible in image 4 of the driving image 401-C1, a frame I3 surrounding the person and a warning icon I4 with an exclamation mark inside a triangle are superimposed. A warning sound may also be added to the driving image 401-C1 and outputted. In the driving image 402-C1, a battery (or fuel) level icon I5 indicating the remaining battery (or fuel) level and a speedometer icon I6 indicating the vehicle speed are superimposed. The superimposed positions of the icons can be arbitrarily set by the system administrator or operator, but may also be determined according to the vehicle status. Thus, the operation support screen data has a screen configuration that corresponds to the vehicle status, and the operator's visibility is enhanced by the overlaying of various icons and the addition of sound.

[0068] As described above, the forklift operation support system in this embodiment integrates two or more images captured by a camera mounted on the forklift according to the vehicle status, generates operation support image data with a smaller data capacity than the two or more original image data, encodes the generated operation support image data, and transmits it to a display device. This reduces the consumption of communication resources compared to transmitting two or more captured image data to the display device, preventing a shortage of communication resources, and allows for simultaneous viewing of multiple images. Therefore, the forklift operation support system in this embodiment can provide operation support images that make it easier for the forklift operator to operate without reducing work efficiency.

[0069] Furthermore, the operation support image data is configured such that the image composition, whether or not the images of the constituent elements are processed, and / or the content of the image processing are determined according to the vehicle condition, making it easier for the operator to check different operational checkpoints depending on the vehicle condition. In addition, the operation support image data can be supplemented with additional machine information (e.g., reach-out) that the operator cannot confirm by visual inspection or operation support image data alone, and with additional warning information that notifies the operator of hazards that could lead to accidents if overlooked. These features further improve the visibility of the operation support image data. Therefore, the forklift operation support system in this embodiment can provide the forklift operator with operation support image data that is appropriate to the vehicle condition, whether the forklift is operated while riding on the actual machine or remotely, without reducing work efficiency and with improved visibility.

[0070] <Second Embodiment> The second embodiment includes a function to monitor whether operation support image data, generated by integrating two or more images according to the vehicle status, is being transmitted correctly to the display device. This function makes it easy for the operator to determine whether to stop the operation due to a system failure or a serious deterioration in communication quality, or to continue the operation due to a temporary problem, when remotely operating a forklift in a warehouse environment or the like where wireless communication quality fluctuates frequently, if the display unit 211 displays an operation support image with some data missing. Note that the same reference numerals are used for the same configurations and processes as in the first embodiment, and their explanations are omitted.

[0071] Figure 11 is a block diagram showing an example of the software configuration in the forklift operation support system 20 in this embodiment. In this embodiment, the operation support control unit 126 of the forklift 121 is equipped with a communication unit 127-A instead of a communication unit 107-A. The display device 202 is equipped with a communication unit 127-B instead of a communication unit 107-B, and further includes a communication monitoring unit 212.

[0072] Communication units 127-A and 127-B measure the communication quality between communication unit 127-A and communication unit 127-B, respectively. Communication quality includes latency, throughput, traffic volume, packet loss rate, and radio signal strength, and is measured by methods such as monitoring IP (Internet Protocol) packets passing through the communication I / F 304, and sending and receiving measurement IP (Internet Protocol) packets containing timestamps and sequence numbers between communication unit 127-A and communication unit 127-B.

[0073] The communication monitoring unit 212 collects the communication quality measured by the communication units 127-A and 127-B, analyzes the collected communication quality, and notifies the operator of the analysis results. The analysis results are notified to the operator in a visualized form, as shown in Figure 12. In addition, the communication monitoring unit 212 may notify the operator of the collected communication quality all at once so that the operator can easily understand the situation in detail. Furthermore, the communication monitoring unit 212 may notify the operator of the analysis results by sound instead of visualized information.

[0074] Figure 13 is a flowchart of the communication analysis process performed by the communication monitoring unit 212 in this embodiment. The communication monitoring unit 212 repeatedly performs this process at predetermined intervals and sequentially provides the operator with the communication analysis results regarding whether the operation support image data is being transmitted correctly.

[0075] Step S1201: The communication monitoring unit 212 acquires the vehicle status from the determination unit 111 via the communication units 127-A and 127-B. Step S1202: The communication quality is acquired from the communication units 127 (127-A, 127-B) of both the operation support control unit 126 and the display device 202. Alternatively, the measurement results of the communication quality may be aggregated in advance in one of the communication units 127-B, etc., and the results may be acquired.

[0076] Step S1203: Based on the vehicle status acquired in step S1201, the communication monitoring unit 212 performs communication analysis processing on the communication quality acquired in step S1202 and acquires the communication analysis results. The acquired analysis results are shown in Figure 12 above and may or may not be transmitted to the display device 202 and displayed on the display unit 211.

[0077] Step S1204: The communication monitoring unit 212 determines, based on the communication analysis results obtained in step S1203, whether the communication quality is showing an abnormal value, that is, whether a system failure or a serious deterioration in communication quality has occurred. For example, the communication monitoring unit 212 stores the communication analysis results under normal conditions in advance, and if the error between the communication analysis results obtained in step S1203 and the communication analysis results under normal conditions becomes large, it determines that there is a system failure or a serious deterioration in communication quality.

[0078] The abnormal values ​​for communication quality used for the determination are created in advance by the system administrator or other relevant personnel for each vehicle state. For example, in driving state M1201, an abnormality is determined if the traffic volume falls outside 10% of 10 Mbps, and in loading state M1202, an abnormality is determined if the delay is greater than 100 ms or the packet loss rate is greater than 0.01%. Note that the determination of abnormal communication quality may also be processed using statistical methods such as machine learning or neural networks. If the communication monitoring unit 212 determines that the communication quality is showing an abnormal value, it proceeds to step S1205, and if it determines that the communication quality is not showing an abnormal value, it terminates the process.

[0079] Step S1205: The communication monitoring unit 212 transmits a message to the display device 202 indicating that the communication quality is abnormal, and notifies the operator by displaying it on the display unit 211, etc.

[0080] As described above, the forklift operation support system in this embodiment monitors and analyzes communication between the forklift and the display device, and can notify the operator whether a system failure or a serious deterioration in communication quality has occurred when distortion of the operation support image occurs. This makes it easier for the operator to decide whether to continue or stop the operation, thereby preventing a decrease in safety and unnecessary work stoppages.

[0081] Furthermore, since the operation support image data transmitted to the display device 202 differs depending on the vehicle status, determining whether a system failure or a serious deterioration in communication quality has occurred can be performed according to the vehicle status, thereby improving the accuracy of the determination. In addition, the operator can understand the current communication quality by being presented with the communication analysis results.

[0082] 10, 20... Remote control system, 101, 121... Forklift, 201, 202... Display device, 103... Sensor, 105... Camera, 111... Judgment unit, 112... Decision unit, 113... Image processing unit, 107, 127... Communication unit, 106, 126... Operation support control unit, 211... Display unit, 212... Communication monitoring unit

Claims

1. A forklift operation support system comprising a forklift capable of communicating with each other, and a display device that displays operation support images to assist in the operation of the forklift, wherein the forklift comprises two or more cameras that capture images of the area around the forklift, a determination unit that acquires status information of the forklift and determines the vehicle status, a determination unit that determines the image configuration of the operation support image based on the determined vehicle status of the forklift, and an image processing unit that integrates the two or more images based on the determined screen configuration to generate the operation support image data, and the display device receives the generated operation support image data and displays the operation support image.

2. The forklift operation support system according to claim 1, characterized in that the forklift is equipped with sensors that acquire status information relating to information of each part of the forklift, and the vehicle status is a driving state, a loading driving state, a loading state, and an unloading state.

3. The forklift operation support system according to claim 2, characterized in that the vehicle state transitions in the order of driving state, loading state, cargo handling driving state, and unloading state, and the unloading state returns to the driving state, the transition from the driving state to the loading state occurs when the reach-out amount of the state information satisfies a predetermined condition, the transition from the loading state to the cargo handling driving state, and from the cargo handling driving state to the unloading state occurs when the mast angle of the state information satisfies a predetermined condition, and the transition from the unloading state to the driving state occurs when the reach-out amount and the value of the load sensor of the state information satisfy a predetermined condition.

4. The forklift operation support system according to claim 1, wherein the determination unit determines whether or not to process the image and the content of the processing based on the determined vehicle state of the forklift, and the image processing unit integrates the processed image data obtained by processing the image determined to have been processed based on the determined processing content and the image determined not to have been processed, based on the determined screen configuration, to generate the operation support image data.

5. The forklift operation support system according to claim 1, wherein the image processing unit adds machine additional information, which is information regarding the state of the forklift machine, to the operation support image data.

6. The forklift operation support system according to claim 1, wherein the image processing unit adds warning information that warns that there is an obstacle in the direction of travel of the forklift and / or that the height of the forklift attachment or mast has deviated from a predetermined value.

7. The forklift operation support system according to claim 1, wherein the image processing unit encodes the generated operation support image data and transmits it to the display device via the communication unit, and the display device decodes and displays the encoded operation support image data upon receiving it.

8. The forklift operation support system according to claim 1, wherein the display device comprises a communication monitoring unit that measures the communication quality between the forklift and the display device, analyzes whether the operation support image data is being transmitted normally based on the measured communication quality and the vehicle status, and outputs the analysis result.

9. A method for a forklift operation support system comprising forklifts capable of communicating with each other and a display device that displays operation support images to assist in the operation of the forklifts, the method comprising: two or more cameras provided on the forklifts capturing images of the area around the forklifts; acquiring status information of the forklifts and determining the vehicle status; determining the image configuration of the operation support images based on the determined vehicle status of the forklifts; integrating the two or more images based on the determined screen configuration to generate the operation support image data; and the display device receiving the generated operation support image data and displaying the operation support images.

10. A forklift operation support system comprising forklifts capable of communicating with each other and a display device that displays operation support images to assist in the operation of the forklifts, wherein the system functions as: a determination unit that acquires status information of the forklifts and determines the vehicle status; a determination unit that determines the image configuration of the operation support images based on the determined vehicle status of the forklifts; and an image processing unit that integrates two or more images of the forklifts' surroundings captured by two or more cameras on the forklifts based on the determined screen configuration to generate the operation support image data, wherein the display device receives the generated operation support image data and displays the operation support image.

Images