Display device, unloading assistance system, industrial vehicle, control method, and computer-readable recording medium

Abstract

Technical Problem The present application provides a display device capable of confirming a place where a cargo transported by an industrial vehicle is placed in advance when the cargo is unloaded, an unloading assistance system, an industrial vehicle, a control method, and a computer-readable recording medium. Solution The display device of the present application is a display device worn on a head of a person, and includes a see-through display portion capable of displaying image data, a target position acquisition portion that acquires position information of a delivery destination of a cargo transported by an industrial vehicle, a position information acquisition portion that acquires position information of the industrial vehicle, and a control portion that displays image data including a target setting range of the cargo and a predicted setting range of the cargo in such a manner that the target setting range included in the image data overlaps with a target setting range in a real world that can be seen through the display portion, wherein the target setting range of the cargo is a target setting range of the cargo based on information of a positional relationship between a person who gets on the industrial vehicle and the industrial vehicle and the position information of the industrial vehicle and the delivery destination, and the predicted setting range of the cargo is a predicted setting range when the cargo is unloaded from the industrial vehicle.

Description

Technical Field

[0001] This disclosure relates to display devices, unloading assistance systems, industrial vehicles, control methods, and procedures. Background Technology

[0002] When transporting containers using forklifts, the view ahead is often obstructed. Therefore, when loading container trucks, the cargo is raised to a position where the view is clear and the truck is close to the unloading point. However, with the cargo raised, it is difficult to determine the correct placement and angle for unloading, making it challenging to accurately place the cargo at the intended location.

[0003] As a related technology, Patent Document 1 discloses a system comprising: a camera for capturing images of the front of a forklift; and a display unit for displaying an image of the image captured by the camera, in which guidance information indicating the distance and height to the location of the object being loaded or unloaded is superimposed on the image.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-20793 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] The aim is to find a technology that allows the location of goods to be placed in advance when unloading goods, regardless of the height of the goods being transported.

[0009] This disclosure provides a display device, unloading assistance system, industrial vehicle, control method, and program that can solve the above problems.

[0010] Technical solution

[0011] The display device disclosed herein is a head-mounted display device comprising: a see-through display unit capable of displaying image data; a target position acquisition unit acquiring target position information as the location information of the destination of goods transported by an industrial vehicle; a position information acquisition unit acquiring vehicle position information as the location information of the industrial vehicle or the person riding in the industrial vehicle; and a control unit that displays the image data, including the target setting range and the desired setting range of the goods, in a manner that overlaps the target setting range included in the image data with the target setting range in the real world visible through the display unit. The target setting range of the goods is the target setting range of the goods as seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle position information, and the target position information. The desired setting range of the goods is the desired setting range of the goods as seen by the person when unloading the goods from the industrial vehicle, based on information about the positional relationship, information about the position of the goods maintained by the industrial vehicle, and information about the size of the goods.

[0012] The unloading assistance system disclosed herein includes: a positioning device for locating the location information of the industrial vehicle or the person riding in the industrial vehicle; and the aforementioned display device.

[0013] The industrial vehicle disclosed herein is equipped with the aforementioned unloading auxiliary system.

[0014] The control method disclosed herein is a control method for a display device worn on a person's head, having a perspective display section capable of displaying image data. The control method includes the following steps: acquiring target location information as location information of the destination of goods transported by an industrial vehicle; acquiring vehicle location information as location information of the industrial vehicle or the person riding in the industrial vehicle; and displaying the image data, including a target setting range and a desired setting range of the goods, in a manner that the target setting range included in the image data overlaps with the target setting range in the real world visible through the display section. The target setting range of the goods is the target setting range of the goods as seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle location information, and the target location information. The desired setting range of the goods is the desired setting range of the goods as seen by the person when the goods are unloaded from the industrial vehicle, based on information about the positional relationship, information about the position of the goods maintained by the industrial vehicle, and information about the size of the goods.

[0015] Furthermore, the program of this disclosure enables the computer of the display device to function as the following unit: the display device is a head-mounted display device having a perspective display section capable of displaying image data; a unit for acquiring target location information as location information of the destination of goods transported by an industrial vehicle; a unit for acquiring vehicle location information as location information of the industrial vehicle or the person riding in the industrial vehicle; and a unit for displaying the image data including the target setting range and the intended setting range of the goods in such a way that the target setting range included in the image data overlaps with the target setting range in the real world visible through the display section, wherein the target setting range of the goods is the target setting range of the goods seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle location information, and the target location information, and the intended setting range of the goods is the intended setting range of the goods seen by the person when the goods are unloaded from the industrial vehicle based on information about the positional relationship, information about the position of the goods held by the industrial vehicle, and information about the size of the goods.

[0016] Beneficial effects

[0017] Based on the aforementioned display device, unloading assistance system, industrial vehicle, control method, and procedure, the location for placing the goods can be determined in advance when unloading goods transported by the industrial vehicle. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating one example of a forklift implementation.

[0019] Figure 2 This is a block diagram illustrating an example of a display device for an implementation method.

[0020] Figure 3A The first figure illustrates the display of the intended setting range and the target setting range of the implementation method.

[0021] Figure 3B The second figure illustrates the display of the intended setting range and the target setting range of the implementation method.

[0022] Figure 4 This is a diagram showing an example of the intended setting range and the target setting range of the implementation method.

[0023] Figure 5 This diagram illustrates the overlap between the intended setting range and the target setting range of the implementation method.

[0024] Figure 6This is a diagram showing an example of a display of the target location for a guided implementation method.

[0025] Figure 7A The first figure shows an example of container loading in an implementation scheme.

[0026] Figure 7B The second figure shows an example of container loading in an implementation scheme.

[0027] Figure 8 This is a flowchart illustrating an example of the display processing of an implementation method.

[0028] Figure 9 This diagram illustrates the alignment of the virtual space of the implementation site with the real world.

[0029] Figure 10 This is a diagram illustrating an example of the hardware configuration of an unloading assistance system according to an implementation method. Detailed Implementation

[0030] <Implementation Method>

[0031] The unloading auxiliary system of this disclosure will now be described with reference to the accompanying drawings.

[0032] (constitute)

[0033] Figure 1 This is a schematic diagram illustrating one example of a forklift in various implementations.

[0034] Forklift 1 includes a body 10, a loading / unloading device 11, a route calculation device 16, and a display device 20. The body 10 has a drive unit and a steering unit (not shown) for driving the forklift 1. The loading / unloading device 11 includes forks 12 and a mast 13 for raising and lowering the forks 12, performing loading and unloading actions such as raising and lowering goods loaded on the forks 12. A tilt sensor 14 is provided on the mast 13 to detect the tilt angle of the mast 13. A load sensor 15 is provided on the mast 13 to detect the weight of the goods loaded on the forks 12. The values ​​measured by the tilt sensor 14 and the load sensor 15 are sent to the display device 20. The body 10, etc., are equipped with a GNSS 30 as a GNSS (Global Navigation Satellite System) receiver (mobile station) for locating the forklift 1. For example, GNSS 30 ( Figure 1 GNSS30(30a) can be installed at any location on the vehicle body 10. Figure 1GNSS 30a (30b) can also be installed on the helmet worn by the driver 100, etc. Either GNSS 30a or 30b is sufficient. Hereinafter, as an example, GNSS 30a is mounted on the vehicle body 10. The location information located by GNSS 30a is sent to the display device 20. The display device 20 is AR glasses or MR (Mixed Reality) glasses. The driver 100 wears the display device 20 to drive the forklift 1. In this specification, as... Figure 1 As shown, the direction in which the loading and unloading device 11 with the forklift 1 is installed is called the front, and the opposite direction is called the rear. The left and right directions when facing the front are called the left and right sides, respectively. In addition, a management system 40 for managing the containers and other goods that are loaded and unloaded by the forklift 1 is set up in the office, data center, etc. at the site where the loading and unloading operation is carried out.

[0035] Figure 2 This is a block diagram illustrating an example of an unloading assistance system implemented in this way.

[0036] The unloading assistance system 200 includes a tilt sensor 14, a load sensor 15, a display device 20, a GNSS 30, and a management system 40. The tilt sensor 14, load sensor 15, and GNSS 30 are communicatively connected to the display device 20. The display device 20 is communicatively connected to the management system 40. The management system 40 is configured as a computer. The display device 20 is configured as AR glasses or MR glasses (hereinafter, both are referred to as AR glasses) including a display device and a computer.

[0037] (The structure of the management system)

[0038] The management system 40 manages the dimensions of each container for each loading and unloading operation, as well as the location information of the target location where the container should be placed. For example, when the forklift 1 is transporting a container, the management system 40 sends the location information of the target location, which is the destination of the container, to the display device 20. The target location may be, for example, the location information of the four corners of the rectangular area where the container should be placed, or information determining the center of gravity of the rectangular area and the direction and angle at which the container should be placed. In addition, the management system 40 may also send information such as the length, width, and height of the container being transported, along with the target location, to the display device 20.

[0039] (Composition of the display device)

[0040] The display device 20 includes a sensor unit 21, a display unit 22, a transceiver unit 23, a control unit 24, a display determination unit 25, a target position acquisition unit 26, a cargo size acquisition unit 27, and a storage unit 28.

[0041] The sensor unit 21 includes sensors for head tracking of the driver 100 (e.g., multiple visible light cameras, IMU), sensors for eye tracking of the driver 100 (e.g., infrared camera), depth sensor, microphone, etc.

[0042] Display unit 22 constitutes the lens (screen) of the AR glasses, for example, it is composed of a see-through holographic display (allowing the viewer to see the outside world through the lens). When virtual objects are not projected onto the lens, the driver 100 can see the real world as it is. Virtual objects refer to image data projected onto display unit 22. When virtual objects are displayed on display unit 22, the driver 100 can see the virtual objects superimposed on the real world.

[0043] The transceiver unit 23 transmits and receives data from other devices. For example, the transceiver unit 23 receives values ​​measured by the tilt sensor 14, load sensor 15, and GNSS 30 via wireless communication such as WiFi (registered trademark) and Bluetooth (registered trademark). In addition, the transceiver unit 23 obtains information such as the target location where the container should be placed from the management system 40.

[0044] The control unit 24 performs display control on the display unit 22, etc. Based on information detected by the sensor unit 21, the control unit 24 identifies the spatial shape surrounding the driver 100, such as identifying the driver 100's position in the driver's seat or the driver 100's posture. The control unit 24 generates a virtual space (vehicle virtual space) corresponding to the space around the forklift 1 based on the forklift 1's 3D CAD information, and generates a virtual space (site virtual space) outside the space of the forklift 1 and its surroundings based on pre-generated 3D map information of the loading and unloading area. The control unit 24 performs calibration, aligning the real world with the vehicle virtual space and the site virtual space. After alignment, the control unit 24 places virtual objects at desired positions in the vehicle virtual space. Thus, the driver 100 is displayed in a manner that the virtual objects actually exist in the real world at the desired positions in the vehicle virtual space. For example, a virtual object represents the "intended setting range" of the area occupied by a container in the case that it is assumed to be unloaded vertically and placed on the ground while still in its original position during transport. The intended setting range is generated, for example, as a thin cube-shaped virtual object. The intended setting range can be displayed at any position relative to the container loaded by the forks 12 (e.g., near the bottom of the container, between the bottom of the container and the ground, etc.). Furthermore, even if there is an obstacle obstructing the view between the driver 100 and the virtual object, the intended setting range can be displayed in a way that the intended setting range can be seen through the obstacle. If there is an obstacle obstructing the view between the driver 100 and the intended setting range, the intended setting range can also be displayed from the driver 100's perspective as if the intended setting range exists on the opposite side of the obstacle (obstruction). In addition, the control unit 24 positions the virtual object at the desired location in the site virtual space. In this way, the driver 100 is displayed in a way that the virtual object actually exists at a real-world location corresponding to the desired location in the site virtual space. For example, a virtual object is a "target setting range" representing the area where a container in transit should be placed. The target setting range can be generated, for example, as a thin cubic virtual object. For example, the virtual object "target setting range" can be displayed at any position based on the target position of the container. The target setting range can also be displayed preferentially through obstacles between the driver 100 and the target position, or displayed by obscuring the target setting range through obstacles. Furthermore, the control unit 24 calculates the driver 100's posture based on information detected by the sensor unit 21, such as the head position when bending over or leaning out of the driver's seat to the left or right.Then, the control unit 24 generates virtual objects based on the positional relationship between the driver 100's head and the virtual objects, making the expected setting range and target setting range appear in the same way as when they actually exist in the real world, and configures them in the virtual space. Thus, even when the driver 100 is performing loading and unloading operations, for example, leaning out of the driver's seat to the right, the "expected setting range" and "target setting range" are displayed in a way that can be seen from that direction.

[0045] AR glasses feature the ability to freely configure virtual objects, including a "pre-defined setting range" in the desired location within the vehicle's virtual space and a "target setting range" in the desired location within the virtual arena's virtual space. This allows the virtual objects to appear as if they exist in their respective real-world locations, and their appearance changes according to the driver's posture. The user sets the desired virtual objects on the display device 20 (the developer uses AR development tools to create the virtual objects and installs the developed program on the display device 20). The control unit 24 then configures the virtual objects at the desired locations according to these settings. In this embodiment, the control unit 24 generates a "pre-defined setting range" and a "target setting range," which are then configured in the virtual space. For example, in the case of a "pre-defined setting range," such as... Figure 3A As shown, the control unit 24 generates a pre-defined setting range 301 based on the dimensions (length and width) of the container 300 acquired by the cargo size acquisition unit 27. For the X-axis of the XY plane in the vehicle virtual space, the center L1 of the left and right forks 12 and the center L2 of the pre-defined setting range in the longitudinal direction are aligned. For the Y-axis, the front edge of the pre-defined setting range, as viewed from the driver 100, is positioned at a predetermined distance L3 from the driver 100. Furthermore, for the Z-axis of the XYZ space in the vehicle virtual space, the control unit 24... Figure 3B The position configuration in the vehicle's virtual space at a specified height H1 from the ground is set within a predetermined range of 301.

[0046] Furthermore, for example, in the case of a "target setting range", the control unit 24 generates a target setting range 302 based on the dimensions (length and width) of the container 300 obtained by the cargo size acquisition unit 27, such as... Figure 3A As shown, for the XY plane in the virtual space of the site, the target setting range 302 is configured, for example, according to the range determined by the target location specified from the management system 40; for the Z-axis, as... Figure 3B As shown, the target setting range 302 is configured at the height where the bottom surface of the target setting range 302 contacts the ground in the virtual space of the site.

[0047] Thus, in Figure 4The diagram illustrates an example (illustrated diagram) of the visual information that driver 100 can see through display device 20 when the "expected setting range" and "target setting range" are respectively configured in the vehicle virtual space and the site virtual space. Driver 100 can confirm the positional relationship between the expected setting range 301 (where the container 300 is unloaded from the ground while maintaining its current vehicle position) and the target setting range 302 (where the container 300 should be unloaded), and manipulate the forklift 1 to place the container 300 in a manner that overlaps the two. For example, if the expected setting range 301 and the target setting range 302 are... Figure 4 As illustrated in the example, the driver 100 can steer the container 300 along the... Figure 5 The forklift 1 is operated by moving in the direction of the arrow shown. Furthermore, the control unit 24 displays the intended setting range 301 and the target setting range 302, as well as their overlapping areas, in a manner that makes them easy to visually understand. For example, the control unit 24 can be configured to display the target setting range 302 using a blue system color such as sky blue, the intended setting range 301 using a red system color such as pink, and the overlapping area using a light purple or similar color. This allows the operator 100 to easily grasp the overlap between the intended setting range 301 and the target setting range 302.

[0048] Furthermore, when transporting container 300, the gantry 13 is sometimes tilted in the forward and backward direction, but the control unit 24 can also be configured to display the desired setting range when the bottom surface of container 300 is parallel to the ground. Alternatively, the control unit 24 can also change the color of the desired setting range when the bottom surface of container 300 is parallel to the ground. Thus, the driver 100 can place container 300 while confirming that it is not tilted. The parallelism of the bottom surface of container 300 to the ground can be determined by the control unit 24, for example, based on the value detected by the tilt sensor 14 installed on the gantry 13.

[0049] Furthermore, for example, when the forklift 1 is far from the target location, since the display of the intended setting range 301 would become an obstacle, the control unit 24 may not display it, but only display the target location as a guide. For example, the control unit 24 may configure a virtual object 303 as a marker at a position corresponding to or above the target location in the virtual space of the site, or it may configure the target setting range 302. Figure 6 The image shows a display example of target setting range 302 and virtual object 303. Figure 6The image shown is an example of visual information that the driver 100 can see through the display unit 22. The driver 100 can use the target setting range 302 and the virtual object 303 as markers to guide the forklift 1 towards the target position. In this way, the control unit 24 switches between displaying and not displaying the virtual objects shown on the display unit 22. For example, in a scenario where the expected setting range 301 and the target setting range 302 are obstacles, they can be set to not be displayed.

[0050] The display determination unit 25 determines whether to display or not display a virtual object. When the display determination unit 25 determines that a certain virtual object should be displayed, the control unit 24 displays (configures) the virtual object; when the display determination unit 25 determines that it should not be displayed, the control unit 24 sets the virtual object to not be displayed (does not configure the virtual object in the virtual space). Hereinafter, an example of the determination process of the display determination unit 25 will be described.

[0051] (A) The display determination unit 25 acquires information detected by the load sensor 15 (the weight of the cargo loaded on the forks 12) through the transceiver unit 23. When the weight detected by the load sensor 15 is above a predetermined threshold (that is, when holding the container 300), the display determination unit 25 determines to display the intended setting range 301, the target setting range 302, and the virtual object 303 guiding the target position. When the weight is less than the threshold, it determines to set the intended setting range 301 to not be displayed. In the case of a weight less than the threshold, the display determination unit 25 may determine to set the target setting range 302 and the virtual object 303 to not be displayed.

[0052] (B) The display determination unit 25 obtains the current position information of the forklift 1 located by the GNSS 30a through the transceiver unit 23. The display determination unit 25 calculates the distance from the current position of the forklift 1 to the target position. When the calculated distance is above a predetermined threshold, it determines that the expected setting range 301 or the expected setting range 301 and the target setting range 302 should not be displayed (for example, display the virtual object 303). When the distance is less than the threshold, it determines that the expected setting range 301 and the target setting range 302 should be displayed, and the virtual object 303 should not be displayed.

[0053] (C) The control unit 24 displays an operation panel (virtual object) in the vehicle's virtual space, for example, based on the driver 100's gestures, voice, etc. This operation panel is equipped with a virtual switch that toggles the display on and off for each virtual object. When the driver 100 instructs to display a virtual object by operating the virtual switch associated with that virtual object, the display determination unit 25 determines that the virtual object should be displayed. When the driver 100 instructs not to display a virtual object by operating the virtual switch, the display determination unit 25 determines that the virtual object should be set to not be displayed. This technology, which allows operation of an operation panel displayed in the vehicle's virtual space, can be implemented using MR (Mobile Reality).

[0054] The display determination unit 25 can also combine (A) to (C) to determine whether to display or not display the virtual object. For example, after determining that the virtual object should be displayed based on (B), the display determination unit 25 determines that the virtual object should be set to not be displayed when the driver 100 operates the virtual switch to indicate that the virtual object should not be displayed.

[0055] The target location acquisition unit 26 acquires information about the target location where container 300 should be placed. For example, when container 300A is loaded onto forklift 12, the target location acquisition unit 26 acquires the target location information of container 300A from management system 40 via transceiver 23. Alternatively, it can be configured such that when driver 100 inputs "Container 300A loaded" from the control panel in the vehicle's virtual space, and the transceiver 23 sends this information to management system 40, management system 40 sends the target location of container 300A to display device 20. Alternatively, it can be configured such that a designated camera captures images of container 300A being loaded onto forklift 1, management system 40 analyzes the captured images to determine which forklift is transporting which container, and sends the target location corresponding to container 300A to display device 20 worn by driver 100 of forklift 1. The target location acquisition unit 26 records the acquired target location information in storage unit 28.

[0056] The cargo size acquisition unit 27 acquires information about the length, width, and height of the container 300 being transported. Length refers to the length of the container 300 as seen by the driver 100 in the longitudinal direction; width refers to the length of the container 300 as seen by the driver 100 in the depth direction; and height refers to the length in the vertical direction. For example, multiple container sizes can be pre-registered in the storage unit 28, and the cargo size acquisition unit 27 acquires the size selected by the driver 100 using an operation panel in virtual space. Alternatively, the cargo size acquisition unit 27 can also acquire the size information of the container 300 from the management system 40 via the receiving / dispatching unit 23. Alternatively, the size information can be acquired at both ends of the container 300 as seen by the driver 100. Figure 3A When the distances from the driver 100 to both ends of the container 300 are measured using the depth sensor of the sensor unit 21 (300a, 300b), the cargo size acquisition unit 27 estimates the size of the container 300 based on the measurement results. For example, the cargo size acquisition unit 27 selects the container size closest to the measurement result from a variety of pre-registered container sizes and sets that value as the size of the container 300. The cargo size acquisition unit 27 records the acquired or estimated size information in the storage unit 28.

[0057] It should be noted that, alternatively, when the cargo size acquisition unit 27 estimates the size of the container 300, the driver 100 may be positioned at both ends of the container 300. Figure 3A The positional relationship of 300a and 300b is output to the control unit 24, which configures the desired setting range 301 based on this information. For example, sometimes... Figure 7A As illustrated in the example, when maintaining a container size of 300, it is tilted to the right or as in Figure 7B As illustrated in the example, the container is tilted forward and backward. Even in this state, if the intended setting range 301 is configured based on the premise that the container 300 is loaded straight without tilting to the left or right, the actual placement range of the container 300 deviates from the intended setting range 301. Therefore, the control unit 24 configures the intended setting range 301 based on the positions of both ends of the container 300 estimated by the cargo size acquisition unit 27, in order to reproduce the correct position and angle for placing the container 300.

[0058] The storage unit 28 stores information detected by the sensor unit 21, information received by the transceiver unit 23, CAD information of the forklift 1 or various types of forklifts, etc. It also stores map information of the facilities where loading and unloading operations are performed.

[0059] (action)

[0060] Next, refer to Figure 8 The process for handling the expected setting range 301, etc., is explained. Figure 8 This is a flowchart illustrating an example of display control in an implementation method.

[0061] When the driver 100, equipped with display device 20, sits in the driver's seat of forklift 1, the display device 20 is activated to align the vehicle's virtual space with the real world (step S1). During the alignment of the vehicle's virtual space with the real world, the positional relationship between the driver 100 and forklift 1 is calculated. For example, the control unit 24 uses the visible light camera of the sensor unit 21 to identify the steering wheel and other structures of the forklift 1, and infers the model of the forklift 1 based on the shape and appearance of the steering wheel, etc., captured in the images. The control unit 24 reads the CAD information corresponding to the inferred model from the CAD information recorded in the storage unit 28, grasps the overall shape of the forklift 1, and aligns it with the pre-generated vehicle virtual space. Furthermore, for example, the control unit 24 calculates the positional relationship between the driver 100 and the steering wheel by analyzing the image of the steering wheel captured by the visible light camera of the sensor unit 21 or by measuring the distance to the steering wheel using the depth sensor of the sensor unit 21. That is, based on the driver 100's seating position, the height of the driver 100's head, or the CAD information of the forklift 1 read, the control unit 24 calculates the positional relationship (horizontal distance, vertical distance, etc.) between the driver 100 and various parts of the forklift 1, and sets the position of the driver 100 in the vehicle virtual space to a position that conforms to reality. Thus, for example, the distance between the driver 100 and the forks 12 can be determined, and the desired setting range 301 can be configured based on the position in the vehicle virtual space corresponding to the position of the forks 12 in the real world. In addition, for example, a two-dimensional mark can be set at a designated position of the forklift 1 (e.g., the driver's seat side of the mast 13), and the two-dimensional mark can be identified by the sensor unit 21. Based on the positional relationship with the two-dimensional mark, the model of the forklift 1 shown by the two-dimensional mark and the CAD information of the model, the vehicle virtual space can be aligned with the real world and the position of the driver 100 in the vehicle virtual space can be adjusted in the same way as described above.

[0062] Next, the control unit 24 aligns the facility's virtual space with the real world (step S2). For example, as... Figure 9As shown, QR codes (registered trademarks) and other markers 91 can be pre-installed at designated locations 90 in the real world (actual space), and these markers 91 are installed in multiple locations within the facility where loading and unloading operations are performed. Furthermore, in a virtual space of the facility pre-generated based on map information of the facility stored in the storage unit 28, information indicating that the location is a marker installation location is marked at a position 92 corresponding to the location 90 of the marker 91 in the real world. The control unit 24 captures images of the markers 91 in the real world using a visible light camera or the like from the sensor unit 21. The marker 91 includes, for example, location information, and the control unit 24 reads the location information of the location 90 of the marker 91 from the images captured by the sensor unit 21. The control unit 24 establishes a correspondence between the location 92 in the facility virtual space and the read location information. This aligns the facility virtual space with the real world. Furthermore, the control unit 24 can also calculate the relative positional relationship between the vehicle body 10 and the driver 100 by establishing a correspondence between the position information of the vehicle body 10 located by GNSS 30a and the position in the facility virtual space corresponding to the location of the vehicle body 10, thereby aligning the facility virtual space with the real world. For example, the control unit 24 acquires the position information located by GNSS 30a at the current position and establishes a correspondence between the acquired position information and the coordinate position in the facility virtual space corresponding to that position information. Here, the coordinates of the facility virtual space are constructed based on the real world, and any coordinate position in the facility virtual space corresponds to the latitude and longitude of the real world. Therefore, the alignment of the vehicle body 10 with the facility virtual space can be achieved by establishing a correspondence between the position information (latitude and longitude) of the vehicle body 10 in the real world located by GNSS 30a and the corresponding coordinate position in the facility virtual space. Moreover, the control unit 24 can, for example, calculate the relative positional relationship between the position of the vehicle body 10 equipped with GNSS 30a and the driver 100 based on the positional relationship between the driver 100 and the forklift 1 calculated in step S1, and use the calculated relative positional relationship to calculate the position information (latitude and longitude) of the driver 100. When calculating the latitude and longitude of the driver 100, the coordinate position of the driver 100 in the facility virtual space can be calculated in the same way as in the case of the vehicle body 10. For example, the control unit 24 calculates the latitude and longitude of the driver 100 each time it obtains location information from GNSS 30a, and calculates the coordinate position of the driver 100 in the facility virtual space each time. In this way, the control unit 24 uses the location information obtained from GNSS 30a to align the facility virtual space with the real world. As a result of the alignment, by configuring a target setting range 302 at the target location in the facility virtual space, the target setting range 302 is also displayed at the target location in the real world.AR glasses have the function of recognizing the surrounding environment and generating map information of the surrounding environment to create a virtual space. In the case of large vehicles such as forklifts, they can automatically recognize the positional relationship between virtual objects placed within a certain range (e.g., the driver's seat, around the forklift 1) and the driver 100, and display the objects within an appropriate range in the virtual space. However, if outside this range, the correct positional relationship cannot always be reproduced. For example, if a virtual object is placed near the steering wheel in the driver's seat, and the driver 100 brings their face close to the steering wheel, the virtual object will appear to exist near the driver 100's face. However, even if a virtual object is placed 10m in front of the forklift 1 and the forklift 1 moves toward the virtual object, the virtual object 10m in front will move along with the forklift 1. To address this, in this embodiment, a virtual space around the forklift 1 (vehicle virtual space) and a virtual space outside of it (facility virtual space) are prepared. Based on the positional relationship with the driver 100, virtual objects are placed at the desired positions in each virtual space, thereby attempting to display the virtual objects at the desired positions in the real world.

[0063] When alignment is complete, driver 100 drives forklift 1 to load container 300 onto forks 12. At this point, target position acquisition unit 26 acquires the target position (step S3). Control unit 24 sets the position on the facility virtual space corresponding to the target position of container 300 as the target position. Furthermore, cargo size acquisition unit 27 acquires the dimensions of container 300 (step S4). Additionally, control unit 24 acquires the position information of forklift 1 located by GNSS 30a via transceiver unit 23 (step S5). When acquiring the position information of forklift 1, control unit 24 uses the positional relationship between driver 100 and forklift 1 obtained in step S1 to correct the current position of forklift 1 to the current position of driver 100, and sets the position on the facility virtual space corresponding to the corrected current position of driver 100 as the current position of driver 100. Next, display determination unit 25 determines whether each virtual object (e.g., expected setting range 301, target setting range 302, virtual object 303) meets the display conditions using the methods described in (A) to (C) above. If the display conditions are met (step S6; Yes), the control unit 24 displays the virtual object that meets the display conditions (step S7). If the display conditions are not met (step S6; No), the control unit 24 sets the virtual object that does not meet the display conditions to not be displayed (step S8). For example, when the forklift 1 is far from the target position, the control unit 24 displays the target setting range 302 and the virtual object 303, and sets the expected setting range 301 to not be displayed. For example, if the forklift 1 and the target position are close to a predetermined distance, the control unit 24 displays the expected setting range 301 and the target setting range 302, and sets the virtual object 303 to not be displayed. Specifically, the control unit 24 calculates the target range for setting the container 300 as seen by the driver 100 in the real world based on the positional relationship between the forklift 1 and the driver 100, the position information of the forklift 1, and the target position, and configures the target setting range 302 to be the same range that the driver 100 can see in the facility virtual space. Furthermore, the control unit 24 maintains the position of the container 300 based on the positional relationship between the forklift 1 and the driver 100 (e.g., Figure 3A The location information of L3) and the size information of container 300 are used to calculate the range of the cargo expected to be unloaded when container 300 is unloaded as it is seen by driver 100 in the real world. The same range configuration expected setting range 301 is seen by driver 100 in the vehicle virtual space.

[0064] Next, the control unit 24 determines whether to end the function of displaying the expected setting range 301, etc., by the unloading assistance system 200 (step S9). For example, if the driver 100 performs an operation to end this function, the control unit 24 determines that the function of displaying the expected setting range 301, etc., should be ended; otherwise, it determines that the function should not be ended. If it is determined that the function should not be ended (step S9; no), the processing after step S5 is repeated. On the other hand, if it is determined that the function should be ended (step S9; yes), the process ends. Figure 8 The processing flow.

[0065] It should be noted that in the above embodiment, the unloading assistance system 200 was described as being applied to a forklift 1 as an example. However, the unloading assistance system 200 can also be applied to industrial vehicles other than forklifts, such as excavators, bulldozers, cranes, and reach stackers. Examples of working devices found on industrial vehicles include the forks 12 of the forklift 1, the boom and bucket of the excavator, the bucket and boom of the bulldozer, and the boom, boom, and hook of the crane. Furthermore, examples of goods include containers, sand, and rubble transported by these working devices.

[0066] (Effect)

[0067] According to this embodiment, the driver is equipped with AR glasses (display device 20), which display the expected setting range 301 and the target setting range 302 in a virtual space (vehicle virtual space and facility virtual space). Therefore, regardless of the height of the container 300, the driver 100 can unload the container 300 while simultaneously confirming the expected setting range for unloading the container 300 while keeping it intact and the target setting range of the container 300. Furthermore, by using AR glasses to overlay the expected setting range 301 onto the real world, the driver 100 can confirm the expected setting range 301 and the target setting range 302 with a natural line of sight without needing to check a monitor in the driver's seat.

[0068] Figure 10 This diagram illustrates an example of the hardware configuration of the unloading assistance system according to the implementation method. The computer 900 includes a CPU 901, a main storage device 902, an auxiliary storage device 903, an input / output interface 904, and a communication interface 905.

[0069] The aforementioned display device 20 and management system 40 are installed in the computer 900. Furthermore, the aforementioned functions are stored as programs in the auxiliary storage device 903. The CPU 901 reads the program from the auxiliary storage device 903 and extends it to the main storage device 902, executing the aforementioned processing according to the program. Additionally, the CPU 901 secures a storage area in the main storage device 902 according to the program. Furthermore, the CPU 901 secures a storage area in the auxiliary storage device 903 according to the program for storing data during processing.

[0070] Alternatively, programs for implementing all or part of the functions of the display device 20 and the management system 40 can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer system and executed to perform processing of each functional unit. Here, "computer system" refers to hardware including an OS (Operating System) and peripheral devices. Furthermore, if the "computer system" utilizes a WWW system, it also includes a homepage providing environment (or display environment). "Computer-readable recording medium" refers to portable media such as CDs, DVDs, and USB drives, as well as storage devices such as hard disks built into the computer system. Alternatively, when the program is distributed to the computer 900 via a communication line, the receiving computer 900 can deploy the program in the main storage device 902 and execute the aforementioned processing. Furthermore, the program can be a program for implementing a portion of the aforementioned functions, or it can be a program that can be implemented in combination with a program that has already stored the aforementioned functions in the computer system.

[0071] As described above, some embodiments of this disclosure have been illustrated, but all of these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, and similarly, within the scope of the invention as described in the claims and its equivalents.

[0072] <Postscript>

[0073] For example, the display device, unloading assistance system, industrial vehicle, control method, and procedure described in each embodiment are understood as follows.

[0074] (1) The display device of the first embodiment is a display device worn on a person's head, comprising: a see-through display unit capable of displaying image data; a target position acquisition unit for acquiring target position information as the location information of the destination of goods transported by an industrial vehicle; a position information acquisition unit for acquiring vehicle position information as the location information of the industrial vehicle or the person riding in the industrial vehicle; and a control unit for displaying the image data including the target setting range of the goods and the expected setting range of the goods in such a way that the target setting range included in the image data overlaps with the target setting range in the real world that can be seen through the display unit, wherein the target setting range of the goods is the target setting range of the goods seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle position information, and the target position information, and the expected setting range of the goods is the expected setting range of the goods seen by the person when the goods are unloaded from the industrial vehicle based on information about the positional relationship, information about the position of the goods maintained by the industrial vehicle, and information about the size of the goods.

[0075] It can assist unloading operations by displaying the predicted range of goods and the target range as the target when unloading goods from industrial vehicles.

[0076] (2) The display device of the second scheme is the display device of (1), wherein the control unit displays the image data when the distance between the industrial vehicle and the destination is within a specified range.

[0077] In unloading scenarios, displaying image data can prevent it from obstructing the driver's view.

[0078] (3) The display device of the third scheme is the display device of (1) to (2), wherein the control unit displays the image data when the goods are loaded into the industrial vehicle.

[0079] Therefore, in scenarios where no goods are being transported, the image data can be set not to be displayed. This prevents it from obstructing the driver's view.

[0080] (4) The display device of the fourth scheme is the display device of (1) to (3), wherein the control unit displays the image data when the bottom surface of the cargo held by the working device of the industrial vehicle is horizontal with the ground.

[0081] When unloading cargo, the tilted gantry is restored to vertical to allow for unloading. By displaying image data when this state is achieved, it prevents the gantry from obstructing the driver's view during previous operations.

[0082] (5) The display device of the fifth scheme is the display device of (1) to (4), wherein the control unit displays a virtual operation unit that accepts instructions to switch between displaying and not displaying the image data, and switches between displaying and not displaying the image data based on the operation of the person on the operation unit.

[0083] Therefore, it is possible to control the display of image data to not occur in scenarios where the driver becomes an obstacle.

[0084] (6) The display device of the sixth scheme is the display device of (1) to (5), wherein the control unit displays second image data that guides the location of the delivery destination without displaying the image data.

[0085] Even outside of unloading cargo, drivers can navigate industrial vehicles toward their target location by displaying the target location guidance.

[0086] (7) The display device of the seventh scheme is the display device of (1) to (6), wherein it further includes a size acquisition unit, the size acquisition unit acquires the size of the goods, and the control unit calculates the target setting range based on the estimated size.

[0087] Therefore, the target setting range corresponding to the cargo size can be calculated.

[0088] (8) The display device of the eighth solution is the display device of (1) to (7), wherein it further includes a size acquisition unit, the size acquisition unit acquires the size of the goods, and the control unit calculates the target setting range based on the acquired size.

[0089] Therefore, the target setting range corresponding to the cargo size can be calculated.

[0090] (9) The unloading assistance system of the ninth scheme includes: a positioning device for locating the location information of the industrial vehicle or the person riding in the industrial vehicle; and a display device of (1) to (8).

[0091] (10) The industrial vehicle of the tenth scheme is equipped with the unloading assistance system of (9).

[0092] (11) The control method of the eleventh solution is a control method for a display device worn on a person's head with a perspective display section capable of displaying image data. The control method includes the following steps: acquiring target location information as location information of the destination of goods transported by an industrial vehicle; acquiring vehicle location information as location information of the industrial vehicle or the person riding in the industrial vehicle; and displaying the image data, which includes the target setting range of the goods and the expected setting range of the goods, in such a way that the target setting range included in the image data overlaps with the target setting range in the real world that can be seen through the display section. The target setting range of the goods is the target setting range of the goods seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle location information, and the target location information. The expected setting range of the goods is the expected setting range of the goods seen by the person when the goods are unloaded from the industrial vehicle, based on information about the positional relationship, information about the position of the goods held by the industrial vehicle, and information about the size of the goods.

[0093] (12) The program of the twelfth embodiment enables the computer of the display device to function as the following unit, the display device being a head-mounted display device having a see-through display section capable of displaying image data: a unit for acquiring target location information as the location information of the destination of goods transported by an industrial vehicle; a unit for acquiring vehicle location information as the location information of the industrial vehicle or the person riding in the industrial vehicle; and for displaying the image data including the target setting range and the intended setting range of the goods in such a way that the target setting range included in the image data overlaps with the target setting range in the real world that can be seen through the display section, wherein the target setting range of the goods is the target setting range of the goods seen by the person based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle location information, and the target location information, and the intended setting range of the goods is the intended setting range of the goods seen by the person when the goods are unloaded from the industrial vehicle based on information about the positional relationship, information about the position of the goods held by the industrial vehicle, and information about the size of the goods.

[0094] Explanation of reference numerals in the attached figures

[0095] 1: Forklift;

[0096] 10: Vehicle body;

[0097] 11: Loading and unloading equipment;

[0098] 12: Forklifts;

[0099] 13: Gantry;

[0100] 14: Tilt sensor;

[0101] 15: Load sensor;

[0102] 20: Display device;

[0103] 21: Sensors Department;

[0104] 22: Display section;

[0105] 23: Receiving and Dispatch Department;

[0106] 24: Control Department;

[0107] 25: Display judgment section;

[0108] 26: Target Location Acquisition Unit;

[0109] 27: Cargo Size Acquisition Department;

[0110] 28: Storage Department;

[0111] 30, 30a, 30b: Positioning devices (GNSS);

[0112] 40: Management system;

[0113] 100: Driver;

[0114] 200: Unloading Assist System;

[0115] 900: Computer;

[0116] 901: CPU;

[0117] 902: Main storage device;

[0118] 903: Auxiliary storage device;

[0119] 904: Input / output interface;

[0120] 905: Communication interface.

Claims

1. A display device, which is worn on a person's head, comprising: A perspective-type display unit capable of displaying image data; The target location acquisition unit acquires target location information, which is the location information of the destination of the goods being transported by the industrial vehicle. The location information acquisition unit acquires vehicle location information, which is the location information of the industrial vehicle or the person riding in the industrial vehicle. as well as The control unit displays the image data, which includes the target setting range and the intended setting range of the cargo, in a manner that overlaps the target setting range included in the image data with the target setting range in the real world that can be seen through the display unit. The target setting range of the cargo is the target setting range of the cargo as seen by the person riding in the industrial vehicle, based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle's position information, and the target position information. The intended setting range of the cargo is the intended setting range of the cargo as seen by the person when the cargo is unloaded from the industrial vehicle, based on information about the positional relationship, information about the position of the cargo held by the industrial vehicle, and information about the size of the cargo.

2. The display device according to claim 1, wherein, The control unit displays the image data when the distance between the industrial vehicle and the delivery destination is within a specified range.

3. The display device according to claim 1 or 2, wherein, The control unit displays the image data when the cargo is loaded into the industrial vehicle.

4. The display device according to claim 1 or 2, wherein, The control unit displays the image data when the bottom surface of the cargo, held horizontally by the working device of the industrial vehicle, is level with the ground.

5. The display device according to claim 1 or 2, wherein, The control unit displays a virtual operation unit that receives instructions to switch between displaying and not displaying the image data, and switches between displaying and not displaying the image data based on the user's operation of the operation unit.

6. The display device according to claim 1 or 2, wherein, The control unit displays second image data that guides the location of the delivery destination without displaying the image data itself.

7. The display device according to claim 1 or 2, It also includes a size acquisition unit to estimate the size of the goods. The control unit calculates the target setting range based on the estimated dimensions.

8. The display device according to claim 1 or 2, It also includes a size acquisition unit to acquire the size of the goods. The control unit calculates the target setting range based on the acquired dimensions.

9. An unloading auxiliary system, comprising: The positioning device locates the position information of the industrial vehicle or the person riding in the industrial vehicle; and The display device according to claim 1 or 2.

10. An industrial vehicle comprising the unloading auxiliary system of claim 9.

11. A control method for a display device worn on a person's head, having a perspective display section capable of displaying image data, the control method comprising the following steps: The steps to obtain target location information as the location information of the destination of goods transported by industrial vehicles; The step of obtaining vehicle location information, which is either the industrial vehicle or the person riding in the industrial vehicle; and The step of displaying the image data, which includes the target setting range and the intended setting range of the goods, in such a way that the target setting range included in the image data overlaps with the target setting range in the real world that can be seen through the display unit, wherein... The target setting range of the cargo is the target setting range of the cargo as seen by the person riding in the industrial vehicle, based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle's position information, and the target position information. The expected setting range of the cargo is the expected setting range of the cargo as seen by the person when unloading the cargo from the industrial vehicle, based on information about the positional relationship, information about the position of the cargo maintained by the industrial vehicle, and information about the size of the cargo.

12. A computer-readable recording medium containing a program for enabling a computer of a display device to function as a head-mounted display device having a perspective display section capable of displaying image data: A unit for acquiring target location information of the destination of goods transported by industrial vehicles. A unit for acquiring vehicle location information, which is either the industrial vehicle or a person riding in the industrial vehicle; and A unit that displays image data including the target setting range and the intended setting range of the goods in such a way that the target setting range included in the image data overlaps with the target setting range in the real world that can be seen through the display unit, wherein... The target setting range of the cargo is the target setting range of the cargo as seen by the person riding in the industrial vehicle, based on information about the positional relationship between the person riding in the industrial vehicle and the industrial vehicle, the vehicle's position information, and the target position information. The expected setting range of the cargo is the expected setting range of the cargo as seen by the person when unloading the cargo from the industrial vehicle, based on information about the positional relationship, information about the position of the cargo maintained by the industrial vehicle, and information about the size of the cargo.

Images