Display device, visibility assistance system, industrial vehicle, control method and program

Abstract

To provide a display device that can check an image obtained by photographing a work object position while viewing the work object position by a work device included in an industrial vehicle.SOLUTION: A display device, which is worn on the head of a person, comprises: a transparent display part that can display image data; an acquisition part that acquires an image photographed by a camera provided on a work device included in an industrial vehicle; and a control part that displays the image data including a virtual monitor on which the image is displayed around a position separated by a focal distance in a direction of the line of sight of the person.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a display device, a visibility assistance system, an industrial vehicle, a control method, and a program. [Background technology]

[0002] Patent Document 1 discloses a loading and unloading support device that installs a camera at the tip of the fork of a forklift and displays the image captured by the camera on a monitor installed in the driver's seat. With this support device, loading and unloading work can be performed while checking the image in front of the raised fork. In such a support device, if the worker is to work while simultaneously looking at the work target position and the monitor, it is recommended to install the monitor above the driver's seat. However, if the worker does this, he or she must look upward even when looking only at the monitor, which is inconvenient when the worker wants to visually check other directions. However, if the monitor is installed in front of the driver's seat, part of the field of view is blocked by the monitor when the vehicle is traveling. In addition, the focal length of the work target position seen from the driver's seat when the fork is raised and the image displayed on the monitor are different, so if the worker tries to check both, the burden of switching the focus is incurred. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4666154 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for technology that allows work to be performed while checking images of the work target position, such as the front of the raised forks.

[0005] The present disclosure provides a display device, a visibility assistance system, an industrial vehicle, a control method, and a program that can solve the above problems. [Means for solving the problem]

[0006] The display device of the present disclosure is a display device worn on a person's head, and includes a transparent display unit capable of displaying image data, an acquisition unit that acquires an image captured by a camera provided on a work device equipped in an industrial vehicle, and a virtual monitor on which the image is displayed. a function of displaying the image data on the display unit as if the image data were located at a position in the real world corresponding to the position in the virtual space by placing the image data at any position in the virtual space; Around a position that is a focal distance away in the direction of the person's line of sight and that can be seen through the display unit The image data is arranged at a corresponding position in the virtual space. and a control unit for controlling the

[0007] The visibility assistance system of the present disclosure includes a camera provided on a work device equipped on an industrial vehicle, and the above-described display device.

[0008] The industrial vehicle of the present disclosure includes a camera provided on a work device equipped on the industrial vehicle, and the above-mentioned display device.

[0009] The control method of the present disclosure is a control method for a display device to be worn on a person's head, which has a transparent display unit capable of displaying image data, and includes acquiring an image captured by a camera provided on a work device provided on an industrial vehicle, and displaying the image data including a virtual monitor on which the image is displayed, a function of displaying the image data on the display unit as if the image data were located at a position in the real world corresponding to the position in the virtual space by placing the image data at any position in the virtual space; Around a position that is a focal distance away in the direction of the person's line of sight and that can be seen through the display unit The image data is arranged at a corresponding position in the virtual space. do.

[0010] The program of the present disclosure includes a computer of a display device to be worn on a person's head, the computer having a transparent display unit capable of displaying image data, a means for acquiring an image captured by a camera provided on a work device provided on an industrial vehicle, and a virtual monitor on which the image is displayed. a function of displaying the image data on the display unit as if the image data were located at a position in the real world corresponding to the position in the virtual space by placing the image data at any position in the virtual space; Around a position that is a focal distance away in the direction of the person's line of sight and that can be seen through the display unit The image data is arranged at a corresponding position in the virtual space. It serves as a means to Effect of the Invention

[0011] According to the above-described display device, visibility assistance system, industrial vehicle, control method, and program, it is possible to check an image captured of a work target position while viewing the work target position using a work device provided on the industrial vehicle. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a forklift according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of a display device according to an embodiment; [Diagram 3] 1A and 1B are diagrams illustrating a focal length and a size of a virtual object according to an embodiment. [Figure 4] FIG. 2 is a diagram showing a forklift during loading and unloading operations according to the embodiment; [Diagram 5] FIG. 2 is a diagram showing a display example of an image captured by a camera according to an embodiment. [Figure 6] 10 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] <Embodiment> Hereinafter, the display device of the present disclosure will be described with reference to the drawings. (composition) FIG. 1 is a schematic diagram showing an example of a forklift according to each embodiment. The forklift 1 includes a vehicle body 10, a loading device 11, a control device 18, and a display device 20. The vehicle body 10 includes a drive device and a steering device (not shown) and drives the forklift 1. The loading device 11 includes a fork 12, a mast 13 for raising and lowering the fork 12, and a backrest 14 for preventing the load from falling backward (towards the mast 13), and performs a loading operation by raising and lowering the load loaded on the fork 12. The loading device 11 is an example of a working device. The mast 13 includes an outer mast (not shown) and an inner mast (not shown), and is configured such that the inner mast can be raised and lowered relative to the stationary outer mast. The mast 13 is provided with a lifting height sensor 15, which detects the height position of the inner mast. In addition, the outer mast is provided with a proximity sensor 16. The proximity sensor 16 detects when movable parts such as the backrest 14, the forks 12, and the inner mast move away from the proximity sensor 16 provided on the outer mast. A camera 17 is provided at the tip of the forks 12. The camera 17 takes an image of the area in front of the forks 12. The image (which may be a video or a still image) taken by the camera 17 is sent to the display device 20. The control device 18 controls the operation of the forklift 1. For example, when the driver 100 operates the lift lever 19, the control device 18 detects the operation and performs an operation to raise and lower the mast 13. The control device 18 is connected to the lift height sensor 15 and the proximity sensor 16, and obtains information detected by these sensors. The control device 18 transmits to the display device 20 information detected by the lift height sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the lift lever 19, and other information such as the moving direction and vehicle speed of the forklift 1. The display device 20 is an AR (Augmented Reality) glass or an MR (Mixed Reality) glass. The driver 100 wears the display device 20 and drives the forklift 1. In this specification, the direction in which the loading device 11 of the forklift 1 is provided as shown in Fig. 1 is called the front, the opposite direction is called the rear, and the left and right directions when facing forward are called the left side and the right side, respectively.

[0014] 2 is a block diagram showing an example of a display device according to an embodiment. The display device 20 is an AR glass or an MR glass including a display device and a computer (hereinafter, both will be referred to as AR glass). The display device 20 is communicably connected to a control device 18 and a camera 17, and acquires information from these devices. The display device 20 displays an image captured by the camera 17. As shown in the figure, the display device 20 includes a sensor unit 21, a display unit 22, a transmission / reception unit 23, a control unit 24, a display determination unit 25, and a storage unit 26.

[0015] The sensor unit 21 includes a sensor for head tracking of the driver 100 (for example, a plurality of visible light cameras and an IMU (Inertial Measurement Unit)), a sensor for eye tracking of the driver 100 (for example, an infrared camera), a depth sensor, a microphone, and the like.

[0016] The display unit 22 constitutes a part of the lens (screen) of the AR glasses, and is constituted by, for example, a transparent holographic display (the outside world is visible through the lens). In a state where a virtual object is not projected onto the lens, the driver 100 can see the real world as it is. The virtual object is image data projected onto the display unit 22. When a virtual object is displayed on the display unit 22, the driver 100 sees the virtual object superimposed on the real world.

[0017] The transmission / reception unit 23 transmits and receives data to and from other devices. For example, the transmission / reception unit 23 transmits and receives data to and from other devices by wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark). For example, the transmission / reception unit 23 receives images captured by the camera 17. For example, the transmission / reception unit 23 receives information detected by the lift height sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the lift lever 19, the moving direction and vehicle speed of the forklift 1, and the like from the control device 18.

[0018] The control unit 24 performs display control of the display unit 22, etc. The control unit 24 recognizes the spatial shape around the driver 100 based on the information detected by the sensor unit 21, creates three-dimensional map information of the surrounding space, and generates a virtual space. The control unit 24 places a virtual object at a desired position in the virtual space based on the created map information. The virtual object is displayed to the driver 100 superimposed on an object existing at that position in the real world. The virtual object can be displayed at (1) a predetermined position in the space (the virtual object exists at a fixed position), (2) a predetermined position based on an object existing in the surrounding space (for example, if the object is the vehicle body 10, the virtual object moves following the running of the vehicle body 10), or (3) a predetermined position based on the driver 100 (for example, the virtual object exists at a position a predetermined distance away in the line of sight of the driver 100), etc. In the case of (1) or (2), the virtual object is recognized by the driver 100 as if it actually exists at that position. In the case of (3), the virtual object can be placed at any position based on the driver 100, and the placement position can be freely changed by moving it closer or farther away from the driver 100. For example, if the virtual object is placed far away, the driver 100 sees the virtual object as if it exists far away, and if it is placed close, the driver 100 sees the virtual object as if it exists close. When the driver 100 is looking 3 m ahead, if a virtual object is placed 3 m ahead in the line of sight, the virtual object appears to exist at that position. For the driver 100 looking 3 m ahead, the virtual object appears to exist exactly at the position where the driver's line of sight is aligned (the position where the driver's eyes are focused / in focus), so there is no need to refocus to view the virtual object. In contrast, when the driver 100 is looking 3 m ahead, if a virtual object is placed 30 cm ahead in the line of sight, the driver 100 looking 3 meters ahead sees the virtual object as if it exists 30 cm away from the driver, so the driver cannot clearly see the virtual object unless he or she switches the focus from 3 m ahead to 30 cm ahead.That is, in order to allow the driver 100 to view the virtual object as it is without imposing a burden on the driver 100 to switch the focus, it is preferable to place the virtual object as close as possible to the focal position of the driver 100. Due to the characteristics of the AR glasses, the position where the virtual object is placed can be freely controlled, so that the burden on the driver 100 to adjust the focus can be reduced by adjusting the position where the virtual object is placed according to the line of sight of the driver 100 to match the focal length at that time.

[0019] The AR glasses have the function of freely arranging virtual objects and arranging a virtual object at a certain position in the virtual space so that the virtual object appears to exist at that position. The user sets the desired virtual object in the display device 20 (a developer uses an AR development tool to develop a virtual object and implements the developed program in the display device 20). The control unit 24 then places the virtual object at the desired position according to the setting. In this embodiment, the control unit 24 is set to follow the line of sight of the driver 100, place a virtual monitor VM (a virtual monitor is a virtual object displayed by AR) near the focal length at that time (corresponding to (3) above), and display an image captured by the camera 17 on the virtual monitor VM. Next, the relationship between the position at which the virtual object is arranged and the size of the virtual object will be described with reference to FIG. 3.

[0020] 3 shows a virtual object 31a arranged at a distance L1 forward from the driver 100, a display range (range of the visual field that fits in the display unit 22) 31b at the distance L1 forward, a virtual object 32a arranged at a distance L2 forward from the driver 100, and a display range 32b at the distance L2 forward. If a real monitor that is not a virtual object is arranged on the rear side of the mast 13 (for example, at a distance L2 forward from the driver 100), the monitor will be hidden by the mast 13 and will not be visible, but in the case of an AR virtual monitor VM, even if it is arranged on the rear side of the mast 13, the virtual monitor VM will not be hidden by the mast because it is displayed on the AR glasses. The AR technology also has an occlusion function that displays the virtual monitor VM as if it is hidden by the mast 13, but in this embodiment, the occlusion function is not used in order to prioritize information displayed on the virtual monitor VM. As shown in FIG. 3, when two virtual objects 31a and 32a of different sizes (virtual object 32a has a larger size in the virtual space) are placed at positions of distances L1 and L2, respectively, due to the nature that the farther an object is, the smaller it appears, it is possible to make both appear to be the same size to the driver 100. In other words, when a virtual object is placed farther away, the size of the virtual object is made larger, and when it is placed closer, the size of the virtual object is made smaller. The specific size of the virtual object can be calculated based on the placement position (distance from the driver 100) and the size at which it is displayed on the display unit 22, and by changing the size of the virtual object according to the distance from the driver 100, it is possible to make the virtual object always appear to the driver 100 at a constant size. Assuming that the virtual object is not hidden by the mast 13 and is not visible, the difference between the two virtual objects 31a and 32a displayed to the driver 100 at the same size is only the placement position, in other words, the focal length. By controlling the placement position and size of the virtual object placed in the virtual space in this way, it is possible to freely change only the focal length without changing the display size of the virtual object as seen by the driver 100.

[0021] By utilizing this property, if a virtual monitor VM with an appropriate size control is placed near the tip of the fork 12, the focus when looking at the tip of the real fork 12 and when looking at the virtual monitor VM will be almost the same, and the driver 100 will be able to see both at the same time. This is shown in FIG. 4. FIG. 4 shows a scene in which the fork 12 is being inserted into a hole 43 in a pallet 42 loaded with luggage 41 placed on a shelf in a warehouse or the like. The camera 17 captures an image of the front at the same height as the fork 12, and the driver 100 grasps the position of the hole 43 from the image captured by the camera 17 and positions the fork 12. In this case, if a real monitor that displays the image captured by the camera 17 is provided in the driver's seat as in the conventional technology, the driver will have to look alternately at the direction of the fork 12 and the monitor, which is inefficient. In addition, a burden of switching the focus will be incurred. Even if the real monitor is installed in a position and direction such that the display screen can be seen when looking up in the direction of the forks 12, the focal length when viewing the luggage 41 and pallet 42 near the tips of the forks 12 is different from the focal length of the image displayed on the monitor. Therefore, even if both the real object and the monitor are included in the field of view, the driver 100 cannot check both at the same time, and actually checks one by one while switching the focus. In contrast, in this embodiment, the virtual monitor VM is placed near the tips of the forks 12 (near the pallet 42) using AR technology, for example, as shown in the figure. Then, the image captured by the camera 17 is displayed on the virtual monitor VM. Then, the driver 100 sees the image captured by the camera 17 displayed on the monitor located near the pallet 42. The focal length of the image displayed on the virtual monitor VM is approximately the same as the focal length when viewing the real object such as the forks 12. In addition, by placing the virtual monitor VM near the pallet 42, the image captured by the camera 17 can be displayed within the field of view when the driver 100 views the real object near the tips of the forks 12. This allows the driver 100 to check the image displayed on the virtual monitor VM while looking at the actual objects such as the fork 12 without switching the focal length.This allows the driver 100 to carry out the loading and unloading work while simultaneously checking both the work target position near the tip of the fork 12 and the image captured by the camera 17 without the burden of focusing. For example, when the fork 12 is raised to a higher position to carry out the loading and unloading work, the control unit 24 places the virtual monitor VM at a higher position. In this case, since the distance from the driver 100 increases, the control unit 24 increases the size of the virtual monitor VM as described with reference to FIG. 3. Even if the size of the virtual monitor VM is increased, the virtual monitor VM (the image captured by the camera 17) appears to the driver 100 to be the same size due to the principle of perspective. In this way, by changing the placement position and size of the virtual monitor VM according to the work target position seen by the driver 100, the driver 100 can carry out the work while checking the work target position in the real world and the image captured by the camera 17 without switching the focal length. As for the target position of the driver's 100 line of sight, for example, when the forks 12 are being raised or lowered, the vicinity of the tips of the forks 12 may be set as the target of the driver's 100 line of sight (for example, the focal length is calculated using the height detected by the lifting height sensor 15 and the horizontal length from the driver's seat to the tips of the forks 12). Alternatively, the target position of the driver's 100 line of sight may be estimated based on a value detected by an eye tracking sensor provided in the sensor unit 21. Alternatively, the focal length may be fixed and set at about several meters. By setting the focal length at about several meters, the virtual monitor VM can be effectively used not only in a scene where the forks 12 are raised to perform loading and unloading work.

[0022] Fig. 5 shows an example of the virtual monitor VM displayed on the display unit 22. In the example of Fig. 5, an image captured by the camera 17 with the forks 12 lowered is displayed on the virtual monitor VM. For example, when driving the forklift 1 to move forward, the image captured by the camera 17 is displayed on the virtual monitor VM, so that the driver 100 can visually check the surroundings of the vehicle and at the same time check the state around the feet of the forklift 1, which is likely to be a blind spot. In this case, too, by arranging the virtual monitor VM according to the focal length when visually checking the surroundings of the vehicle, the driver 100 can easily check the real world around the vehicle and the virtual monitor VM.

[0023] The position and range of the virtual monitor VM displayed on the display unit 22 can be set arbitrarily so as not to obstruct the field of vision. Also, the virtual monitor VM can be switched between display and non-display automatically or by the action or operation of the driver 100. For example, in a scene where the virtual monitor VM is obstructive, the virtual monitor VM can be hidden.

[0024] The display determination unit 25 determines whether to display or hide the virtual monitor VM. When the display determination unit 25 determines that the virtual monitor VM is to be displayed, the control unit 24 displays the virtual monitor VM (places the virtual monitor VM in the virtual space), and when the display determination unit 25 determines that the virtual monitor VM is to be hidden, the control unit 24 hides the virtual monitor VM (does not place the virtual monitor VM in the virtual space). For example, the display determination unit 25 determines that the virtual monitor VM is to be displayed when loading and unloading work is in progress, and determines that the virtual monitor VM is to be hidden when this is not the case. An example of the determination process of the display determination unit 25 is described below.

[0025] (A) The display determination unit 25 acquires information such as a signal indicating that the lift lever 19 of the driver's seat has been operated and the vehicle speed of the forklift 1 from the control device 18 through the transmission / reception unit 23. When the operation of the lift lever 19 is detected, the display determination unit 25 determines that the virtual monitor VM is to be displayed. Also, when the operation of the lift lever 19 is detected and then no longer detected, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (for example, about 10 seconds) from that time. Alternatively, when the operation of the lift lever 19 continues for a predetermined period of time (for example, about 1 to 2 seconds) or more, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (for example, about 10 seconds) from that time. During that time, if there is even a slight lift or tilt operation by the lift lever 19, the 10-second count is reset each time. In other words, the display determination unit 25 determines that the virtual monitor VM is to be displayed for a certain period of time (for example, about 10 seconds) from the detection of the lift or tilt operation. Furthermore, when the vehicle speed reaches or exceeds a certain level (for example, 4 km / h or higher), the display determination unit 25 determines that the loading and unloading work has ended, forcibly ends the 10-second count, and determines that the virtual monitor VM should not be displayed.

[0026] (B) The display determination unit 25 acquires information detected by the height sensor 15 (the height of the forks 12 and the inner mast) from the control device 18 via the transmission / reception unit 23. When the height detected by the height sensor 15 is equal to or greater than a predetermined threshold, the display determination unit 25 determines to display the virtual monitor VM, and when the height is less than the threshold, the display determination unit 25 determines to hide the virtual monitor VM.

[0027] (C) The display determination unit 25 obtains information detected by the proximity sensor 16 (whether the outer mast on which the proximity sensor 16 is provided is in proximity to the backrest 14, the forks 12, the inner mast, etc.) from the control device 18 via the transmission / reception unit 23. If the proximity sensor 16 detects that the outer mast and the inner mast, etc. are not in proximity, the display determination unit 25 determines to display the virtual monitor VM, and if it detects that the outer mast and the inner mast, etc. are in proximity, it determines to hide the virtual monitor VM.

[0028] (D) The display determination unit 25 uses the position of the connecting member of the inner mast, whether the rod of the lift cylinder (not shown) is visible, etc. as landmarks to determine whether the inner mast is raised using the visible light camera provided in the sensor unit 21 (the rod of the lift cylinder is hidden and not visible from the driver's seat when the mast 13 is not raised, but becomes visible from the driver's seat when the mast 13 is raised). The display determination unit 25 calculates the height position of the inner mast based on how the landmarks are visible. The display determination unit 25 determines to display the virtual monitor VM when the inner mast is raised by a predetermined threshold or more, and determines to hide the virtual monitor VM when the distance by which the inner mast has been raised is less than the threshold.

[0029] (E) The display determination unit 25 may determine that loading and unloading work is to be performed when a predetermined time has elapsed since the loading of cargo onto the forks 12, and may determine to display the virtual monitor VM, and may determine to hide the virtual monitor VM when the cargo is unloaded from the forks 12. The loading of cargo onto the forks 12 may be detected by a load sensor (not shown) provided on the mast 13 or the like, or the presence or absence of cargo may be detected by analyzing an image captured by a visible light camera provided in the sensor unit 21 above the forks 12.

[0030] (F) The display determination unit 25 may determine to display the virtual monitor VM when the line of sight of the driver 100 is a predetermined angle or more and continues to face upward for a predetermined time or more, and may determine to hide the virtual monitor VM when the line of sight of the driver 100 is a predetermined angle or less. The line of sight of the driver 100 can be estimated from information detected by an eye tracking sensor or an IMU for head tracking provided in the sensor unit 21.

[0031] The storage unit 26 stores information detected by the sensor unit 21, information received by the transmission / reception unit 23, virtual objects, CAD information of the forklift 1 or various types of forklifts, and the like.

[0032] (operation) Next, control of the display device 20 will be described with reference to FIG. FIG. 6 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. When the driver 100 wearing the display device 20 sits in the driver's seat of the forklift 1, the driver starts up the display device 20 and performs calibration (step S1). In the calibration, the positional relationship between the driver 100 and the forklift 1 is calculated. For example, the control unit 24 recognizes the handle and other structures of the forklift 1 using the visible light camera of the sensor unit 21, and estimates the type of vehicle of the forklift 1 from the shape and appearance of the photographed handle, etc. The control unit 24 reads out CAD information corresponding to the estimated type of vehicle from the CAD information recorded in the storage unit 26, and grasps the overall shape of the forklift 1. Also, for example, the control unit 24 calculates the positional relationship between the driver 100 and the steering wheel, that is, the position where the driver 100 sits in the driver's seat, the height of the driver 100's head, or the positional relationship between the driver 100 and each part of the forklift 1 (horizontal distance, vertical distance, etc.) from the read CAD information of the forklift 1 by analyzing the image of the steering wheel captured by the visible light camera of the sensor unit 21 or measuring the distance to the steering wheel by the depth sensor of the sensor unit 21. This makes it possible to grasp the accurate distance between the driver 100 and the tip of the fork 12, and to place the virtual monitor VM near the tip of the raised fork 12, for example. Also, for example, a two-dimensional marker may be provided at a predetermined position of the forklift 1, the two-dimensional marker may be recognized by the sensor unit 21, and the positional relationship between the driver 100 and each part of the forklift 1 may be calculated in the same manner as above from the positional relationship with the two-dimensional marker, the type of the forklift 1 indicated by the two-dimensional marker, and the CAD information of the type of the forklift 1.

[0033] When the calibration is completed, the driver 100 performs a predetermined operation on the display device 20 to activate a function for displaying the virtual monitor VM. When the function is activated, the control unit 24 acquires an image captured by the camera 17 from the camera 17 through the transmission / reception unit 23 (step S2). Next, the display determination unit 25 determines whether to display or not display the virtual monitor VM by the determination method described in (A) to (F) above (step S3). When the display determination unit 25 determines to display the virtual monitor VM (step S4; Yes), the control unit 24 places the virtual monitor VM at a predetermined position in the virtual space (step S5). The predetermined position is, for example, a position (gaze target position) that is a focal length away in the line of sight of the driver 100. The control unit 24 places the virtual monitor VM at, for example, a position around the gaze target position so that the virtual monitor VM is displayed at the periphery of the field of view of the display unit 22 (for example, the four corners). In addition, the control unit 24 sets the size of the virtual monitor VM so that it falls within a predetermined range of the field of view of the driver 100 (a size that does not get in the way). The display size of the virtual monitor VM (the size as seen by the driver 100, the size it occupies on the display unit 22) and where on the display unit 22 the virtual monitor VM is displayed may be configured so that the driver 100 can arbitrarily set them. For example, an input interface of a virtual object that allows the driver 100 to input these settings may be displayed in the virtual space, and the driver 100 may be able to input the display position and display size of the virtual monitor VM from this input interface. A technique that enables an operation to be performed on the input interface displayed in the virtual space in this way can be realized by MR. Also, as described above, the control unit 24 changes the size of the virtual monitor VM (the size at the arrangement position) according to the focal length of the driver 100. For example, a table correlating focal length with the display size of the virtual monitor VM may be registered in memory unit 26, and control unit 24 may calculate the focal length based on the detection value of height sensor 15 or the detection value of the eye tracking sensor of sensor unit 21 (the focal length may be set to a fixed value (e.g., several meters) rather than being calculated), and calculate the size of the virtual monitor VM based on that focal length, the display size set by driver 100, and the table registered in memory unit 26.In step S3, when the display determination unit 25 determines that the display should be hidden (step S4; No), the process proceeds to step S6.

[0034] Next, the control unit 24 judges whether or not to terminate the function of displaying the virtual monitor VM (step S6). For example, if the driver 100 performs an operation to terminate the function of displaying the virtual monitor VM, the control unit 24 judges that the function of displaying the virtual monitor VM is to be terminated, and otherwise judges that the function is not to be terminated. If it is judged that the function is not to be terminated (step S6; No), the processing from step S2 onward is repeated. If it is judged that the function is to be terminated (step S6; Yes), the processing flow of FIG. 6 is terminated.

[0035] In the above embodiment, the visibility assistance system including the camera 17 and the display device 20 is applied to the forklift 1, but the visibility assistance system can also be applied to industrial vehicles other than the forklift, such as an excavator, a bulldozer, a crane truck, and a reach stacker. Although the camera 17 is provided to capture an image in front of the forks 12, the image capturing direction of the camera 17 is not limited to the front, and may be in another direction. In the case of other industrial vehicles, the camera can be provided at any position on the working device (for example, the boom or bucket of an excavator, the bucket or arm of a bulldozer, the boom, jib, or hook of a crane, etc.) so as to capture an image in any direction.

[0036] (effect) Conventionally, a system has been provided in which a real monitor is provided in front of the driver's seat and an image of the area in front of the forks 12 is displayed on the monitor. In the case of this system, the monitor obstructs the view while driving, and since the focal distance between the real work target position and the monitor is different, a burden of switching the focus occurs when comparing the two. In contrast, in this embodiment, the driver is made to wear AR glasses (display device 20), a virtual monitor VM is displayed in a virtual space, and an image of the area in front of the forks 12 is displayed on the virtual monitor VM. As a result, for example, in a loading and unloading operation, the driver 100 can stably align the forks 12 and the pallet 42 without looking up even if the height of the shelf is high. Even if the driver 100 looks up, the driver 100 can check the work target position and the image without switching the focal distance by matching the distance to the actual object of the line of sight with the distance to the position where the virtual monitor VM is placed. In addition, not only can the virtual monitor VM be placed anywhere in the virtual space, but it can also be prevented from obstructing the view by erasing the virtual monitor VM when it is not needed. Furthermore, by arranging the virtual monitor VM so that it follows the line of sight of the driver 100, the driver 100 can check the image ahead of the fork 12 while looking at other positions, such as the periphery of the vehicle.

[0037] FIG. 7 is a diagram illustrating an example of a hardware configuration of the display device according to the embodiment. The computer 900 includes a CPU 901 , a main memory device 902 , an auxiliary memory device 903 , an input / output interface 904 , and a communication interface 905 . The above-mentioned display device 20 is implemented in a computer 900. The above-mentioned functions are stored in the auxiliary storage device 903 in the form of a program. The CPU 901 reads the program from the auxiliary storage device 903, loads it in the main storage device 902, and executes the above-mentioned processing in accordance with the program. The CPU 901 also reserves a storage area in the main storage device 902 in accordance with the program. The CPU 901 also reserves a storage area in the auxiliary storage device 903 for storing data being processed in accordance with the program.

[0038] In addition, a program for realizing all or part of the functions of the display device 20 may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing by each functional unit. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes a homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to a portable medium such as a CD, DVD, or USB, or a storage device such as a hard disk built into a computer system. In addition, if the program is distributed to the computer 900 through a communication line, the computer 900 that receives the program may deploy the program in the main storage device 902 and execute the above processing. In addition, the above program may be for realizing part of the above-mentioned functions, and may further be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0039] As described above, several embodiments according to the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

[0040] <Additional Notes> The display device, the visibility assistance system, the industrial vehicle, the control method, and the program described in each embodiment can be understood, for example, as follows.

[0041] (1) The display device of the first embodiment is a display device 20 worn on the head of a person (driver 100 or passenger), and includes a transparent display unit 22 capable of displaying image data, an acquisition unit (transmitter / receiver 23) that acquires an image captured by a camera 17 provided on a work implement (fork 12) equipped on an industrial vehicle (forklift 1), and a control unit 24 that displays the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position a focal length away in the direction of the person's line of sight, which is visible through the display unit. With this, by using AR technology to place a virtual monitor VM around a position that is a focal length away from the driver 100 in the line of sight direction, it is possible to display an image captured by the camera 17 at approximately the same focal length as the target line of sight of the driver 100 within the field of view of the driver 100. This allows the driver 100 to perform work (e.g., loading and unloading work) while checking the surroundings (e.g., forward) of the work equipment (e.g., raised forks 12) equipped on the industrial vehicle (forklift 1).

[0042] (2) A second aspect of the present invention relates to a display device according to (1), wherein the control unit changes the size of the virtual monitor in accordance with the focal length. This allows the size of the virtual monitor seen by the driver 100 to remain unchanged.

[0043] (3) A display device according to a third aspect is the display device according to (1) to (2), in which the control unit displays the image data on a periphery of the display unit. This makes it possible to ensure visibility in the central part of the display unit by displaying on the peripheral part of the display unit.

[0044] (4) A display device according to a fourth aspect is the display device according to (1) to (3), in which the control unit sets the focal length based on a state (height, position) of the working implement (fork 12). For example, by setting the focal length, that is, the position of the object, based on the detection value of a height sensor, the driver 100 can check the periphery of the working device without the burden of switching his or her line of sight.

[0045] (5) A display device according to a fourth aspect is a display device according to any one of (1) to (4), further comprising a display determination unit that determines whether to display or hide the image data, and when the display determination unit determines that the image data should be displayed, the control unit displays the image data, and when the display determination unit determines that the image data should be hidden, the control unit does not display the image data. This allows the virtual monitor VM to be displayed only when necessary. For example, the virtual monitor VM can be hidden in situations where it is obstructing the view and getting in the way.

[0046] (6) A display device according to a sixth aspect is the display device of (5), in which the display determination unit determines to display the image data when it detects an operation of the working device. This makes it possible to display the virtual monitor VM only when the operating device is in operation.

[0047] (7) A display device according to a seventh aspect is the display device according to (5) to (6), wherein the display determination unit determines whether to display or not display the image data based on a position of the working device. This allows the virtual monitor VM to be displayed only when the working device is in a specified position.

[0048] (8) The display device according to an eighth aspect is the display device according to any one of (5) to (7), wherein the display determination unit determines whether to display or not display the image data based on a line of sight of the person. This allows the virtual monitor VM to be displayed only when the driver's gaze is directed in a certain direction.

[0049] (9) A visibility assistance system according to a ninth aspect includes a camera provided on a work device provided on an industrial vehicle, and a display device according to any one of (1) to (8).

[0050] (10) An industrial vehicle according to a tenth aspect includes a camera provided on a work device provided on the industrial vehicle and a display device according to any one of (1) to (8).

[0051] (11) A control method according to an eleventh aspect is a control method for a display device worn on a person's head, having a transparent display unit capable of displaying image data, which acquires an image captured by a camera attached to a work device equipped on an industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position a focal length away in the direction of the person's line of sight, which is visible through the display unit.

[0052] (12) A program according to a twelfth aspect causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring images captured by a camera attached to a work device equipped on an industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, in the vicinity of a position a focal length away in the direction of the person's line of sight, which is visible through the display unit. [Explanation of symbols]

[0053] 1. Forklift 10. Body 11. Cargo handling equipment 12···Fork 13 Mast 14 Backrest 15. Height sensor 16. Proximity sensor 17. Camera 18. Control device 19 Lift lever 20...Display device 21 Sensor section 22...Display section 23 Transmitter / receiver 24 Control section 25...Display judgment section 26...Storage section 100 Driver VM...Virtual Monitor 900...Computer 901···CPU 902...Main memory 903...Auxiliary storage device 904 Input / Output Interface 905 Communication Interface

Claims

1. A display device that is worn on a person's head, a transparent display unit capable of displaying image data; an acquisition unit that acquires images captured by a camera provided on a work device provided on the industrial vehicle; a control unit that has a function of displaying the image data, including a virtual monitor on which the image is displayed, at any position in a virtual space so that the image data is displayed on the display unit as if the image data were located at a position in the real world corresponding to the position, and that positions the image data at a position in the virtual space corresponding to the vicinity of a position that is a focal length away in the line of sight of the person and that is visible through the display unit; A display device comprising:

2. The control unit changes the size of the virtual monitor according to the focal length. The display device according to claim 1 .

3. The control unit displays the image data on a peripheral portion of the display unit. The display device according to claim 1 or 2.

4. The control unit sets the focal length based on a state of the working device. The display device according to claim 1 or 2.

5. a display determination unit that determines whether the image data is to be displayed or not; Further equipped with The control unit displays the image data when the display determination unit determines that the image data is to be displayed, and does not display the image data when the display determination unit determines that the image data is to be hidden. The display device according to claim 1 or 2.

6. The display determination unit determines that the image data is to be displayed when the operation of the working device is detected. The display device according to claim 5 .

7. The display determination unit determines whether to display or not display the image data based on a position of the working device. The display device according to claim 5 .

8. The display determination unit determines whether to display or not display the image data based on a line of sight direction of the person. The display device according to claim 5 .

9. A camera provided on a work device of the industrial vehicle; The display device according to claim 1 or 2; A visual aid system.

10. A camera installed on a work device of an industrial vehicle The display device according to claim 1 or 2; An industrial vehicle equipped with

11. A method for controlling a display device to be worn on a person's head, the display device having a transparent display unit capable of displaying image data, comprising the steps of: Acquire images captured by a camera installed on a work device of the industrial vehicle, By using a function that allows the image data, including a virtual monitor on which the image is displayed, to be displayed on the display unit as if the image data were present at a real-world position corresponding to the position by placing the image data at an arbitrary position in a virtual space possessed by the display device, the image data is placed at a position in the virtual space corresponding to the focal length in the line of sight of the person, which is visible through the display unit; Control methods.

12. A computer of a display device mounted on a person's head, the computer having a transparent display capable of displaying image data, A means for acquiring images captured by a camera provided on a work device of the industrial vehicle; a means for arranging the image data, including a virtual monitor on which the image is displayed, at any position in a virtual space possessed by the display device, and displaying the image data on the display unit as if the image data were present at a position in the real world corresponding to the position; and a means for arranging the image data at a position in the virtual space corresponding to a focal length in the line of sight of the person, which is visible through the display unit; A program to function as a

Images