Display device, visibility assistance system, industrial vehicle, control method and program

Abstract

To provide a display device which can check the situation in front of a vehicle without further obstructing a visual field of an operator even in such a situation that the visual field in front the vehicle is obstructed by baggage and the like when the operator faces the front of the vehicle.SOLUTION: A display device, which is worn on the head of a person, comprises: a transparent display part that can display image data; an acquisition part that acquires an image of the front of an industrial vehicle photographed by a camera provided on the industrial vehicle; and a control part that displays the image data including a virtual monitor on which the image is displayed so as to be superimposed on an object loaded on the industrial vehicle that obstructs the front visual field.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a display device, a visibility assistance system, an industrial vehicle, a control method, and a program. [Background technology]

[0002] There is a system that mounts a camera on the side of the fork of the forklift or on the bottom of the mast and projects the image on a monitor installed near the driver's seat. With this system, even in a situation where the forward view is blocked by the load on the fork, the driver can check the situation ahead by checking the monitor. In such a system, the size of the monitor may be designed to be small in order to ensure a good forward view, but the driver must stare at the monitor to check the image displayed.

[0003] As a related technology, Patent Document 1 discloses an assistance device that is equipped with a camera that captures an image of an area including the tips of the forks, and superimposes a virtual fork indicating the position of the forks onto the image captured by the camera and displays it on a head-mounted display worn by the driver. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6217472 Summary of the Invention [Problem to be solved by the invention]

[0005] There is a demand for technology that allows the driver to check what is ahead without further obstructing the driver's view, even in situations where the driver's view is obstructed when facing forward of the vehicle.

[0006] The present disclosure provides a display device, a visibility assistance system, an industrial vehicle, a control method, and a program that can solve the above problems. [Means for solving the problem]

[0007] The display device of the present disclosure is a display device worn on a person's head and includes a transparent display unit capable of displaying image data, an acquisition unit that acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a control unit that displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and is visible through the display unit.

[0008] The visibility assistance system of the present disclosure includes a camera mounted on an industrial vehicle and the above-described display device.

[0009] The industrial vehicle of the present disclosure includes a camera and the above-described display device.

[0010] The control method disclosed herein is a control method for a display device worn on a person's head, having a transparent display unit capable of displaying image data, which acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and can be seen through the display unit.

[0011] In addition, the program disclosed herein causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, superimposed on an object loaded on the industrial vehicle that blocks the forward view and can be seen through the display unit. Effect of the Invention

[0012] According to the above-mentioned display device, visibility assistance system, industrial vehicle, control method, and program, even in a situation where the driver's forward visibility is obstructed when the driver faces forward of the vehicle, the driver can check what is happening ahead without further obstructing his or her visibility. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram illustrating an example of a forklift according to an embodiment. [Diagram 2] 1 is a block diagram showing an example of a display device according to an embodiment; [Figure 3A] 1A and 1B are diagrams illustrating a focal length and a size of a virtual object according to an embodiment. [Figure 3B] 10A and 10B are diagrams illustrating a relationship between the driver's posture and the position of a virtual object according to the embodiment. [Figure 4] FIG. 2 is a diagram showing a display example of an image captured by a camera according to an embodiment. [Figure 5A] FIG. 2 is a first diagram for explaining the vertical display angle and the height display position of the virtual monitor according to the embodiment. [Figure 5B] 11A is a second diagram illustrating the vertical display angle and the height display position of the virtual monitor according to the embodiment. FIG. [Figure 6A] FIG. 2 is a first diagram illustrating the horizontal display angle of the virtual monitor according to the embodiment. [Figure 6B] FIG. 2 is a second diagram illustrating the horizontal display angle of the virtual monitor according to the embodiment. [Figure 7A] FIG. 2 is a first diagram illustrating control of switching between display and non-display of a virtual monitor according to an embodiment. [Figure 7B] FIG. 11 is a second diagram illustrating control of switching between display and non-display of a virtual monitor according to an embodiment. [Figure 8] 10 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a hardware configuration of a display device according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] <Embodiment> Hereinafter, the display device of the present disclosure will be described with reference to the drawings. (composition) FIG. 1 is a schematic diagram showing an example of a forklift according to each embodiment. The forklift 1 includes a vehicle body 10, a loading device 11, a control device 18, and a display device 20. The vehicle body 10 includes a drive device and a steering device (not shown) and drives the forklift 1. The loading device 11 includes a fork 12, a mast 13 for raising and lowering the fork 12, and a backrest 14 for preventing the load from falling backward (towards the mast 13), and performs a loading operation by raising and lowering the load loaded on the fork 12. The mast 13 includes an outer mast (not shown) and an inner mast (not shown), and the inner mast is configured to rise and fall relative to the stationary outer mast. The mast 13 is provided with a load sensor 15, which detects the weight of the load loaded on the fork 12. The outer mast is also provided with a proximity sensor 16. The proximity sensor 16 detects whether the fork 12 is loaded with a load. The proximity sensor 16 is provided at a height (at a relatively high position) that can detect whether or not the load is loaded to the extent that it blocks the driver's 100's view. The loading device 11 is equipped with a camera 17 that takes pictures of the front. For example, a camera 17a is mounted on the upper part of the mast 13 facing diagonally downward so as to take pictures of the tips of the forks 12. Also, for example, a camera 17b is provided on the tips of the forks 12. The camera 17b takes pictures of the front of the forks 12. The loading device 11 may be equipped with both the cameras 17a and 17b, or only one of them. In the following, as an example, the camera 17a is described as being equipped. Images (which may be video or still images) taken by the camera 17a are transmitted to the display device 20. The control device 18 controls the operation of the forklift 1. For example, when the driver 100 operates the handlebars 19a or the lift lever 19b, the control device 18 detects the operation and controls the traveling direction of the vehicle and raises and lowers the mast 13. The control device 18 is connected to the load sensor 15 and the proximity sensor 16, and acquires information detected by these sensors. The control device 18 transmits information such as the information detected by the load sensor 15 and the proximity sensor 16 and a signal indicating that the driver 100 has operated the handlebars 19a or the lift lever 19b to the display device 20.The display device 20 is an AR (Augmented Reality) glass or an MR (Mixed Reality) glass. The driver 100 wears the display device 20 and drives the forklift 1. In this specification, the direction in which the loading device 11 of the forklift 1 is provided as shown in Fig. 1 is called the front, the opposite direction is called the rear, and the left and right directions when facing forward are called the left side and the right side, respectively.

[0015] 2 is a block diagram showing an example of a display device according to an embodiment. The display device 20 is an AR glass or an MR glass including a display device and a computer (hereinafter, both are referred to as AR glass). The display device 20 is communicably connected to the control device 18 and the camera 17, and acquires information from these devices. The display device 20 displays an image captured by the camera 17a. As shown in the figure, the display device 20 includes a sensor unit 21, a display unit 22, a transmission / reception unit 23, a control unit 24, a display determination unit 25, and a storage unit 26.

[0016] The sensor unit 21 includes a sensor for head tracking of the driver 100 (for example, a plurality of visible light cameras and an IMU (Inertial Measurement Unit)), a sensor for eye tracking of the driver 100 (for example, an infrared camera), a depth sensor, a microphone, and the like.

[0017] The display unit 22 constitutes a part of the lens (screen) of the AR glasses, and is constituted by, for example, a transparent holographic display (the outside world is visible through the lens). In a state where a virtual object is not projected onto the lens, the driver 100 can see the real world as it is. The virtual object is image data projected onto the display unit 22. When a virtual object is displayed on the display unit 22, the driver 100 sees the virtual object superimposed on the real world.

[0018] The transmission / reception unit 23 transmits and receives data to and from other devices. For example, the transmission / reception unit 23 transmits and receives data to and from other devices by wireless communication such as WiFi (registered trademark) or Bluetooth (registered trademark). For example, the transmission / reception unit 23 receives an image captured by the camera 17. For example, the transmission / reception unit 23 receives, from the control device 18, information detected by the load sensor 15 and the proximity sensor 16, a signal indicating that the driver 100 has operated the handlebars 19a or the lift lever 19b, and the like.

[0019] The control unit 24 performs display control of the display unit 22, etc. The control unit 24 recognizes the spatial shape around the driver 100 based on the information detected by the sensor unit 21, creates three-dimensional map information of the surrounding space, and generates a virtual space. The control unit 24 places a virtual object at a desired position in the virtual space based on the created map information. The virtual object is displayed to the driver 100 superimposed on an object existing at that position in the real world. The virtual object can be displayed at (1) a predetermined position in the space (the virtual object exists at a fixed position), (2) a predetermined position based on an object existing in the surrounding space (for example, if the object is the vehicle body 10, the virtual object moves following the running of the vehicle body 10), or (3) a predetermined position based on the driver 100 (for example, the virtual object exists at a position a predetermined distance away in the line of sight of the driver 100), etc. In the case of (1) or (2), the virtual object is recognized by the driver 100 as if it actually exists at that position. The virtual object can be placed at any position in the virtual space, and the placement position can be freely changed by moving it closer to or farther away from the driver 100. For example, if the virtual object is placed far away, the driver 100 sees the virtual object as being far away, and if the virtual object is placed close to the driver 100, the driver 100 sees the virtual object as being close to the driver. In addition, the control unit 24 calculates the position of the driver 100's head when the driver 100 assumes a posture such as bending or leaning to the right or left from the driver's seat, based on the information detected by the sensor unit 21. Then, the control unit 24 generates and displays a virtual object based on the positional relationship between the driver 100's head and the virtual object so that the virtual object appears to be in the same way as when it actually exists in the real world.

[0020] The AR glasses have the functions of freely arranging virtual objects, arranging a virtual object at a certain position in the virtual space so that the virtual object appears to exist at that position, and changing the appearance of the virtual object according to the posture of the driver 100. The user sets the desired virtual object for the display device 20 (a developer uses an AR development tool to develop a virtual object and implements the developed program on the display device 20). Then, the control unit 24 places the virtual object at the desired position according to the setting. In this embodiment, the control unit 24 displays the virtual object at any position within the range occupied by the luggage loaded on the fork 12 (corresponding to (2) above). For example, if a virtual object is placed in the range where the luggage is loaded on the fork 12, the virtual object can be displayed superimposed on the luggage. Since the view of the range where the luggage is loaded is originally blocked by the luggage, the view of the driver 100 is not further deteriorated even if the virtual object is displayed superimposed on the luggage. In this embodiment, a virtual monitor VM that displays an image captured by the camera 17a as a virtual object is placed at any position within the range occupied by the luggage. As a result, even in a situation where the forward view is obstructed by luggage, the image of the forward view captured by the camera 17a can be displayed within the visual field of the driver 100 without obstructing the view.

[0021] 3A shows a virtual object 31a arranged at a distance L1 forward from the driver 100, a display range (range of the visual field that fits in the display unit 22) 31b at the distance L1 forward, a virtual object 32a arranged at a distance L2 forward from the driver 100, and a display range 32b at the distance L2 forward. If a real monitor that is not a virtual object is arranged on the rear side of the mast 13 (for example, the distance L2 forward from the driver 100), the monitor will be hidden by the mast 13 and will not be visible, but in the case of an AR virtual monitor VM, even if it is arranged on the rear side of the mast 13, the virtual monitor VM will not be hidden by the mast because it is displayed on the AR glasses. The AR technology also has an occlusion function that displays the virtual monitor VM as if it is hidden by the mast 13, but in this embodiment, the occlusion function is not used because priority is given to information displayed on the virtual monitor VM. As shown in FIG. 3, when two virtual objects 31a and 32a of different sizes (virtual object 32a has a larger size in the virtual space) are placed at positions of distances L1 and L2, respectively, due to the nature that the farther an object is, the smaller it appears, it is possible to make both appear to be the same size to the driver 100. In other words, when a virtual object is placed farther away, the size of the virtual object is made larger, and when it is placed closer, the size of the virtual object is made smaller. The specific size of the virtual object can be calculated based on the placement position (distance from the driver 100) and the size at which it is displayed on the display unit 22, and by changing the size of the virtual object according to the distance from the driver 100, it is possible to make the virtual object always appear to the driver 100 at a constant size. Assuming that the virtual object is not hidden by the mast 13 and is not visible, the difference between the two virtual objects 31a and 32a displayed to the driver 100 at the same size is only the placement position, in other words, the focal length. By controlling the placement position and size of the virtual object placed in the virtual space in this way, it is possible to freely change only the focal length without changing the display size of the virtual object as seen by the driver 100.

[0022] By utilizing this property, when the virtual monitor VM is placed near the tip of the fork 12, the focus when viewing the tip of the real fork 12 and when viewing the virtual monitor VM are almost the same, and the driver 100 can view both at the same time. In other words, in order to show the image captured by the camera 17a without forcing the driver 100 to focus, it is preferable to place the virtual object as close as possible to the focal position of the driver 100. When moving the forklift 1 forward with a load loaded, the driver 100 may want to check a shelf or the like further forward of the tip of the fork 12 from the driver's seat. In this case, if the virtual monitor VM is also placed slightly forward of the tip of the fork 12, the virtual monitor VM appears to be present at that position. For the driver 100 who is visually checking the area further forward of the tip of the fork 12, the virtual object appears to be present at the exact position where the driver's line of sight is aligned, so there is no need to refocus in order to check the virtual monitor VM. However, in a similar situation where the forklift 1 is moving forward with a load, as shown in FIG. 3B, the driver 100 may want to lean forward to the right and check the positional relationship between the load and a shelf or the like located further forward from the side. In this case, if the virtual monitor VM is placed in front of the tip of the fork 12, the shelf or the like to be checked and the virtual monitor VM will appear to overlap, making it difficult to check the positional relationship between the load and the shelf. Therefore, in this embodiment, the virtual monitor VM is placed in front of the tip of the fork 12 so as not to block the driver 100's field of vision even when the driver 100 takes a posture as shown in FIG. 3B. Also, if the virtual monitor VM is placed too far forward (for example, VM1 in FIG. 3B), the virtual monitor VM will fall out of the field of vision range H1 when the driver 100 takes a posture as shown in FIG. 3B, and the driver 100 must move his / her line of sight to check the virtual monitor VM1. To prevent this from happening, in this embodiment, the virtual monitor VM is placed in front of the backrest 14. That is, the virtual monitor VM is placed at any position in the range H2. This allows the image captured by the camera 17a to be displayed within the field of view of the driver 100 without blocking the driver's 100 field of view, even if some effort is required for focusing.FIG. 4 shows an example of the virtual monitor VM displayed on the display unit 22. As shown in FIG.

[0023] Next, control of the vertical display angle and height display position of the virtual monitor VM will be described with reference to FIGS. 5A and 5B. The display surface P in FIG. 5A indicates the display position in the front-rear direction and the angle in the vertical direction of the virtual monitor VM. The control unit 24 places the virtual monitor VM at a predetermined position in the range H2, and tilts the display surface of the virtual monitor VM upward by an angle θ so as to be perpendicular to the line of sight of the driver 100. The line of sight of the driver 100 may be calculated using the eye tracking function of the sensor unit 21, or a predetermined angle may be set according to the head position of the driver 100. In addition, the size (vertical size) of the virtual monitor VM may be set as large as possible so that the driver 100 can better recognize the state behind the luggage 50 as seen from the driver 100 (for example, size 51 in FIG. 5A), or may be set to a size that matches the size of the luggage so that the driver 100's field of vision is not obstructed even a little by the virtual monitor VM (for example, size 52 in FIG. 5A). The vertical size of the virtual monitor VM may be determined in advance based on the size of an average load of luggage in a warehouse or the like, or, for example, the control unit 24 may recognize the boundary between luggage 50 and other areas (e.g., the ground) from an image recognized by a visible light camera provided in the sensor unit 21, calculate the size of the virtual monitor VM such that the upper end of the virtual monitor VM seen from the visible light camera does not exceed corner 53 of luggage 50, and display the virtual monitor VM at the calculated size. Also, even if the virtual monitor VM exceeds the range of the luggage, the image captured by the camera 17a may be displayed on the virtual monitor VM as a see-through image (an image displayed in such a way that the real world on the other side of the image can be seen through when viewed from the driver 100) so that the real world on the other side of the virtual monitor VM can be recognized.

[0024] FIG. 5B shows an example of the display of the virtual monitor VM when the driver 100 is crouching. When the driver 100 is in a crouching state, the control unit 24 slides the virtual monitor VM downward to reduce the size in the vertical direction or reduce the overall size. In this case, the control unit 24 also adjusts the angle of the display surface of the virtual monitor VM so that it is perpendicular to the line of sight of the driver 100. The control unit 24 may set the size of the virtual monitor VM as large as possible (for example, size 51' in FIG. 5B) or may set it to a size that matches the size of the luggage (for example, size 52' in FIG. 5A).

[0025] Next, control of the horizontal display angle of the virtual monitor VM will be described with reference to FIGS. 6A and 6B. The display surface P in FIG. 6A indicates the display position in the front-rear direction and the horizontal angle of the virtual monitor VM. The control unit 24 places the virtual monitor VM at a predetermined position in the range H2 and directs the display surface of the virtual monitor VM so as to be perpendicular to the line of sight of the driver 100. The control unit 24 may set the size (horizontal size) of the virtual monitor VM as large as possible (for example, size 61 in FIG. 6A) or may set it to a size that fits within the luggage 60 (for example, size 62 in FIG. 5A). The horizontal size of the virtual monitor VM may be determined in advance based on the average size of luggage loaded in a warehouse or the like, or, for example, the control unit 24 may recognize the boundary between the luggage 60 and other objects from an image recognized by a visible light camera provided in the sensor unit 21, calculate a size that fits within the luggage 60, and display the virtual monitor VM at the calculated size. The control unit 24 may also display an image captured by the camera 17a as a transparent image.

[0026] FIG. 6B shows an example of the display of the virtual monitor VM when the driver 100 leans to the right side of the driver's seat. When the driver 100 moves to the right side, the control unit 24 adjusts the horizontal angle of the virtual monitor VM around point O according to the driver's 100 posture. For example, the control unit 24 may rotate the virtual monitor VM in the horizontal direction so that the line of sight of the driver 100 and the display surface of the virtual monitor VM are perpendicular to each other, or may rotate the virtual monitor VM counterclockwise around point O by a predetermined angle according to the amount of movement of the driver 100 to the right. The control unit 24 may set the size of the virtual monitor VM as large as possible (for example, size 61' in FIG. 5B) or may set the size according to the size of the luggage (for example, size 62' in FIG. 5A). However, when the size of the virtual monitor VM is displayed large, the control unit 24 does not display the virtual monitor VM in an area overlapping with the illustrated range 63' so as not to obstruct the view toward the tip of the fork 12. For example, the control unit 24 may hide a predetermined range on the right side of the virtual monitor VM, shrink the size in the horizontal direction, or slide the display position in the direction of the arrow 64' in accordance with the amount of rightward movement of the driver 100 so as not to block the field of view of the range 63'. The control unit 24 may also hide a range on the opposite side (left side) of the virtual monitor VM that is outside the angle of view of the camera 17a.

[0027] In this way, the position, angle, and range of the virtual monitor VM displayed on the display unit 22 are controlled according to the posture of the driver 100. Also, the display and non-display of the virtual monitor VM can be switched automatically or by the movement or operation of the driver 100. For example, in a scene where the virtual monitor VM is an obstacle, the virtual monitor VM can be hidden.

[0028] The display determination unit 25 determines whether to display or hide the virtual monitor VM. When the display determination unit 25 determines to display the virtual monitor VM, the control unit 24 displays (places) the virtual monitor VM, and when the display determination unit 25 determines to hide the virtual monitor VM, the control unit 24 hides the virtual monitor VM (does not place the virtual monitor VM in the virtual space). For example, the display determination unit 25 determines to display the virtual monitor VM when the front of the vehicle is obscured by luggage, and determines to hide the virtual monitor VM otherwise. An example of the determination process of the display determination unit 25 is described below.

[0029] <When automatically detecting when the vehicle is unable to see ahead> (A) The display determination unit 25 obtains information detected by the proximity sensor 16 (detects whether the outer mast on which the proximity sensor 16 is provided is in close proximity to the luggage loaded on the forks 12. If no luggage is loaded, it is considered that no luggage is loaded until it reaches a height that can be detected by the proximity sensor 16) from the control device 18 via the transmission / reception unit 23. If the proximity sensor 16 detects that the outer mast is in close proximity to the luggage, the display determination unit 25 determines to display the virtual monitor VM, and if not, determines to hide the virtual monitor VM.

[0030] (B) The display determination unit 25 acquires information detected by the load sensor 15 (weight of the baggage loaded on the forks 12) from the control device 18 via the transmission / reception unit 23. When the weight detected by the load sensor 15 is equal to or greater than a predetermined threshold, the display determination unit 25 determines to display the virtual monitor VM, and when the height is less than the threshold, determines to hide the virtual monitor VM.

[0031] (C) The sensor unit 21 takes an image of the vicinity of the mast 13 when no luggage is loaded using a visible light camera or the like, and records the image or the shape of the vicinity of the mast 13 when no luggage is loaded in the storage unit 26. The display determination unit 25 then compares the image of the vicinity of the mast 13 taken by the sensor unit 21 with the image recorded in the storage unit 26 each time, and determines whether luggage is loaded based on, for example, the difference between the images. For example, if the average of the differences of each pixel in the height range that blocks the view of the captured image is equal to or greater than a threshold, the display determination unit 25 determines that luggage is loaded up to that height. If it is determined that luggage is loaded, the display determination unit 25 determines that the virtual monitor VM is to be displayed, and if not, determines that the virtual monitor VM is to be hidden.

[0032] <When display / hide is determined by the driver's operation> (D) The control unit 24 displays an operation panel (virtual object) in a virtual space based on, for example, a gesture or voice of the driver 100. This operation panel is provided with a virtual switch that switches between displaying and hiding the virtual monitor VM. When the driver 100 instructs display by operating the virtual switch, the display determination unit 25 determines that the virtual monitor VM is to be displayed. When the driver 100 instructs not to display by operating the virtual switch, the display determination unit 25 determines that the virtual monitor VM is to be hidden. A technology that enables operations to be performed on an operation panel displayed in a virtual space in this way can be realized by MR.

[0033] (E) A button for switching between displaying and hiding the virtual monitor VM is provided on the handle 19a or the lift lever 19b, and when this button is operated, the display determination unit 25 determines whether to display or hide the virtual monitor VM.

[0034] (F) A two-dimensional marker (e.g., a QR code (registered trademark)) is attached to the front pipe of the head guard of the mast 13 (driver's seat side of the outer mast). For example, as illustrated in FIG. 7A, a two-dimensional marker 71 including information corresponding to display (ON) is attached to the mast 13 on the right side as viewed from the driver's side, and a two-dimensional marker 72 including information corresponding to non-display (OFF) is attached to the mast 13 on the left side as viewed from the driver's side. When the driver 100 gazes at either of the two-dimensional markers 71, 72 (for example, for a predetermined period of time), the display determination unit 25 determines to display or hide the virtual monitor VM. Whether the driver 100 is gazing at either of the two-dimensional markers 71, 72 can be detected, for example, by an eye tracking sensor of the sensor unit 21. It may be possible to recognize whether the display or non-display has been instructed based on whether the user gazed simply at the 2D marker 71 on the right side or the 2D marker 72 on the left side. However, in order to prevent erroneous operation, it is preferable to make the information contained in the left and right 2D markers different, and the display determination unit 25 determines whether the display or non-display is to be performed based on the content read from the 2D marker. Also, the switching between display and non-display using the 2D marker may be enabled only when it is determined that luggage is loaded to a certain height by the above-mentioned detection methods (A) to (C). This makes it possible to prevent the virtual monitor VM from being displayed when the user happens to gaze at the 2D marker 71 that corresponds to the display even when no luggage is loaded, or when the virtual monitor VM is not needed even when luggage is loaded.

[0035] (G) Also, instead of attaching a QR code (registered trademark) to the mast 13 or the like, a virtual switch for display and non-display may be placed near a position corresponding to the mast 13 in the virtual space. For example, as illustrated in FIG. 7B, a virtual switch 71' for instructing display (ON) is placed on the right side, and a virtual switch 72' for instructing non-display (OFF) is placed on the left side. When the driver 100 gazes at any of the virtual switches, the display determination unit 25 determines that the virtual monitor VM is to be displayed or hidden. In this case, too, the switching between display and non-display by the virtual switch may be enabled only when it is determined that luggage is loaded to a certain height by the detection methods (A) to (C) described above.

[0036] (H) Furthermore, the display determination unit 25 may determine whether to display or hide the virtual monitor VM by voice recognition. For example, when the driver 100 utters "display," the microphone of the sensor unit 21 detects the utterance, and the display determination unit 25 recognizes through voice recognition that an instruction to display the virtual monitor VM has been issued. The display determination unit 25 then determines that the virtual monitor VM is to be displayed. The same applies to the case of not displaying the virtual monitor VM. In this case, too, the switching between display and non-display using the virtual switch may be enabled only when it is determined that luggage is loaded to a certain height by the detection methods (A) to (C) described above.

[0037] The storage unit 26 stores information detected by the sensor unit 21, information received by the transmission / reception unit 23, virtual objects, CAD information of the forklift 1 or various types of forklifts, and the like.

[0038] (operation) Next, control of the display device 20 will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of display control of the virtual monitor according to the embodiment. When the driver 100 wearing the display device 20 sits in the driver's seat of the forklift 1, the driver starts up the display device 20 and performs calibration (step S1). In the calibration, the positional relationship between the driver 100 and the forklift 1 is calculated. For example, the control unit 24 recognizes the handle and other structures of the forklift 1 using the visible light camera of the sensor unit 21, and estimates the type of vehicle of the forklift 1 from the shape and appearance of the photographed handle, etc. The control unit 24 reads out CAD information corresponding to the estimated type of vehicle from the CAD information recorded in the storage unit 26, and grasps the overall shape of the forklift 1. Furthermore, for example, the control unit 24 calculates the positional relationship between the driver 100 and the steering wheel, that is, where the driver 100 sits in the driver's seat, how high the driver 100's head is, or the positional relationship between the driver 100 and each part of the forklift 1 (horizontal distance, vertical distance, etc.) from the read CAD information of the forklift 1 by analyzing an image of the steering wheel captured by the visible light camera of the sensor unit 21 or measuring the distance to the steering wheel by the depth sensor of the sensor unit 21. This makes it possible to grasp the exact distance between the driver 100 and the tip of the fork 12 or the backrest 14, and to calculate, for example, the position and angle of the virtual monitor VM, and the position and angle of the virtual monitor VM when the driver 100 leans to the right. In addition, for example, a two-dimensional marker may be provided at a predetermined position on the forklift 1 (e.g., the outer mast), the two-dimensional marker may be recognized by the sensor unit 21, and the positional relationship between the driver 100 and each part of the forklift 1 may be calculated in a similar manner to that described above based on the positional relationship with the two-dimensional marker, the type of forklift 1 indicated by the two-dimensional marker, and CAD information for that type of vehicle.

[0039] When the calibration is completed, the driver 100 performs a predetermined operation on the display device 20 to activate a function for displaying the virtual monitor VM. When the function is activated, the control unit 24 acquires an image captured by the camera 17 from the camera 17 through the transmission / reception unit 23 (step S2). Next, the display determination unit 25 determines whether to display or not display the virtual monitor VM by the determination method described in (A) to (G) above (step S3). When the display determination unit 25 determines to display the virtual monitor VM (step S4; Yes), the control unit 24 places the virtual monitor VM at a predetermined position in the virtual space (step S5). The predetermined position is, for example, between the tip of the fork 12 and the backrest 14 in the front-rear direction, and is a range that overlaps with the height of the luggage in the vertical direction. The height of the luggage may be grasped by a predetermined image recognition from an image captured by the camera 17a or the visible light camera of the sensor unit 21, or a certain height may be set in advance. The control unit 24 displays the virtual monitor VM, for example, superimposed on the luggage loaded on the fork 12, in as large a size as possible. Furthermore, as described with reference to Figures 5A to 6B, the control unit 24 changes the direction of the display surface of the virtual monitor VM according to the line of sight of the driver 100, and changes the vertical or horizontal position and display size of the virtual monitor VM according to the posture of the driver 100. In step S3, when the display determination unit 25 determines that the display should be hidden (step S4; No), the process proceeds to step S6.

[0040] Next, the control unit 24 judges whether or not to terminate the function of displaying the virtual monitor VM (step S6). For example, if the driver 100 performs an operation to terminate the function of displaying the virtual monitor VM, the control unit 24 judges that the function of displaying the virtual monitor VM is to be terminated, and otherwise judges that the function is not to be terminated. If it is judged that the function is not to be terminated (step S6; No), the processing from step S2 onward is repeated. If it is judged that the function is to be terminated (step S6; Yes), the processing flow of FIG. 8 is terminated.

[0041] In the above embodiment, the visibility assistance system including the camera 17 and the display device 20 is applied to the forklift 1 as an example. However, the visibility assistance system can also be applied to industrial vehicles other than forklifts, such as excavators, bulldozers, cranes, and reach stackers.

[0042] (effect) Conventionally, a system has been provided in which a real monitor is provided in front of the driver's seat and an image of the front of the fork 12 is displayed on the monitor. In the case of this system, the monitor obstructs the view during driving, and the driver must stare at the monitor to check the image displayed on the real monitor, which is designed to be small to ensure a forward field of view. In contrast, in this embodiment, the driver is made to wear AR glasses (display device 20), a virtual monitor VM is placed in a virtual space, and an image of the front of the fork 12 or the like is displayed on the virtual monitor VM. In addition, since the virtual monitor VM can be placed at any position, it is possible to superimpose it on the luggage blocking the view in front of the vehicle, thereby enabling the driver 100 to check the front without blocking the view any further. In addition, since the virtual monitor VM can be displayed larger than the real monitor, the driver 100 does not need to stare at it while facing the front of the vehicle, and can check the image of the front projected on the luggage. In addition, by placing the virtual monitor VM somewhat forward (between the backrest 14 and the tip of the fork 12), the focal length can be set farther than when a real monitor is provided in the driver's seat. This makes it easier for the driver 100 who drives the forklift 1 while visually checking the surroundings of the vehicle to focus, and makes it possible to prevent the driver 100 from being burdened by checking the image displayed on the virtual monitor VM. Also, by turning off the virtual monitor VM when it is not needed, it is possible to prevent it from interfering with the driver's view.

[0043] FIG. 9 is a diagram illustrating an example of a hardware configuration of the display device according to the embodiment. The computer 900 includes a CPU 901 , a main memory device 902 , an auxiliary memory device 903 , an input / output interface 904 , and a communication interface 905 . The above-mentioned display device 20 is implemented in a computer 900. The above-mentioned functions are stored in the auxiliary storage device 903 in the form of a program. The CPU 901 reads the program from the auxiliary storage device 903, loads it in the main storage device 902, and executes the above-mentioned processing in accordance with the program. The CPU 901 also reserves a storage area in the main storage device 902 in accordance with the program. The CPU 901 also reserves a storage area in the auxiliary storage device 903 for storing data being processed in accordance with the program.

[0044] In addition, a program for realizing all or part of the functions of the display device 20 may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing by each functional unit. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes a homepage providing environment (or display environment). In addition, the term "computer-readable recording medium" refers to a portable medium such as a CD, DVD, or USB, or a storage device such as a hard disk built into a computer system. In addition, if the program is distributed to the computer 900 through a communication line, the computer 900 that receives the program may deploy the program in the main storage device 902 and execute the above processing. In addition, the above program may be for realizing part of the above-mentioned functions, and may further be capable of realizing the above-mentioned functions in combination with a program already recorded in the computer system.

[0045] As described above, several embodiments according to the present disclosure have been described, but all of these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents as described in the claims, as well as in the scope and gist of the invention.

[0046] <Additional Notes> The display device, the visibility assistance system, the industrial vehicle, the control method, and the program described in each embodiment can be understood, for example, as follows.

[0047] (1) The display device of the first embodiment is a display device 20 worn on the head of a person (driver 100 or passenger), and includes a transparent display unit 22 capable of displaying image data, an acquisition unit (transmitter / receiver 23) that acquires an image of the front of the industrial vehicle captured by a camera 17 mounted on the industrial vehicle, and a control unit 24 that displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object (luggage) carried by the industrial vehicle that blocks the forward view and is visible through the display unit. Using AR technology, an object blocking the driver's 100 forward view is utilized to display an image of the front of the industrial vehicle captured by a camera thereon, allowing the driver to check what is ahead even when the driver's forward view is blocked.

[0048] (2) A second aspect of the display device is the display device of (1), in which the control unit changes the position and / or angle at which the image data is displayed depending on the posture of the person. This makes it easier for the driver 100 to check the virtual monitor.

[0049] (3) A display device according to a third aspect is the display device according to (1) to (2), wherein the control unit controls the area in which the image data is displayed so that the area does not extend beyond the object. This prevents the virtual monitor VM from obscuring your view of what's ahead.

[0050] (4) A display device according to a fourth aspect is a display device according to (1) to (3), further comprising a display determination unit that determines whether to display or hide the image data, and when the display determination unit determines that the image data should be displayed, the control unit displays the image data, and when the display determination unit determines that the image data should be hidden, the control unit does not display the image data. This allows the virtual monitor VM to be displayed only when necessary. For example, the virtual monitor VM can be hidden in situations where it is obstructing the view and getting in the way.

[0051] (5) A display device according to a fifth aspect is the display device of (4), in which the display determination unit determines to display the image data when it detects that the industrial vehicle is holding the object. This makes it possible to display the virtual monitor VM only when the operating device is in operation. When an industrial vehicle has an object blocking the view, it becomes difficult to see ahead, so the virtual monitor is displayed when it detects that an object blocking the view is detected. This automatically displays the virtual monitor VM when the view ahead is blocked, making it possible to check the view ahead on the virtual monitor VM.

[0052] (6) A sixth aspect of the display device is a display device according to any one of (4) to (5), wherein the display determination unit detects that the industrial vehicle is carrying the object based on any one of a change in weight detected by a mechanism for the industrial vehicle to carry the object, a change in an image captured around the mechanism, and detection information from a proximity sensor provided around the mechanism for detecting the approach of the object. This makes it possible to detect whether the industrial vehicle has an object blocking the view.

[0053] (7) A seventh aspect of the display device is a display device according to any one of (4) to (6), wherein the control unit displays a virtual switch that accepts an instruction to switch between displaying and hiding the image data, and the display determination unit determines whether to display or hide the image data based on the person's operation of the switch. This makes it possible to toggle the visibility of virtual monitors on and off without having to set up a hardware switch.

[0054] (8) A display device according to an eighth aspect is a display device according to any one of (4) to (7), wherein the display determination unit determines whether to display or hide the image data based on whether the person is staring at a first marker associated with the display of the image data or a second marker associated with the non-display of the image data. This allows the driver to switch the virtual monitor on and off by simply looking at the screen, even if their hands are occupied.

[0055] (9) A visibility assistance system according to a ninth aspect includes a camera provided on an industrial vehicle and a display device according to any one of (1) to (8).

[0056] (10) An industrial vehicle according to a tenth aspect includes a camera and a display device according to any one of (1) to (8).

[0057] (11) A control method according to an eleventh aspect is a control method for a display device worn on a person's head, having a transparent display unit capable of displaying image data, which acquires an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and displays the image data, including a virtual monitor on which the image is displayed, superimposed on an object carried by the industrial vehicle that is visible through the display unit and blocks the view ahead.

[0058] (12) A program according to a twelfth aspect causes a computer of a display device worn on a person's head, having a transparent display unit capable of displaying image data, to function as a means for acquiring an image of the front of the industrial vehicle captured by a camera mounted on the industrial vehicle, and a means for displaying the image data, including a virtual monitor on which the image is displayed, superimposed on an object carried by the industrial vehicle that is visible through the display unit and blocks the view ahead. [Explanation of symbols]

[0059] 1. Forklift 10. Body 11. Cargo handling equipment 12···Fork 13 Mast 14 Backrest 15...Load sensor 16. Proximity sensor 17. Camera 18. Control device 19a···Handle 19b Lift lever 20...Display device 21 Sensor section 22...Display section 23 Transmitter / receiver 24 Control section 25...Display judgment section 26...Storage section 100 Driver VM...Virtual Monitor 900...Computer 901··CPU 902...Main memory 903...Auxiliary storage device 904 Input / Output Interface 905 Communication Interface

Claims

1. A display device that is worn on a person's head, a transparent display unit capable of displaying image data; an acquisition unit that acquires an image of a front portion of the industrial vehicle captured by a camera provided on the industrial vehicle; a control unit that displays the image data including a virtual monitor on which the image is displayed, superimposed on an object that is loaded on the industrial vehicle and blocks a forward view and that is visible through the display unit; A display device comprising:

2. The control unit changes a position and / or an angle at which the image data is displayed in accordance with the posture of the person. The display device according to claim 1 .

3. The control unit controls a range in which the image data is displayed so as not to extend beyond the object. The display device according to claim 1 or 2.

4. a display determination unit that determines whether the image data is to be displayed or not; Further equipped with The control unit displays the image data when the display determination unit determines that the image data is to be displayed, and does not display the image data when the display determination unit determines that the image data is to be hidden. The display device according to claim 1 or 2.

5. the display determination unit determines to display the image data when detecting that the industrial vehicle has the object. The display device according to claim 4.

6. the display determination unit detects that the industrial vehicle has the object based on any one of a change in weight detected by a mechanism for the industrial vehicle to have the object, a change in an image captured around the mechanism, and detection information by a proximity sensor provided around the mechanism to detect the approach of the object. The display device according to claim 5 .

7. the control unit displays a virtual switch that receives an instruction to switch between displaying and not displaying the image data; the display determination unit determines whether to display or not display the image data based on the person's operation of the switch. The display device according to claim 4.

8. the display determination unit determines whether the image data is displayed or not based on whether the person gazes at a first marker associated with display of the image data or a second marker associated with non-display of the image data; The display device according to claim 4.

9. A camera installed in an industrial vehicle; The display device according to claim 1 or 2; A visual aid system.

10. Camera and The display device according to claim 1 or 2; An industrial vehicle equipped with

11. A method for controlling a display device to be worn on a person's head, the display device having a transparent display unit capable of displaying image data, comprising the steps of: An image of a front portion of the industrial vehicle is acquired by a camera installed in the industrial vehicle; The image data including the virtual monitor on which the image is displayed is displayed in a manner superimposed on an object that is loaded on the industrial vehicle and blocks the forward view and is visible through the display unit. Control methods.

12. A computer of a display device mounted on a person's head, the computer having a transparent display capable of displaying image data, A means for acquiring an image of a front portion of the industrial vehicle captured by a camera provided on the industrial vehicle; a means for displaying the image data including a virtual monitor on which the image is displayed, superimposed on an object that is loaded on the industrial vehicle and blocks the forward view and is visible through the display unit; A program to function as a

Images