Head-mounted display device

Abstract

An object of the present invention is to provide a head-mounted display device that can prevent danger to the user when wearing the head-mounted display device. Therefore, a video transmission-type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit includes: a second image input unit that acquires an image of the wearer's eye state; and a control unit that, based on the image of the wearer's eye state acquired by the second image input unit, determines that the wearer is awake if the eyes have not been closed for a predetermined time or longer, determines that the wearer is asleep if the eyes have been closed for a predetermined time or longer, and displays the external image acquired by the first image input unit on the display unit when the wearer transitions from sleep to awake.

Description

Technical Field

[0001] The present invention relates to a head-mounted display device (hereinafter referred to as HMD: Head Mount Display). Background Art

[0002] In recent years, the use of HMDs equipped with a display and worn on the head to provide visual and audio information has been increasing in the fields of gaming and work assistance.

[0003] Patent Document 1 is a background technology in this technical field. Patent Document 1 discloses a head-mounted display device comprising: a display unit configured to display an image based on an image signal on a display screen while transmitting external light, configured to be wearable on a user's head so as to control the ratio of the image display portion to the external light transmitting portion; a detection unit configured to detect a direction intended for viewing by the user or the posture or position of the user's head; and a display control unit configured to control the ratio of the image display portion to the external light transmitting portion on the display screen of the display unit based on the detection by the detection unit.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 8-328512 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] In Patent Document 1, a user wearing the HMD can view an image displayed on the HMD and external images according to the direction the user wants to observe or the posture or position of the user's head.

[0009] However, there is a possibility that a user might fall asleep while wearing an HMD and then forget they are wearing it upon waking up or becoming partially awake. In this case, the HMD blocks the user's field of vision, preventing them from seeing their surroundings. Consequently, the user loses their perspective and, in a panic, could potentially lead to a dangerous situation. Patent Document 1 does not consider this situation.

[0010] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an HMD that can avoid danger to the user when wearing the HMD.

[0011] Solutions for solving problems

[0012] To give an example, the present invention is a video transmission type head-mounted display device that uses a first image input unit to obtain an external image and displays it on a display unit. The head-mounted display device is constructed as follows: a second image input unit that obtains an image of the state of the wearer's eyes; and a control unit. The control unit determines that the wearer is in an awake state if the state of closing the eyes does not continue for more than a predetermined time based on the image of the state of the wearer's eyes obtained by the second image input unit; and determines that the wearer is in a sleeping state if the state of closing the eyes continues for more than a predetermined time. When the wearer changes from a sleeping state to a waking state, the control unit displays the external image obtained by the first image input unit on the display unit.

[0013] Effects of the Invention

[0014] According to the present invention, it is possible to provide an HMD that can avoid danger to a user when wearing the HMD. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a block diagram showing the hardware configuration of the HMD in the first embodiment.

[0016] Figure 2 This is a diagram showing the appearance and wearing of the HMD in Example 1.

[0017] Figure 3 This is a functional configuration diagram of the HMD in Example 1.

[0018] Figure 4 This is a flowchart of the HMD process in the first embodiment.

[0019] Figure 5 This is a flowchart of the HMD process in the second embodiment.

[0020] Figure 6 This is an example of a warning message display in the second embodiment.

[0021] Figure 7 This is a flowchart of the HMD process in the third embodiment.

[0022] Figure 8 This is a flowchart of the HMD process in the fourth embodiment.

[0023] Figure 9 This is an example of a warning message displayed by the HMD in the fourth embodiment.

[0024] Figure 10 This is an example of the external world display of the HMD in the fourth embodiment.

[0025] Description of Reference Numerals

[0026] 100: HMD (Head Mounted Display), 101: Main Control Unit, 103: RAM, 110: Storage Unit, 120: Operation Input Unit, 121: Operation Key, 122: Touch Sensor, 130: Image Processing Unit, 131: Display Unit, 132: Image Signal Processing Unit, 133: First Image Input Unit, 134: Second Image Input Unit, 140: Sound Processing Unit, 141: Sound Output Unit, 142: Sound Signal Processing Unit, 143: Sound Input Unit, 151: Position Information Acquisition Unit, 160: Sensor Unit, 1 70: Communication unit, 180: Extension interface unit, 1001: Basic operation program, 1002: Application, 1009: Various data storage areas, 1101: Basic operation function unit, 1111: Binocular status confirmation unit, 1112: Communication status confirmation unit, 1113: Application control unit, 1114: Display control unit, 1199: Temporary storage area, U1: User, M11, M12: Warning message, AP11: Application screen (application image), O11, O12: Preset icons, OW11: External image. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0028] Example 1

[0029] Figure 1 is a block diagram showing the hardware configuration of the HMD 100 in this embodiment. Figure 1 In the figure, 101 is a main control unit (CPU / MCU, etc.), 102 is a bus serving as a path for sending and receiving commands and data, 103 is a RAM serving as a working area when executing basic action programs and other action programs, and 110 is a storage unit serving as a non-volatile storage medium such as Flash ROM / EEPROM / SSD / HDD.

[0030] Reference numeral 120 denotes an operation input unit serving as a user operation interface, and includes operation keys 121 such as a switch key, a power key, and a volume key, and a touch sensor 122 such as a touch pad.

[0031] In addition, 130 is an image processing unit such as an image (video) processor, which has a display unit (display) 131, an image signal processing unit (image (video) signal processor) 132, an external camera for acquiring external images, namely a first image input unit 133, and an internal camera for binocular confirmation, namely a second image input unit 134.

[0032] In addition, reference numeral 140 denotes a sound processing unit (audio processor), which includes a sound output unit (speaker) 141 , a sound signal processing unit (audio signal processor) 142 , and a sound input unit (microphone) 143 .

[0033] Reference numeral 151 denotes a position information acquiring unit (GPS receiving unit), and reference numeral 160 denotes a sensor unit including a gyro sensor 161 , a geomagnetic sensor 162 , and an acceleration sensor 163 .

[0034] In addition, reference numeral 170 denotes a communication unit including a LAN (Wi-Fi (registered trademark)) communication unit, a mobile communication unit, a Bluetooth (registered trademark) communication unit, and the like, and reference numeral 180 denotes an expansion interface unit having a function such as a charging terminal.

[0035] Figure 2 This is the appearance and wearing diagram of the HMD in this embodiment. Figure 2 As shown, the HMD in this embodiment is a goggle-type HMD. Rather than an optical projection type with a window on the front surface, it is a video transmission-type, sealed HMD that captures external images using a camera and displays them on a display within the HMD. (A) is a top view of the HMD 100 worn on the head of user U1, (B) is a front view, and (C) is a side view.

[0036] exist Figure 2 In the figure, 133L is the first image input unit (L) for the left external camera, and 133R is the first image input unit (R) for the right external camera. It should be noted that there may be only one first image input unit (for example, one first image input unit 133 may be provided in the center).

[0037] In addition, 122L is a touch sensor (L), 122R is a touch sensor (R) (not shown), and the touch sensor can also be only one of the left or right. In addition, 141L is a sound output unit (L) (stereo speaker L), and 141R is a sound output unit (R) (stereo speaker R) (not shown).

[0038] Although not shown in the figure, a monaural microphone is provided as the sound input unit 143. Furthermore, a display unit 131 composed of an LCD or the like is provided inside the HMD housing and is configured to be visually recognized by the user. Alternatively, the display unit 131 may be composed of a left-eye display unit 131L and a right-eye display unit 131R. Furthermore, a second image input unit 134, which is an internal camera configured to monitor the status of the user's eyes, is provided inside the HMD housing. Alternatively, the second image input unit 134 may be composed of a second image input unit 134L capable of monitoring the status of the left eye and a second image input unit 134R capable of monitoring the status of the right eye.

[0039] Figure 3 This is a functional structure diagram of the HMD in this embodiment. Figure 3In the embodiment, the storage unit 110 includes a basic action program 1001 which is a basic program such as an OS, an application 1002 including an application program (hereinafter abbreviated as application program) for executing the functions of this embodiment and other applications, and an area 1009 for storing various action setting values and various information (dynamic images / static images / sounds, etc.), namely, various data storage areas.

[0040] The program / application stored in the storage unit 110 is expanded (loaded) into the RAM 103 , and the main control unit 101 executes the expanded program / application, thereby executing the functional units of this embodiment and other functional units.

[0041] exist Figure 3 In the RAM 103, the expanded execution program is recorded according to the function. Specifically, the HMD 100 includes: a basic operation function unit 1101 for executing a basic operation program of the HMD 100; a two-eye state confirmation unit 1111 for analyzing an image acquired by the second image input unit (internal camera) 134 to confirm the state of the eyes and the line of sight of the user U1 wearing the HMD 100; a communication state confirmation unit 1112 for analyzing information received via the communication unit 170 to confirm whether the received information is an emergency alert, etc.; an application control unit 1113 for controlling the activation and execution of various applications in accordance with an operation instruction of the user U1 wearing the HMD 100 and generating an application screen (application image); and a display control unit 1114 for selecting one of the application screen image (application image) generated by the application control unit 1113 and the external image acquired by the first image input unit (external camera) based on the state of the user's eyes and the line of sight confirmed by the two-eye state confirmation unit 1111 or whether the received information is an emergency alert confirmed by the communication state confirmation unit 1112, and controlling the display on the display unit 131. In addition, the RAM 103 has a temporary storage area 1199 as a temporary storage area for various information generated / acquired by application programs.

[0042] Figure 4 This is a processing flow chart of the HMD in this embodiment. Figure 4First, in step S101, the display control unit 1114 displays the application screen (application image) generated by the application control unit 1113 upon activation / execution in accordance with a user's operation instruction on the display unit 131, and the process proceeds to S102. In S102, the second image input unit (internal camera) 134 acquires binocular images of the user, and the process proceeds to S103. In S103, the binocular state confirmation unit 1111 analyzes the binocular images acquired in S102. If it determines that both eyes of the user are closed, the process proceeds to S104. If it determines that both eyes of the user are not closed, the process returns to S102. In S104, the binocular state confirmation unit 1111 initializes the count value of the timer, and the process proceeds to S105. In S105, the second image input unit (internal camera) 134 acquires binocular images of the user, and the process proceeds to S106. In S106, the binocular state confirmation unit 1111 analyzes the binocular images of the user acquired in S105. If it determines that either of the user's eyes is open, the process returns to S102. If it determines that neither of the user's eyes is open, the process proceeds to S107. In S107, the binocular state confirmation unit 1111 counts the timer count value, and the process proceeds to S108. In S108, the binocular state confirmation unit 1111 compares the timer count value with a predetermined value. If the timer count value is greater than or equal to the predetermined value, the process proceeds to S109. Otherwise, the process returns to S105.

[0043] In S109, if the user is in a sleeping state, the display control unit 1114 generates a sleeping screen (a dark screen with no image) and displays it on the display unit 131, and the process proceeds to S110. It should be noted that the process of S109 may not be performed.

[0044] In S110, the second image input unit (internal camera) 134 captures images of the user's eyes, and the process proceeds to S111. In S111, the binocular state confirmation unit 1111 analyzes the binocular images captured in S110. If it determines that either eye is open, the process proceeds to S112. If it determines that neither eye is open, the process returns to S110. It should be noted that a timer may be added after S111, and the process proceeds to S112 if it detects that the eyes have been open for a predetermined time or longer.

[0045] In S112, if the user is awake (or semi-awake; the same applies hereinafter), first image input unit (external camera) 133 acquires an external image, and the process proceeds to S113. In S113, display control unit 1114 displays the external image acquired in S112 on display unit 131, and the process ends.

[0046] As described above, in this embodiment, the internal camera monitors the state of the wearer's (user's) eyes. If the eyes remain closed for a predetermined period or less, the wearer (user) is determined to be awake. If the eyes remain closed for a predetermined period or more, the wearer (user) is determined to be asleep. Then, when the wearer transitions from awake to asleep, the application screen switches to a sleep screen (no image). Subsequently, when the wearer returns to awake state, the sleep screen switches to a video viewfinder image (external camera image). This allows the wearer (user) to monitor their surroundings (real space) even if they fall asleep while wearing the HMD and later wake up.

[0047] It should be noted that when transitioning from an awake state to a sleep state, the application image can continue to be displayed without switching to the sleep screen (no image). However, by switching to a dark, no-image screen, i.e., the sleep screen, it is possible to conserve HMD battery consumption while the wearer is asleep. Furthermore, when transitioning from an awake state to a sleep state, the application image can be switched to the video view image (external camera image).

[0048] Furthermore, if the application screen (application image) displayed on the display unit 131 in S101 is accompanied by sound output (sound output from the sound output unit 141), the sound output may also be stopped when the screen switches to the sleep screen (no image) in S109. Alternatively, to indicate that the HMD is operating (or was operating), the sound output may continue even when the screen switches to the sleep screen (no image) in S109. Alternatively, the sound output may continue while the volume is lowered.

[0049] As described above, according to this embodiment, it is possible to provide an HMD that allows a user to check their surroundings and avoid danger even if the user falls asleep while wearing the HMD and wakes up later.

[0050] Example 2

[0051] In the first embodiment, a configuration is described in which, when waking up from a sleep state, the user can check his or her surroundings by switching to an external camera image, thereby avoiding danger. In this embodiment, another example of avoiding danger is described.

[0052] Figure 5 This is a processing flow chart of the HMD in this embodiment. Figure 5 In, with Figure 4 The same processes are denoted by the same reference numerals and their descriptions are omitted. Figure 5 In, with Figure 4The difference lies in steps S122 and S123 , in which a message indicating that the wearer (user) is wearing the HMD is displayed upon waking up from the sleep state.

[0053] That is, in Figure 5 Among them, S101~S111 and Figure 4 The action is the same. It should be noted that Figure 5 S109 is not recorded, but it can also be used with Figure 4 Similarly, when the state changes from the awake state to the sleep state, the sleep screen is displayed.

[0054] If the answer in S111 is "Yes," the user has woken up from sleep, and the process proceeds to S122. In S122, the display control unit 1114 generates a message to warn the user that the user is wearing an HMD, and the process proceeds to S123. In S123, the display control unit 1114 superimposes the warning message generated in S122 on the application screen (application image) generated by the application control unit 1113 in response to the user's operation instruction to start / execute the application, and displays it on the display unit 131, ending the process.

[0055] Figure 6 This is an example of a warning message display in this embodiment. Figure 6 In the display unit 131, the warning message M11 generated in S122 is superimposed on the application screen (application image) AP11. It should be noted that when the sleep screen is displayed when the user transitions from the awake state to the sleep state, the warning message M11 is superimposed on the sleep screen, which is a dark, imageless screen. In addition to the aforementioned examples, other examples of warning message display include a message such as "I slept for X hours" or a simple display of the current time. In other words, any display that reminds the user that they are wearing an HMD is sufficient.

[0056] In addition, if Figure 6 As shown, the warning message may be not only displayed on the display unit 131 but also output as an audio output from the audio output unit 141. In this case, if the same warning message is repeated multiple times, the possibility of the user missing the warning message can be reduced. Alternatively, the audio output of the warning message may be repeated until the user instructs to stop the audio output of the warning message.

[0057] As described above, according to this embodiment, it is possible to provide an HMD that allows the user to confirm a warning message and avoid danger even when the user falls asleep while wearing the HMD and wakes up later.

[0058] Note that, when the state changes from the awake state to the sleep state, a warning message may be displayed superimposed on the application screen displayed in the awake state on the display unit 131 .

[0059] In addition, after confirming the warning message, if the user wishes to switch to the display of the external image, the user can switch by an instruction, gesture, etc. from the operation input unit 120. In addition, the warning message can be deleted and the display of the application screen can be restarted according to the user's instruction.

[0060] Example 3

[0061] In this embodiment, another example of avoiding danger when wearing an HMD will be described.

[0062] Figure 7 This is a processing flow chart of the HMD in this embodiment. Figure 7 First, in step S201, the display control unit 1114 displays the application screen (application image) generated by starting / executing the application control unit 1113 according to the user's operation instruction on the display unit 131, and then proceeds to S202.

[0063] In S202, the communication unit 170 receives a message sent from a device on the external network, a device in a collaborative operation, or a portable information terminal such as a smartphone, and the process proceeds to S203. In S203, the communication status confirmation unit 1112 analyzes the message received in S202. If it determines that the received message is an emergency alert, the process proceeds to S204. If it determines that the received message is not an emergency alert, the process returns to S202.

[0064] In S204, the first image input unit (external camera) 133 acquires an external image, and the process proceeds to S205. In S205, the display control unit 1114 displays the external image acquired in S204 on the display unit 131, and the process ends.

[0065] As described above, in this embodiment, when the HMD receives an emergency alert (email, etc.) such as an earthquake bulletin via the communication unit, the application image is switched to the video view image (the outside world image), regardless of whether the user is awake or asleep. This provides an HMD that allows the user to confirm their surroundings and avoid danger in an emergency, even when wearing a video-transmissive, sealed HMD, by switching to the outside world image.

[0066] Note that a warning message notifying the contents of the emergency alert may be superimposed and displayed on the display unit.

[0067] Alternatively, the sound input unit 143 may acquire sound information (instead of processing in S202 ), analyze the acquired sound information, and switch to the outside world image when determining that the sound information is an alarm or the like (instead of processing in S203 ).

[0068] Example 4

[0069] In the second embodiment, an HMD has been described that can avoid danger by superimposing a warning message on the display screen when the user wakes up from a sleep state.

[0070] In this embodiment, the following is described: after the warning message is superimposed and displayed, the image is switched to the external image by gazing at a predetermined icon.

[0071] Figure 8 This is a processing flow chart of the HMD in this embodiment. Figure 8 In the example, the first step enclosed by the dotted line is Figure 5 The processing from S101 to S111 is omitted and its description is also omitted.

[0072] If the answer in S111 is "Yes," the user has woken up from sleep, and the process proceeds to S124. In S124, the display control unit 1114 generates a message to warn the user that the user is wearing an HMD, and the process proceeds to S125. In S125, the display control unit 1114 superimposes the warning message generated in S124 on the application screen (application image) generated by the application control unit 1113 in response to the user's operation instruction to start / execute the application, and displays it on the display unit 131.

[0073] Figure 9 This is an example of a warning message display in this embodiment. Figure 9 In the display unit 131, the warning message M12 generated in S124 is superimposed on the application screen (application image) AP11. A predetermined icon O11 is also displayed, indicating that the emergency door is closed. The warning message M12 displays a message stating that the user is wearing an HMD and a message stating, "Please look at the door icon if you need to check the outside world." As will be described later, if the user's gaze is directed at the predetermined icon O11 for a predetermined period of time or longer, the image displayed on the display unit switches from the application image to the outside world image.

[0074] It should be noted that, in S125 , only the scheduled icon O11 may be displayed. That is, the display of the scheduled icon O11 may also include the meaning of the warning message M12 .

[0075] Next, in S302, the second image input unit (internal camera) 134 obtains binocular images of the user, and the process proceeds to S303. In S303, the binocular state confirmation unit 1111 analyzes the binocular images of the user obtained in S302, detects the user's line of sight, and proceeds to S304. In S304, the binocular state confirmation unit 1111 confirms the user's line of sight detected in S303, and if it is determined that the user's line of sight is on the predetermined icon, the process proceeds to S305. If it is determined that the user's line of sight is not on the predetermined icon, the process returns to S302. It should be noted that the user's line of sight may also be determined to be on the predetermined icon if the user's line of sight is within a predetermined range that includes and is larger than the predetermined icon.

[0076] In S305, it is determined that the user is viewing the predetermined icon. The binocular state confirmation unit 1111 initializes the timer count value, and the process proceeds to S306. In S306, the second image input unit (internal camera) 134 acquires binocular images of the user, and the process proceeds to S307. In S307, the binocular state confirmation unit 1111 analyzes the binocular images acquired in S306 to detect the user's gaze position, and the process proceeds to S308. In S308, the binocular state confirmation unit 1111 confirms the user's gaze position detected in S307. If it is determined that the user's gaze position has deviated from the predetermined icon (within the predetermined range), the process returns to S302. If it is determined that the user's gaze position has not deviated from the predetermined icon (within the predetermined range), the process proceeds to S309. In S309, the binocular state confirmation unit 1111 counts the timer count value, and the process proceeds to S310. In S310, the eye state confirmation unit 1111 compares the timer count with a predetermined value. If the timer count is greater than or equal to the predetermined value, it is determined that the eye gaze has been directed at the predetermined icon for a predetermined time or longer, i.e., the user is gazing, and the process proceeds to S311. If the timer count is not greater than or equal to the predetermined value, the process returns to S306.

[0077] In S311, the first image input unit (external camera) 133 acquires an external image, and the process proceeds to S312. In S312, the display control unit 1114 displays the external image acquired in S311 on the display unit 131, and the process ends.

[0078] Figure 10 This is an example of external display in this embodiment. Figure 10 In the display unit 131, the external image OW11 is displayed through S312. Figure 9When the scheduled icon O11 in the emergency door is on for more than the scheduled time, the image displayed on the display unit 131 is switched from the warning message superimposed on the application image to the external image, and changes from the scheduled icon O11 marking the closed state of the emergency door to the scheduled icon O12 marking the open state of the emergency door.

[0079] It should be noted that when the user's gaze is directed at the scheduled icon O12 for a predetermined period of time or longer, the image displayed on the display unit 131 switches from the external image to the application image. Furthermore, the design of the scheduled icon changes from the scheduled icon O12 indicating the emergency door's open state to the scheduled icon O11 indicating the emergency door's closed state.

[0080] It should be noted that the predetermined icon may not be an emergency door mark, as long as it is an icon displayed to the outside world.

[0081] Alternatively, the waiting loop for external switching based on gaze may be switched to the application screen display by an interruption process, that is, an instruction from the user of the operation input unit 120 .

[0082] As described above, according to this embodiment, even if the user falls asleep while wearing the HMD and wakes up later, the user can confirm the warning message and automatically switch to the outside world image by looking at a predetermined icon.

[0083] It should be noted that the scheduled icon O11 can also be displayed superimposed on the application image AP11 at all times during application execution, regardless of the user's transition between sleep and wakefulness. Specifically, since users may feel uneasy if their field of vision is obscured from the real world by the application image AP11 for a long period of time, danger can be avoided if the user can always check the outside world by focusing their gaze on the scheduled icon O11.

[0084] Furthermore, the above embodiments describe a case where the HMD is a video-transmissive, sealed HMD. However, even if the HMD is an optical projection HMD, if an object generated by executing an application covers a large portion of the display, by performing the same process as described in the above embodiments, even if a user falls asleep while wearing the HMD and later wakes up, the wearer (user) can still confirm their surroundings (the real world) without being obstructed by the application screen (application image). It should be noted that in the case of an optical projection HMD, the process of switching to the video viewfinder image (external camera image) in the above embodiments can be replaced with a process of stopping the display of the application image.

[0085] While the embodiments have been described above, they are described in detail to facilitate understanding of the present invention and are not necessarily limited to having all of the described structures. Furthermore, a portion of the structure of one embodiment can be replaced with the structure of another embodiment, and a portion of the structure of one embodiment can be added to the structure of another embodiment. Furthermore, other structures can be added, deleted, or substituted for a portion of the structure of each embodiment.

Claims

1. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes obtained by the second image input unit, when the state of closing the eyes does not continue for more than a predetermined time, it is judged that the wearer is in an awake state; when the state of closing the eyes continues for more than a predetermined time, it is judged that the wearer is in a sleeping state. When the wearer changes from the awake state to the sleeping state, the display of the display unit is switched from the application image displayed in the awake state to a non-image screen. Thereafter, when the wearer changes from the sleeping state to the awake state, the display of the display unit is switched from the non-image screen to the external image obtained by using the first image input unit.

2. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes acquired by the second image input unit, if the state of closing the eyes does not continue for more than a predetermined time, it is determined that the wearer is in an awake state; if the state of closing the eyes continues for more than a predetermined time, it is determined that the wearer is in a sleeping state. When the state changes from the awake state to the sleep state, the application image displayed in the awake state continues to be displayed as the display of the display unit, and then, when the state changes from the sleep state to the awake state, the display of the display unit is switched from the application image to the external image obtained using the first image input unit.

3. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes acquired by the second image input unit, if the state of closing the eyes does not continue for more than a predetermined time, it is determined that the wearer is in an awake state; if the state of closing the eyes continues for more than a predetermined time, it is determined that the wearer is in a sleeping state. When the state changes from the awake state to the sleep state, the display on the display unit is switched from the application image displayed in the awake state to the external image acquired by the first image input unit.

4. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes acquired by the second image input unit, if the state of closing the eyes does not continue for more than a predetermined time, it is determined that the wearer is in an awake state; if the state of closing the eyes continues for more than a predetermined time, it is determined that the wearer is in a sleeping state. When the state changes from the awake state to the sleep state, the display of the display unit is switched from the application image displayed in the awake state to a non-image screen, and then, when the state changes from the sleep state to the awake state, a warning message is superimposed on the non-image screen to warn that the head-mounted display device is being worn.

5. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes acquired by the second image input unit, if the state of closing the eyes does not continue for more than a predetermined time, it is determined that the wearer is in an awake state; if the state of closing the eyes continues for more than a predetermined time, it is determined that the wearer is in a sleeping state. When the state changes from the awake state to the sleep state, the application image displayed in the awake state continues to be displayed as the display of the display unit, and then, when the state changes from the sleep state to the awake state, a warning message is superimposed on the application image to warn that the head-mounted display device is being worn.

6. A head-mounted display device, which is a video transmission type head-mounted display device that acquires an external image using a first image input unit and displays it on a display unit, characterized in that: have: a second image input unit for acquiring images of the state of the wearer's eyes; as well as Control Department, The control unit Based on the image of the state of the wearer's eyes acquired by the second image input unit, if the state of closing the eyes does not continue for more than a predetermined time, it is determined that the wearer is in an awake state; if the state of closing the eyes continues for more than a predetermined time, it is determined that the wearer is in a sleeping state. When the state changes from the awake state to the sleep state, a warning message is displayed superimposed on the application image displayed in the awake state, warning that the head mounted display device is being worn.

7. The head mounted display device according to any one of claims 4 to 6, wherein: As the warning message, the control unit displays an icon prompting external display and a message indicating that the external image can be confirmed by observing the icon. When it is judged that the wearer's line of sight is directed at the icon for more than a predetermined time based on the image of the state of the wearer's eyes obtained by the second image input unit, the display of the display unit is switched to the external image obtained by the first image input unit.

8. The head mounted display device according to claim 7, wherein: The icon is a mark indicating the closed state of the emergency door.

Images