Control method

Abstract

While considering the user's immersion in the head-mounted display, the system aims to mitigate user inconvenience caused by reduced visibility resulting from the difference between the brightness of the head-mounted display screen and the brightness of ambient light. [Solution] The head-mounted display compares the ambient brightness, based on measurements from an ambient light sensor, with a pre-set ambient light threshold for issuing a warning. If it detects that the ambient brightness is below the ambient light threshold and that the user wearing the head-mounted display has performed an action to prepare for viewing the outside world, it executes control to output notification information indicating that visibility of the outside world has decreased, or a control signal for a lighting fixture.

Description

Technical Field

[0001] The present invention relates to display technology for a head-mounted display (hereinafter abbreviated as "HMD").

Background Art

[0002] As background art in this technical field, Patent Document 1 describes that in a see-through type HMD, when the brightness of the periphery suddenly changes, the user can quickly visually recognize the surrounding environment.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an immersive HMD, since it is often used by blocking out the outside world, it is difficult to grasp the brightness of the outside world in real time. Therefore, for example, if the outside world (real space) becomes dark during the use of the HMD and the user does not notice this and cannot grasp the surrounding situation after removing the HMD, the user may risk hitting their body against objects, walls, etc. in subsequent actions in the dark.

[0005] On the other hand, if the brightness of the screen of the head-mounted display is changed in real time according to the brightness of the outside world, the immersion feeling of the head-mounted display will be impaired.

[0006] Therefore, immersive HMDs require technology that appropriately assists the user when removing the HMD, while also considering the immersive experience of the HMD. In this regard, since Patent Document 1 mentioned above relates to a see-through type HMD, applying it directly to an immersive HMD is insufficient in terms of achieving both the maintenance of immersion and user assistance, and there is room for further improvement.

[0007] This invention has been made in view of the above circumstances, and aims to alleviate the inconvenience to the user caused by reduced visibility resulting from the difference between the brightness of the head-mounted display screen and the brightness of ambient light, while taking into consideration the user's sense of immersion in the head-mounted display. [Means for solving the problem]

[0008] To solve the above problems, the present invention comprises the configuration described in the claims. To give one example, the present invention is a method for controlling a head-mounted display, wherein the light detection unit of the head-mounted display measures the brightness of the outside world, and the control unit of the head-mounted display compares the brightness of the outside world measured by the light detection unit with an ambient light threshold set in advance for alerting the user, and when it is determined that the brightness of the outside world is less than or equal to the ambient light threshold and that the user wearing the head-mounted display has performed an ambient light viewing preparation operation to switch from viewing the display unit of the head-mounted display to viewing the outside world, the control unit controls the output of notification information from the notification unit of the head-mounted display to notify the user that the visibility of the outside world has decreased. [Effects of the Invention]

[0009] According to the present invention, it is possible to reduce user inconvenience caused by reduced visibility resulting from the difference between the brightness of the head-mounted display screen and the brightness of ambient light, while taking into consideration the user's immersion in the head-mounted display. Other issues, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0010] [Figure 1] This is an external view of an immersive HMD (Head-Mounted Display). [Figure 2] This is a hardware configuration diagram for the HMD. [Figure 3] This is a block diagram showing the functional configuration of the HMD. [Figure 4] This is a flowchart showing the processing flow of the HMD according to the first embodiment. [Figure 5] This figure shows an example of the screen display shown in the first embodiment. [Figure 6] This figure shows an example of a preparatory action for visualizing the outside world. [Figure 7] This figure shows an example of a preparatory action for visualizing the outside world. [Figure 8] This figure shows an example of a preparatory action for visualizing the outside world. [Figure 9] This figure shows an example of a notification method. [Figure 10] This flowchart shows the processing flow related to a modified example of the second embodiment. [Figure 11] This is a diagram showing a risk classification table. [Figure 12] This is a flowchart showing the processing flow of the HMD according to the third embodiment. [Figure 13] This figure shows the notification screen according to the level of risk. [Figure 14] This is a flowchart showing the processing flow of the HMD according to the fourth embodiment. [Figure 15] This is an example of a light control screen. [Figure 16A] This figure shows examples of the lights being on. [Figure 16B] This figure shows examples of the lights being on. [Figure 17] This is an explanatory diagram showing an example of a scene to which the fourth embodiment is applied. [Figure 18] This is a flowchart showing the processing flow of the HMD according to the fourth embodiment. [Figure 19] This is an explanatory diagram showing the processing flow of the HMD-linked system according to the fifth embodiment. [Figure 20] It is a diagram showing an example of an operation screen of a lighting fixture. [Figure 21] It is a diagram showing an example of an operation screen of a lighting fixture. [Figure 22] It is a diagram showing an example of an operation screen of a lighting fixture. [Figure 23] It is a flowchart showing the flow of processing of the HMD cooperation system according to the sixth embodiment. [Figure 24] It is a diagram showing variations of a recommended lighting fixture selection screen. [Figure 25] It is a diagram showing an example of a luminance recommendation table. [Figure 26] It is a diagram showing an example of a recommended luminance proposal screen. [Figure 27] It is a diagram showing an example of a warning stop screen.

Mode for Carrying Out the Invention

[0011] In the present invention, when the HMD, particularly an immersive HMD, is removed while the user is viewing with it worn, the user's eyes may be dazzled due to the difference between the luminance of the HMD screen and the brightness of the outside world. Therefore, the present invention relates to an HMD, an HMD cooperation system, and a control method for an HMD that aim to provide user support in consideration of dark adaptation or light adaptation when removed after using the HMD. Thus, since the present invention can be expected to improve the usability of technology related to head-mounted displays for labor-intensive industries, it can be expected to contribute to 8.2 of the Sustainable Development Goals (SDGs) proposed by the United Nations (Increase the productivity of the economy through diversification, technology improvement, and innovation, focusing on industries that enhance the value of goods and services and labor-intensive industries).

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiments, the same members are generally denoted by the same reference numerals, and repeated explanations thereof are omitted.

[0013] <First Embodiment> The first embodiment is an embodiment that, when attempting to remove an immersive HMD 10 in a relatively dark external environment, notifies or warns the user that the external environment is dark because the user's eyes cannot instantly adapt to the darkness, thereby prompting the user to pay attention.

[0014] Figure 1 shows an external view of the immersive HMD10.

[0015] The HMD10 primarily consists of a wearable body 11, which houses a display 119, a microphone 121, and a speaker 122.

[0016] The mounting body 11 is a component that maintains the state in which the HMD 10 is attached to the user's head and holds other components of the HMD 10.

[0017] The display 119 is positioned in front of both eyes of the user wearing the HMD 10. The display 119 is an opaque liquid crystal display. The display 119 has a variable transparency mechanism (e.g., a shutter) that changes the transparency, and the processor 125 (see Figure 2), described later, controls the variable transparency mechanism to transition the display 119 from an opaque state to a transparent state. The display 119 functions as a notification device that outputs notification information using the display of warning messages or a predetermined low-brightness screen. In addition, the display 119 notifies or warns the user that the outside world is dark by adjusting the brightness of the screen, and also encourages the user to adapt to the dark.

[0018] The microphone 121 is one form of an input device from the user to the HMD 10.

[0019] Speaker 122 functions as a notification device that outputs notification information to the user from HMD 10 using music, warning sounds, or warning messages.

[0020] Figure 2 is a hardware configuration diagram of the HMD10.

[0021] The HMD10 includes an out-camera 111, an in-camera 112, a distance sensor 113, an illuminance sensor 114, an accelerometer 115, a gyroscope 116, a geomagnetic sensor 117, a GPS receiver 118, a display 119, a network communicator 120, a microphone 121, a speaker 122, a timer 124, a processor 125, memory 128, and a power button 130, all of which are connected to each other via a bus 140 that connects each component. In the fourth embodiment, a light 131 is also provided.

[0022] The network communication device 120 is, for example, a short-range wireless communication device or a wireless LAN communication device. The network communication device 120 is connected to the antenna 123. The HMD 10 is connected to a communication network 9, such as a home LAN, from the network communication device 120 and the antenna 123. In the embodiment described later, the lighting fixture 3 is also connected to the communication network 9. Then, an HMD linkage system 1 is constructed that links the HMD 10 and the lighting fixture 3. The details of this will be described later.

[0023] The short-range wireless communication device may be a communication device that conforms to a wireless LAN communication method such as Bluetooth®, IrDA (Infrared Data Association®, registered trademark), Zigbee®, HomeRF (Home Radio Frequency®, registered trademark), or Wi-Fi®.

[0024] The HMD10 is equipped with an ambient light sensor that detects the brightness of the outside world. The illuminance sensor 114 is one example. Another example of detecting ambient brightness is to analyze the brightness of an ambient image generated by an external camera 111 capturing images of the outside world. In that case, the external camera 111 functions as an ambient light sensor.

[0025] Memory 128 consists of flash memory and non-volatile memory. Memory 128 also stores programs 126 such as the OS (Operating System) and operation control applications, as well as data 127 used by the processor 125.

[0026] The processor 125 is composed of, for example, a CPU. The processor 125 loads the program 126 into memory 128 and executes it, and reads data 127 as needed and uses it in the execution process of the program 126.

[0027] Figure 3 is a block diagram showing the functional configuration of the HMD10.

[0028] The HMD10's processor 125 executes an ambient light monitoring program to configure the ambient light monitoring unit 201, image analysis unit 202, preparation operation detection unit 203, notification control unit 204, communication control unit 205, risk determination unit 206, light control unit 207, display control unit 208, lighting fixture control unit 209, and display switching unit 210. The functions of each unit will be described later with reference to the flowchart. It is not necessary to configure all of the above units; each embodiment only needs to have the necessary configurations.

[0029] Figure 4 is a flowchart showing the processing flow of the HMD10 according to the first embodiment.

[0030] When the HMD10 starts up, the processor 125 reads the ambient light monitoring program from memory 128 and starts processing. The ambient light sensor observes the ambient light around the user of the HMD10 and detects the brightness of the outside world (S01).

[0031] When the illuminance sensor 114 is used as the ambient light sensor, the ambient light monitoring unit 201 acquires the sensor output from the illuminance sensor 114 and uses this sensor output to determine the ambient brightness in step S02. Alternatively, when the luminance of the ambient image generated by the rear camera 111 is used as the ambient light sensor, the image analysis unit 202 calculates representative values ​​of the ambient image luminance, such as the average value of each pixel or the median value of the pixel distribution, and uses the results of this calculation to determine the ambient brightness in step S02.

[0032] The ambient light monitoring unit 201 compares the ambient light B, based on the measurement results from the ambient light sensor, with a first ambient light threshold B_th1 that has been set in advance to issue a warning. If the ambient light B is brighter than the first ambient light threshold B_th1 (S02: No), the unit proceeds to step S05.

[0033] If the ambient brightness B is the same as or darker than the first ambient light threshold B_th1 (S02: Yes), and no preparation for ambient visibility is detected (S03: No), proceed to step S05.

[0034] The external view preparation action is a preparatory action performed by a user wearing the HMD10 to switch from viewing the display 119 to viewing the outside world. In other words, it can be described as the action of the user attempting to remove the HMD10.

[0035] Figure 5 shows an example of a screen display shown in the first embodiment. The HMD10 determines that an external view preparation operation has been performed when the shutdown screen 301 (power button 130 turned OFF) is displayed, when the termination selection screen 302 for at least one application running on the HMD10 is displayed, when the termination icon displayed on the application screen is operated on the operation screen 303, or when the video pause screen 304 is displayed (the video being played is stopped). This allows the user to be alerted in step S04, described later, just before the user shuts down the HMD10, when use has finished, or when video viewing is not interrupted. Operations on each screen may also be performed by the in-camera 112 capturing a facial image including the user's eyes, and the image analysis unit 202 detecting the movement of the gaze (functioning as a gaze sensor). Alternatively, operations may be performed using a controller linked to the HMD10. The display of the shutdown screen 301 can be considered to include operations that the user selects and displays from the HMD10's menu screen. Furthermore, if the power is turned off without displaying the shutdown screen 301, the power-off operation itself, such as pressing the power button 130 or selecting it from the HMD10 menu screen, may be considered as an action to prepare for external visibility.

[0036] Figures 6 to 8 show examples of preparatory actions for visualizing the external environment.

[0037] The image analysis unit 202 analyzes the external image captured by the rear camera 111 and determines that the user is placing their hand on the HMD 10 (see Figure 6) as captured in the image. If the distance measuring sensor 113 detects that something is close enough to touch the HMD 10, the unit considers that the user has placed their hand on the HMD 10, and the preparation action detection unit 203 determines that an external view preparation action has been performed.

[0038] Furthermore, in Figure 7, the image analysis unit 202 pre-analyzes a facial image including the user's eyes and pre-determines the standard relative position of the user's eyes to the HMD 10. Then, as a result of the facial image analysis in step S01, if the user's eyes deviate from the acceptable range based on the standard position, it is determined that the eyes have shifted from the standard position, i.e., the user is attempting to remove the HMD 10, and a preparatory action for viewing the outside world has been performed.

[0039] Furthermore, in Figure 8, the preparation action detection unit 203, based on the sensor output (angle information) of the gyro sensor 16 or acceleration sensor 115 (a 3-axis acceleration sensor is preferable), determines that the user's head, while wearing the HMD 10, is facing downwards at or above a predetermined angle threshold. This is considered an action performed before removing the HMD 10, and the unit determines that a preparation action for viewing the outside world has been performed. By determining these actions shown in Figures 6 to 8 as preparation actions for viewing the outside world, it is possible to provide a warning to the user in step S04, described later, in case of sudden removal of the HMD 10, while still considering the user's immersion in the HMD 10.

[0040] Returning to Figure 4, when the preparation operation detection unit 203 detects an external visibility preparation operation when the external brightness B is the same as or darker than the first external light threshold B_th1 (S02:Yes), the notification control unit 204 notifies the user that the visibility of the external environment has decreased (S04).

[0041] Figure 9 shows an example of a notification method.

[0042] The notification control unit 204 may display the warning message 310 "It is getting dark. Please be careful." on the display 119 via the display control unit 208. Alternatively, the notification control unit 204 may play the voice message "It is getting dark. Please be careful." from the speaker 122.

[0043] As another example of a notification method, the notification control unit 204 may display a screen allowing the user to choose whether or not to view low-luminance video or music for a predetermined period of time. For low-luminance video, a dedicated video may be prepared that gradually dims to promote dark adaptation, or the brightness of the display 119 showing the running application or video being played may be reduced. This allows the user to enjoy videos or music while having time for their eyes to adjust to the darkness.

[0044] Until the HMD10 shuts down, for example, when the power is turned off (S05: No), the process returns to S01 and continues.

[0045] When the HMD10 is powered off (S05:Yes), the series of processes in the ambient light monitoring program are terminated.

[0046] According to this embodiment, when viewing an immersive HMD 10, if the outside world is relatively dark (dark relative to the brightness of the display 119), there is a risk that the user's eyes will not adjust to the darkness when putting on or taking off the HMD 10, making it difficult to see the surrounding environment and potentially causing them to bump into objects or trip. According to this embodiment, when the user is detected preparing to remove the HMD 10, if the brightness of the outside world is darker than a predetermined brightness, a notification indicating that the outside world is dark is issued. This allows the user of the HMD 10 to be alerted while taking into consideration the user's sense of immersion in the HMD 10.

[0047] <Second Embodiment> Figure 10 is a flowchart showing the processing flow according to a modified example of the second embodiment.

[0048] In the second embodiment, when the HMD10 is started up or resumed from sleep mode, the elapsed time T is measured by the timer 124. When the preparation operation detection unit 203 detects an external view preparation operation (S03:Yes), if the elapsed time T since the HMD10 was started up or resumed from sleep mode is less than a predetermined time threshold T_th1 (S031:No), the process proceeds to step S05.

[0049] On the other hand, if the preparation operation detection unit 203 determines that the elapsed time T is equal to or greater than a predetermined time threshold T_th1 (S031:Yes), it notifies that the visibility of the outside world has decreased (S04).

[0050] According to this embodiment, if the time threshold T_th1 is determined based on the preparation time for wearing the HMD10, operations such as adjusting the positional misalignment after wearing the HMD10 or restarting the HMD10 due to malfunction will not be detected as external visibility preparation operations, thereby suppressing unnecessary notifications.

[0051] Furthermore, if the time threshold T_th1 is determined considering the time required for the HMD10 user to adapt to the light, the user's eyes will not be accustomed to the brightness of the display 119 immediately after starting to view the HMD10. Therefore, even if the external visibility preparation action is detected in this state, there will be little decrease in external visibility. Thus, frequent notifications can be avoided, improving the user experience.

[0052] <Third Embodiment> The third embodiment is one in which a facial image is captured by the in-camera 112, including the user's eyes, is captured, and the user's face is classified into a multi-level risk category based on the size of the pupil, the transparency of the lens, and information such as the user's age, dark adaptation speed, and visual acuity that has been previously recorded in the memory 128 or server 2, and a warning is issued according to the classified risk category.

[0053] Generally, the pupil's response to changes in brightness is immediate, dilating as it gets darker. Dark adaptation is also influenced by the secretion of a light-sensing protein called "rhodopsin," and since the amount of rhodopsin secreted differs with age, a correlation has been observed between age and the speed of dark adaptation, making it usable as an indicator for classifying risk.

[0054] Furthermore, the transparency of the lens and visual acuity also affect visibility, so these can be used as indicators for classifying the degree of danger in darkness.

[0055] Figure 11 shows the risk classification table 410.

[0056] The risk classification table 410 shown in Figure 11 is stored in memory 128 or server 2, and if it is stored in server 2, the HMD 10 accesses it via the communication network 9.

[0057] The risk classification table 410 defines the risk level to darkness in three stages, relating it to pupil size, age, and dark adaptation speed. Furthermore, the transparency of the user's lens and visual acuity may also be considered. Specifically, the transparency of the user's lens, as captured by the in-camera 112, is compared with the transparency of a typical lens stored in memory 128 or server 2. If the risk determination unit 206 determines that the transparency of the user's lens is poor, the risk level may be increased by one stage. Similarly, if the visual acuity is below a predetermined value, the risk level may also be increased by one stage. In either case, if the risk level is already at the highest level, it does not need to be changed. Furthermore, if the risk level is determined based on only one of the following pieces of information—the user's pupil size, lens transparency, age, dark adaptation speed, or visual acuity—the risk level may be determined based on the degree of each piece of information.

[0058] Figure 12 is a flowchart showing the processing flow of the HMD according to the third embodiment. In Figure 12, the same reference numerals are used for steps that are the same as in Figure 10, and redundant explanations are omitted.

[0059] If the determination in step S031 is positive, the in-camera 112 captures an image of the user's face, including their eyes, thereby capturing images of the pupil and lens (S032).

[0060] The risk assessment unit 206 detects the size of the user's pupils and the transparency of their lens based on the facial image (S033). Furthermore, the risk assessment unit 206 acquires information on the user's age, dark adaptation speed, and visual acuity (collectively referred to as "user attribute information") stored in memory 128 or server 2, as well as general information on the transparency of the lens (S034).

[0061] The risk assessment unit 206 obtains the user's risk level to darkness by referring to the detected and acquired information on the user's pupil size and lens transparency, user attribute information, and the risk classification table 410. If the information and the risk classification table 410 are stored in the server 2, they are downloaded from the server 2 via the communication network 9, and the user's risk level to darkness is determined (S035).

[0062] The notification control unit 204 issues a notification according to the user's risk level (S04).

[0063] Figure 13 shows the notification screen according to the level of risk.

[0064] The notification control unit 204 selects a low-risk notification screen 331, a medium-risk notification screen 332, or a high-risk notification screen 333 according to the risk level determined in S035 and outputs it to the display control unit 208, which then displays it on the display 119. The brightness of the display 119 screen may also be changed according to the risk level. For example, the brightness of the display 119 screen may be controlled to decrease when the risk level is high.

[0065] According to this embodiment, the degree of risk can be classified according to the user's eye condition and attributes, and notifications (including warnings) can be issued according to that classification.

[0066] <Fourth Embodiment> The fourth embodiment is a head-mounted display equipped with a light 131, in which, when an action to prepare for viewing the outside world is detected, the light 131 is turned on and the display 119 is switched from opaque to transparent (see-through display) or the display 119 is switched to an external image captured by the out-camera 111.

[0067] Figure 14 is a flowchart showing the processing flow of the HMD according to the fourth embodiment. Figure 15 is an example of the light operation screen. Figures 16A and 16B show examples of the light's illumination state.

[0068] When the notification control unit 204 performs the notification control in step S04, it causes the display control unit 208 to display the light operation screen 340 (see Figure 15) on the display 119. If the user does not select to turn on the light (S041: No), the process proceeds to the termination determination process of the HMD 10 (S05).

[0069] On the other hand, if the user selects to turn on light 131 (S041: Yes), the light control unit 207 turns on light 131 (S042).

[0070] The front camera 112 captures a facial image including the user's eyes (S043). When it determines that the user's eyes are in the standard position (S044: Yes), the display switching unit 210 outputs a shutter open signal to the display 119, switching the display 119 to a transparent display. This makes the screen of the display 119 see-through (see Figure 16A) (S045). When the screen of the display 119 is made see-through, the display of virtual images may be turned off to allow the user to more accurately perceive the external environment. Alternatively, in S045, instead of making the display see-through, the display 119 may be switched to the external image captured by the rear camera 111. This allows the user to perceive the external environment even if the display 119 does not have a transparency variable mechanism. Furthermore, if the screen of the display 119 is converted to a see-through display and then deactivated after a predetermined time has elapsed while the HMD 10 is worn, the brightness of the display 119 screen may be controlled to gradually increase from low brightness to reduce the strain on the user's eyes due to changes in brightness.

[0071] Furthermore, when the HMD10 is attached or detached, the outside world can be illuminated by leaving the HMD10 powered on (S05:No) and the light 131 on (see Figure 16B).

[0072] According to this embodiment, when an action to prepare for external world visibility is detected, the light 131 is turned on to increase the transparency of the display 119, or the display on the display 119 is switched to an external world image captured by the out-camera 111, allowing the user to see the outside world and act while adapting to darkness, even while wearing the HMD 10.

[0073] <Fifth Embodiment> The fifth embodiment is one in which the HMD 10 identifies the direction of the switch 4 or remote control of the lighting fixture 3 in the outside world and turns on at least some of the lights 131 in the direction of the switch 4 or remote control of the lighting fixture 3. Figure 17 is an explanatory diagram showing an example scene to which the fourth embodiment is applied. Figure 18 is a flowchart showing the processing flow of the HMD 10 according to the fifth embodiment.

[0074] As shown in Figure 17, this embodiment pre-determines the direction and distance of the switch 4 of a lighting fixture 3 installed in a room as seen from the HMD 10, and when it detects a preparation for viewing the outside world, it notifies the user that the outside world is dark and illuminates the switch 4 with a light. The light 131 is equipped with an aperture mechanism and a beam direction control mechanism, and focuses the light to illuminate in a specific direction, for example, the direction of the switch 4 or controller. If multiple lights 131 are provided in different locations on the HMD 10, control may be implemented to turn on only the lights 131 that can illuminate in the direction of the switch 4 of the lighting fixture 3 or the remote control without aperture control or beam direction control. Through these controls, the user can understand the direction they should go to turn on the lighting fixture.

[0075] As shown in Figure 18, when the HMD 10 is started up, it begins processing to understand the surrounding environment. Specifically, when the HMD 10 is started up, the ambient light sensor detects the brightness B of the outside world (S001), and if the brightness of the outside world is equal to or greater than the second ambient light threshold B_th2 which determines the start of the processing to understand the surrounding environment (S002), the out camera 111 takes an image of the outside world (for example, indoors) (S003), and the distance to the surrounding area is measured by the distance measuring sensor 113 (S004).

[0076] The image analysis unit 202 analyzes the external image and recognizes the switch 4 through subject recognition processing (S005), and stores the direction and distance of the switch 4 in the HMD 10 (S006). This completes the process of understanding the surrounding environment. A remote control may be used instead of the switch 4.

[0077] Subsequently, the processes from step S01 onwards are executed. Then, in the notification process of step S04, the notification control unit 204 notifies that it is dark outside, and the light control unit 207 may control the direction of the light 131 to illuminate in the direction of the switch 4 or remote control, based on the position and distance of the switch 4 or remote control and the orientation of the HMD 10. At that time, in order to let the user know that the direction of illumination has been controlled, the light 131 may be controlled to blink for a predetermined time.

[0078] Multiple lights 131 may be provided. In that case, the light control unit 207 may control the direction of illumination of the first light using switch 4 and the second light using remote control.

[0079] As another example, the display 119, which is either a see-through display or has been switched to display an external image captured by the rear camera 111, may also display an image (such as a line or arrow) indicating the direction of the switch 4 or remote control. Alternatively, instead of illuminating in a specific direction, the light control unit 207 may rotate the illumination direction of the light 131. These modifications are particularly useful when using the HMD 10 in an unfamiliar location.

[0080] <Sixth Embodiment> The sixth embodiment is an HMD linkage system 1 that links the HMD 10 and a lighting fixture 3, and is an embodiment in which the HMD 10 controls the lighting fixture 3 to turn on when it detects the HMD 10 preparing to see the outside world.

[0081] The HMD linkage system 1 is configured by connecting the HMD 10 and the lighting fixture 3 via a communication network 9 (see Figure 1).

[0082] Figure 19 is an explanatory diagram showing the processing flow of the HMD-linked system 1 according to the fifth embodiment.

[0083] When the HMD10 starts up, the processor 125 reads the ambient light monitoring program from memory 128 and starts processing. The ambient light sensor observes the ambient light around the user of the HMD10 and detects the brightness of the outside world (S01).

[0084] If the ambient light monitoring unit determines that the ambient light B, based on the measurement results from the ambient light sensor, is brighter than the first ambient light threshold B_th1, which is set in advance to issue a warning (S02: No), the process returns to step S01.

[0085] If the external light monitoring unit determines that the external brightness B is the same as or darker than the first external light threshold B_th1 (S02:Yes), and the preparation operation detection unit 203 does not detect an external visibility preparation operation (S03:No), the process proceeds to step S05.

[0086] On the other hand, when the external light monitoring unit detects that the ambient light B is the same as or darker than the first ambient light threshold B_th1 (S02:Yes), and the preparation operation detection unit 203 detects an external visibility preparation operation (S03:Yes), the lighting fixture control unit 209 acquires information on registered lighting fixtures present around the HMD 10 (S11). For example, it identifies the lighting fixture 3 by performing image recognition on the ambient image captured by the out-camera 111.

[0087] The lighting fixture control unit 209 reads the operation screen of the identified lighting fixture 3 from the data 127 in the memory 128, and the display control unit 208 displays it on the display 119 (S12).

[0088] When the user performs a lighting operation (S13: Yes) on the operation screen, a control signal for the lighting fixture 3 is generated (S14), and the lighting fixture 3 executes the request from the HMD 10 according to the control signal (S15).

[0089] According to this embodiment, when a preparatory action for viewing the outside world is detected, the lighting of the lighting fixture 3 can be controlled to reduce the strain on the user's eyes. As will be described later, there can be various forms of control by changing the content of the operation screen and the control signals. Furthermore, controlling the lighting of the lighting fixture 3 also includes controlling it so that it does not feel dazzling.

[0090] (Example of a lighting fixture control screen 1) Figure 20 shows an example of a lighting fixture operation screen. The operation screen 500 for lighting fixture 3 in Figure 20 is a screen for selecting whether or not to turn on the light. When the operation screen 500 for lighting fixture 3 is displayed (S12) and the user gives a turn-on command on the operation screen 500 (S13), a control signal to turn on lighting fixture 3 is sent (S14), and lighting fixture 3 turns on (S15).

[0091] (Lighting fixture operation screen 2) Figure 21 shows an example of a lighting fixture operation screen. In Figure 21, the lighting fixture 3 selection screen 501 displays the names, installation locations, and functions of multiple lighting fixtures 3 located near the HMD10, in association with each other. When the lighting fixture 3 selection screen 501 is displayed (S12), and one of the lighting fixtures 3 is selected, the user transitions to the operation screen 502 for that lighting fixture 3. When the user issues a lighting fixture 3 on the operation screen 502 (S13), a control signal is sent to turn on, dim, or change the color of the lighting fixture 3 (S14), and the lighting fixture 3 turns on, dims, or changes color (S15).

[0092] (Lighting fixture operation screen 3) Figure 22 shows an example of a lighting fixture operation screen. In Figure 22, the user transitions from the lighting fixture 3 selection screen 501 to an operation screen 503 that instructs whether or not to control the brightness of the lighting in stages, and then to a screen 504 that sets the speed (time interval) at which the brightness of the lighting is increased in stages. When "OK" is selected on the setting screen 504 (S13), a control signal is sent to increase the brightness of the lighting fixture 3 at the time interval set on the setting screen 504 (S14), and the lighting fixture 3 increases its brightness (S15).

[0093] <Seventh Embodiment> The seventh embodiment is an HMD linkage system that links the HMD 10 and a lighting fixture 3, and when the HMD 10 is detected to be ready to see the outside world, in certain situations where it is preferable to brighten the entire room, the system recommends turning on the lighting fixture that can brighten the entire room. Examples of such specific situations include when there are other people in the same room or nearby, or when a pet approaches the user's feet.

[0094] Figure 23 is a flowchart showing the processing flow of the HMD-linked system according to the sixth embodiment.

[0095] When the lighting fixture control unit 209 obtains information on the registered nearby lighting fixtures 3 (S11), it determines whether the HMD 10 is in a specific state (S111). For example, it switches the rear camera 111 to infrared imaging mode, captures an image of the outside world, and determines that the HMD 10 is in a specific state if other people or pets are captured in the image.

[0096] When the lighting fixture control unit 209 determines that a specific state exists (S111), it generates and displays a recommended lighting fixture selection screen that prioritizes displaying lighting fixtures capable of illuminating the entire room from among multiple lighting fixtures (S112).

[0097] Figure 24 shows variations of the recommended lighting fixture selection screen.

[0098] The recommended lighting fixture selection screen 511 displays only the lighting fixtures that can illuminate the entire room (installation location: ceiling) from among the multiple available lighting fixtures.

[0099] The recommended lighting fixture selection screen 512 prioritizes displaying lighting fixtures that can illuminate the entire room (installation location: ceiling).

[0100] The recommended lighting fixture selection screen 513 highlights and displays lighting fixtures that can illuminate the entire room (installation location: ceiling).

[0101] The recommended lighting fixture selection screen 514 prioritizes displaying the closest available lighting fixture if there are no options for a light fixture that can illuminate the entire room (installation location: ceiling).

[0102] When a lighting fixture is selected on one of the recommended lighting fixture selection screens 511 to 514, the operation screen for the selected lighting fixture is displayed (S12).

[0103] The lighting fixture control unit 209 displays either the operation screen 500 or the selection screen 501 unless a specific condition is met (S111: No).

[0104] According to this embodiment, the user can be prompted to turn on lighting fixtures according to the external conditions, and if it is better to brighten the entire room, lighting fixtures that can brighten the entire room are prioritized or highlighted, and the decrease in visibility due to the difference between ambient light and the brightness of the display 119 when the HMD 10 is removed can be mitigated.

[0105] <Eighth Embodiment> The eighth embodiment is an HMD linkage system that links the HMD 10 and the lighting fixture 3, and in the embodiment that, upon detecting the external environment viewing preparation operation of the HMD 10, proposes a recommended brightness setting for the lighting fixture according to the situation that affects dark adaptation.

[0106] In this embodiment, in the third embodiment, instead of or in addition to warnings corresponding to the classified risk level, a process is performed to suggest recommended settings for the brightness of lighting fixtures according to the circumstances affecting dark adaptation.

[0107] Figure 25 shows an example of a brightness recommendation table.

[0108] The brightness recommendation table 521 correlates the screen brightness of the display 119 with the brightness of the recommended lighting fixture 3.

[0109] The brightness recommendation table 522 correlates pupil size with the brightness of the recommended lighting fixture 3.

[0110] Brightness recommendation tables 521 and 522 are stored as data 127 in memory 128, and the lighting fixture control unit 209 compares the pupil size (S033) detected with the brightness recommendation table 522 to determine the recommended brightness. Alternatively, in the recommendation setting proposal process, the lighting fixture control unit 209 obtains the brightness of the display 119 screen (screen brightness), compares the screen brightness with the brightness recommendation table 521 to determine the recommended brightness. The lighting fixture control unit 209 may also determine the brightness of the lighting fixture 3 based on both the screen brightness and the pupil size.

[0111] Figure 26 shows an example of a recommended brightness suggestion screen. Generally, when a bright screen is displayed or the pupil is small, lighting the lighting fixture 3 at a brightness higher than normal will reduce the strain on the user's eyes, so a recommended brightness suggestion screen 531 is displayed on the display 119. On the other hand, when a dark screen is displayed or the pupil is large, lighting the lighting fixture 3 at a brightness lower than normal will reduce the strain on the user's eyes, so a recommended brightness suggestion screen 532, which recommends a darker screen, is displayed on the display 119. The lighting fixture control unit 209 transmits control signals to the lighting fixture 3 indicating the operation content on the recommended brightness suggestion screens 531 and 532.

[0112] According to this embodiment, the HMD 10 can control the brightness of the lighting fixture 3 to reduce the strain on the user's eyes.

[0113] The present inventors have described the invention in detail based on its embodiments above, but it goes without saying that the present invention is not limited to the embodiments described above and can be modified in various ways without departing from its essence.

[0114] For example, any combination of the above embodiments is included in the present invention. Furthermore, modifications to add further functionality to each of the above embodiments are also possible. For example, as shown in Figure 27, there may be a modification in which, in step S04, a notification is given that visibility of the outside world has decreased, and in step S05, the HMD 10 is not terminated but the system returns to step S01, and if the preparation operation for visibility of the outside world is detected again (S03: Yes), a re-notification is not given.

[0115] As a specific example, in the first step S04, after issuing a notification, warning function stop screens 541 and 542, as shown in Figure 27, may be displayed. Stop screen 541 is a screen that stops the warning until the application is terminated (or closed). Stop screen 542 is a screen that stops the warning function until the next power-up. This suppresses unnecessary notifications. As in this example, modifications that do not depart from the gist of the present invention are included in the present invention.

[0116] Furthermore, the numbers and messages appearing in the text and diagrams are merely examples, and using different ones will not impair the effects of the present invention.

[0117] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program. The order in which each processing step is executed may also be changed.

[0118] The functions of the present invention described above may be implemented in hardware, in whole or in part, for example, by designing them using an integrated circuit. Alternatively, they may be implemented in software by having a processor unit or the like interpret and execute an operating program that realizes each of these functions. Hardware and software may also be used in combination.

[0119] Furthermore, the control lines and information lines shown in the diagram are those deemed necessary for explanation and do not necessarily represent all control lines and information lines on the product. In reality, it is reasonable to assume that almost all components are interconnected.

[0120] The above embodiments include the following forms. (Form of the first perspective) In head-mounted displays, Processor and A display having opacity, An ambient light sensor that detects the brightness of the outside world, A notification device that outputs notification information to the user of a head-mounted display, Equipped with, The aforementioned processor, The system compares the ambient brightness based on the measurement results from the ambient light sensor with an ambient light threshold set in advance for issuing a warning. When it detects that the ambient brightness is below the ambient light threshold and that the user wearing the head-mounted display has performed an action to prepare for viewing the outside world, the system executes control to output notification information from the notification device indicating that visibility of the outside world has decreased. Head-mounted display.

[0121] (Form of the second perspective) In head-mounted displays, Processor and A display having opacity, An ambient light sensor that detects the brightness of the outside world, Lights that illuminate the outside world, Equipped with, The display is equipped with a transmissive variable mechanism that varies the transmissivity using the processor, The aforementioned processor, The system compares the ambient brightness based on the measurement results from the ambient light sensor with an ambient light threshold set in advance for alerting the user. When it is detected that the ambient brightness is below the ambient light threshold and that the user wearing the head-mounted display has performed an action to prepare for viewing the outside world, the system controls the transparency of the display and turns on the lights. Head-mounted display.

[0122] (The form of the third perspective) In head-mounted displays, Processor and A display having opacity, An ambient light sensor that detects the brightness of the outside world, Lights that illuminate the outside world, An out-of-camera that captures images of the outside world and generates an image of the outside world, Equipped with, The aforementioned processor, The system compares the ambient brightness based on the measurement results from the ambient light sensor with an ambient light threshold set in advance for alerting the user. When it detects that the ambient brightness is below the ambient light threshold and that the user wearing the head-mounted display has performed an action to prepare for viewing the outside world, the system displays the image of the outside world captured by the rear camera on the display and controls the illumination of the lights. Head-mounted display.

[0123] (The form of the fourth perspective) In a head-mounted display integration system that links a head-mounted display with a lighting fixture, Head-mounted displays are Processor and A display having opacity, An external light sensor, A notification device that outputs notification information to the user of a head-mounted display, It includes a wireless communication device that communicates wirelessly with lighting fixtures, The aforementioned processor, The external light sensor measures the external environment and compares the sensor output obtained with the ambient light brightness with a preset ambient light threshold for issuing a warning. The system monitors whether the ambient light brightness is below the ambient light threshold, indicating an insufficient ambient light state. In the aforementioned state of insufficient ambient light, when it is detected that the user wearing the head-mounted display has performed an action to prepare for viewing the outside world, in order to switch from viewing the display to viewing the outside world, the control is executed to transmit a power-on instruction signal to the lighting fixture from the wireless communication device. A head-mounted display integration system.

[0124] (The form of the fifth perspective) In a head-mounted display integration system that links a head-mounted display with a lighting fixture, Head-mounted displays are Processor and A display having opacity, An external light sensor, It comprises a lighting fixture and a wireless communication device, The aforementioned processor, The external light sensor measures the external environment and compares the sensor output obtained with the ambient light brightness with a preset ambient light threshold for issuing a warning. The system monitors whether the ambient light brightness is below the ambient light threshold, indicating an insufficient ambient light state. In the aforementioned state of insufficient ambient light, when the user of the head-mounted display detects an action that transitions from viewing the display to observing the outside world, the system executes control to display at least one of the lighting fixture selection screens or lighting fixture operation screens, which are linked to the head-mounted display, on the display. A head-mounted display integration system.

[0125] (Form of the sixth perspective) In a head-mounted display integration system that links a head-mounted display with a lighting fixture, Head-mounted displays are Processor and A display having opacity, An external light sensor, An obstacle detection sensor that detects the presence of objects around the user of the head-mounted display, A notification device that outputs notification information to the user, A wireless communication device that communicates wirelessly with lighting fixtures, Equipped with, The aforementioned processor, The external light sensor measures the external environment and compares the sensor output obtained with the ambient light brightness with a preset ambient light threshold for issuing a warning. The system monitors whether the ambient light brightness is below the ambient light threshold, indicating an insufficient ambient light state. In the aforementioned state of insufficient ambient light, when it is detected that a user wearing the head-mounted display has performed an action to prepare for viewing the outside world, based on the sensor output of the obstacle detection sensor, it is determined whether there is an obstacle within a distance range determined to determine whether or not it will interfere with the user's actions. If an obstacle is detected, the notification device performs output control to change the format of the notification information output regarding the lighting fixture according to the obstacle detection result. A head-mounted display integration system.

[0126] (The form of the seventh perspective) In a head-mounted display control method, The system compares the ambient brightness based on measurements from an ambient light sensor with a pre-set ambient light threshold for issuing a warning. If the ambient brightness is below the ambient light threshold, and the system detects that a user wearing a head-mounted display has performed an action to prepare for viewing the outside world, it executes control to output notification information indicating that visibility of the outside world has decreased, or a control signal for a lighting fixture. A method for controlling a head-mounted display. [Explanation of Symbols]

[0127] 1: HMD Integration System 2: Server 3: Lighting fixtures 4: Switch 9: Communication Network 10: HMD 11: Wearing body 16: Gyroscope sensor 111: Rear Camera 112: Front camera 113: Distance measuring sensor 114: Illuminance sensor 115: Accelerometer 116: Gyroscope sensor 117: Geomagnetic sensor 118: GPS receiver 119: Display 120: Network communication device 121: Mike 122: Speaker 123: Antenna 124: Timer 125: Processor 126: Program 127: Data 128: Memory 130: Power button 131: Light 140: Bus 201: Outdoor Light Monitoring Department 202: Image Analysis Department 203: Preparation operation detection unit 204: Notification Control Unit 205: Communication Control Unit 206: Risk Assessment Department 207: Light Control Unit 208: Display Control Unit 209: Lighting fixture control unit 210: Display switching unit 301: Screen 302: Exit Selection Screen 303: Operation screen 304: Operation screen 310: Warning message 331: Notification screen 332: Notification screen 333: Notification screen 340: Pause screen 410: Hazard Classification Table 500: Operation screen 501: Selection screen 502: Operation screen 503: Operation screen 504: Settings screen 511: Recommended lighting fixture selection screen 512: Recommended lighting fixture selection screen 513: Recommended lighting fixture selection screen 514: Recommended lighting fixture selection screen 521: Recommended Brightness Table 522: Brightness Recommendation Table 531: Recommended brightness suggestion screen 532: Recommended brightness suggestion screen 541 :Stop screen 542 :Stop screen

Claims

1. A method for controlling a head-mounted display, The light detection unit of the head-mounted display measures the brightness of the outside world. The control unit of the head-mounted display compares the brightness of the external environment measured by the light detection unit with an external light threshold set in advance for alerting the user. If the control unit determines that the brightness of the external environment is less than or equal to the external light threshold, and that the user wearing the head-mounted display has performed an external environment viewing preparation action to switch from viewing the display unit of the head-mounted display to viewing the external environment, it controls the head-mounted display's notification unit to output notification information indicating that the visibility of the external environment has decreased. Control method.

2. A control method according to claim 1, The imaging unit of the head-mounted display captures the facial region including the user's eyes to generate a facial image. The control unit analyzes the facial image generated by the imaging unit, classifies the user's eye condition into a risk level defined in multiple stages according to at least one of the user's dark adaptation speed information, the user's age, and the user's visual acuity, and controls the notification unit to output notification information, including a warning corresponding to the classified risk level. Control method.

3. A method for controlling a head-mounted display, The light detection unit of the head-mounted display measures the brightness of the outside world. The control unit of the head-mounted display compares the brightness of the external environment measured by the light detection unit with an external light threshold set in advance for alerting the user. If the control unit determines that the brightness of the external environment is less than or equal to the external light threshold, and that the user wearing the head-mounted display has performed an external environment viewing preparation action to switch from viewing the display unit of the head-mounted display to viewing the external environment, it controls the unit to increase the transparency of the display unit and turn on the light of the head-mounted display. Control method.

4. A control method according to claim 3, The imaging unit of the head-mounted display captures the outside world and generates an image of the outside world. When the control unit determines that the brightness of the outside world is below the ambient light threshold and that the user has performed the action to prepare for viewing the outside world, it performs subject recognition processing on the outside world image generated by the imaging unit to identify the switch of a lighting fixture present in the outside world, generates an image indicating the direction to the switch, and controls the display unit to display it. Control method.

5. A method for controlling a head-mounted display, The light detection unit of the head-mounted display measures the brightness of the outside world. The imaging unit of the head-mounted display captures the outside world and generates an image of the outside world. The control unit of the head-mounted display compares the brightness of the external environment measured by the light detection unit with an external light threshold set in advance for alerting the user. If the control unit determines that the brightness of the external environment is less than or equal to the external light threshold, and that the user wearing the head-mounted display has performed an external environment viewing preparation action to switch from viewing the display unit of the head-mounted display to viewing the external environment, the control unit displays the external environment image generated by the imaging unit on the display unit and controls the head-mounted display's lights to turn on. Control method.

6. A control method according to claim 5, When the control unit determines that the brightness of the outside world is below the ambient light threshold and that the user has performed the preparation action for viewing the outside world, it performs subject recognition processing on the outside world image to identify the switch of a lighting fixture present in the outside world, generates an image indicating the direction to the switch, and controls the display unit to display it. Control method.

7. A control method according to any one of claims 1 to 6, The aforementioned external view preparation operation is either turning off the power of the head-mounted display or terminating at least one application running on the head-mounted display. Control method.

8. A control method according to any one of claims 1 to 6, The aforementioned preparation operation for viewing the external world is the operation to stop the video currently playing on the head-mounted display. Control method.

9. A control method according to any one of claims 1 to 6, The control unit analyzes the sensor output measured by at least one of the gyro sensor and accelerometer mounted on the head-mounted display, and if it determines that the user's head movement is tilted downward by more than a predetermined angle threshold, it determines that the user has performed the action to prepare for viewing the outside world. Control method.

10. A control method according to claim 1 or claim 3, The imaging unit of the head-mounted display captures the outside world and generates an image of the outside world. The control unit analyzes the external image generated by the imaging unit and, if it determines that the user's action of placing their hand on the head-mounted display has been captured, determines that the user has performed the action of preparing to view the external world. Control method.

11. A control method according to claim 1 or claim 3, The imaging unit of the head-mounted display captures the facial region including the user's eyes to generate a facial image. The control unit analyzes the face image generated by the imaging unit and determines that the region in which the user's eyes are captured in the face image deviates from a predetermined standard position, and determines that the user has performed the action to prepare for viewing the outside world. Control method.

Images