Display device, display control device, method for controlling display device, and program

Abstract

To provide a technique for controlling display to be more suitable for a user when allowing the user to visually recognize a real space.SOLUTION: A display device mountable on a head of a user has a first state in which the user is allowed to visually recognize a field-of-view scene image that is an image of a real space, and a second state in which the user is allowed to directly visually recognize the real space, and includes a control unit configured to switch a state of the display device between the first state and the second state based on biological information of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a display device capable of optical see-through display and video see-through display, a display control device, a control method for a display device, and a program. [Background technology]

[0002] Head-mounted displays are known that are worn on a user's head and have a display unit positioned in front of the user's eyes. One display method for head-mounted displays is an optical see-through method, which uses an optical method such as a half mirror to display a predetermined superimposed image on the background of the real space that the user looks directly at. Another display method is a video see-through method, which displays a composite image by superimposing an image of the user's field of view captured by an imaging unit and the superimposed image.

[0003] Patent Document 1 describes a technology for controlling switching of display modes in a head-mounted display that combines an optical see-through system and a video see-through system. Patent Document 2 also discloses a technology for controlling switching of display modes in a head-mounted display that combines an optical see-through system and a video see-through system according to the situation in real space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2016-197816 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-083290 Summary of the Invention [Problem to be solved by the invention]

[0005] In this way, in Patent Document 2, the display method is switched between the optical see-through method and the video see-through method based on the situation in the real space. However, even if the state of the real space is the same, how each user perceives that state varies. For this reason, it is not possible to switch to a display method that is suitable for the user.

[0006] Therefore, an object of the present invention is to provide a technique for controlling the display to be more suitable for the user when allowing the user to view the real space. [Means for solving the problem]

[0007] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. A display device that can be worn on a user's head, the display device having a first state in which the user visually recognizes a field of view image that is an image of real space, and a second state in which the user directly visually recognizes the real space, an acquisition means for acquiring biometric information of the user; a control means for switching a state of the display device between the first state and the second state based on the biometric information of the user; The display device is characterized by having:

[0008] One aspect of the present invention is a method for producing a medicament for the treatment of a pulmonary arthritis. A method for controlling a display device that can be worn on a user's head, comprising: the display device has a first state in which the user visually recognizes a field of view image, which is an image of a real space, and a second state in which the user directly visually recognizes the real space; The control method includes: an acquisition step of acquiring biometric information of the user; a control step of switching a state of the display device between the first state and the second state based on the biometric information of the user; The present invention relates to a method for controlling a display device, comprising the steps of: [Effects of the Invention]

[0009] According to the present invention, when a user is allowed to view a real space, the display can be controlled to be more suitable for the user. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a hardware configuration diagram of an HMD according to a first embodiment. [Figure 2] 3A to 3C are diagrams illustrating display control of the HMD according to the first embodiment. [Figure 3] 4 is a flowchart of a display switching process according to the first embodiment. [Figure 4] FIG. 10 is a hardware configuration diagram of an HMD according to a second embodiment. [Figure 5] 10A and 10B are diagrams illustrating display control of an HMD according to a second embodiment. [Figure 6] 10 is a flowchart of a display switching process according to the second embodiment. [Figure 7] 10A and 10B are diagrams illustrating display control of an HMD according to a third embodiment. [Figure 8] 10 is a flowchart of a display switching process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present embodiment will be described in detail below with reference to the accompanying drawings. Note that the embodiment described below is an example of a method for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. Furthermore, each embodiment can also be combined as appropriate.

[0012] <Embodiment 1> 1 shows a part of a hardware configuration diagram of a head-mounted display (hereinafter referred to as "HMD") 100 according to the first embodiment. The HMD 100 is a display device that can be worn on the head of a user. Note that the first embodiment can also be applied to a method for controlling the display of the HMD 100 (display control method).

[0013] The HMD 100 has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103. The HMD 100 has an optical system 104, a state switching unit 105, an imaging processing unit 106, a development processing unit 107, and a network interface 108. The HMD 100 has a display image processing unit 109, a display control unit 110, a display unit 111, a pupil measurement unit 112, a storage device 113, a short-range communication interface 114, a superimposed image processing unit 115, and an internal bus 120. Note that part of the configuration of the HMD 100 (for example, other than the display unit 111) may be included in a display control device such as a computer, and the display control device may control the HMD 100.

[0014] The CPU 101 is a control unit that performs overall control of each component of the HMD 100 via an internal bus 120 using a program stored in a ROM 102. The results of the program execution by the CPU 101 can be projected (displayed) as an image by a display unit 111.

[0015] The ROM 102 is a storage medium such as a flash memory, and stores various setting information, application programs, and the like.

[0016] The RAM 103 is a storage medium having a memory and a work area used by the CPU 101. The RAM 103 is used to temporarily store various information such as image buffer and management information.

[0017] The network interface (IF) 108 is a module for connecting to a network.

[0018] The optical system 104 is, for example, an optical system related to imaging. The optical system 104 includes a lens, an aperture, and a shutter. The optical system 104 is, for example, an optical system related to display (display unit). The optical system 104 includes a transparent reflector such as a half mirror. The optical system 104 can switch, for example, between a "first state in which the user can view an image of real space displayed by the display unit 111" and a "second state in which the user can directly view the real space." To switch the state of the optical system 104, for example, a method described in Patent Document 1 can be used. In the first embodiment, the first state in which the user can view an image of real space displayed on the display unit 111 is called a "video see-through state." The second state in which the user can directly view the real space in front of the user's eyes is called an "optical see-through state."

[0019] The state switching unit 105 switches the state of the optical system 104 (=the state of the HMD 100) between the video see-through state and the optical see-through state under the control of the CPU 101.

[0020] The imaging processing unit 106 includes an imaging element (imaging unit; imaging device), etc. The imaging processing unit 106 executes an imaging operation to acquire an image of real space under the control of the CPU 101. The imaging processing unit 106 stores a RAW image obtained by the imaging operation in the RAM 103 via the internal bus 120.

[0021] The development processing unit 107 reads out a RAW image stored in the RAM 103. The development processing unit 107 then performs development processing on the read out RAW image. The development processing includes, for example, pixel interpolation processing, filter processing, resizing processing, color conversion processing, signal format conversion processing, brightness level adjustment, color correction, contrast adjustment, and the like. However, the development processing is not limited to these. The signal format conversion processing is, for example, processing to convert the image representation format from RGB format to YCbCr format. The development processing unit 107 stores the developed image in the RAM 103.

[0022] The superimposition image processing unit 115 generates display items (images for superimposition; superimposition images) such as character strings or icons. The generated display items are stored in the RAM 103.

[0023] The display control unit 110 operates in accordance with the state of the optical system 104 controlled by the state switching unit 105. In the video see-through state, for example, the display control unit 110 reads out a developed image from the RAM 103. Then, the display control unit 110 generates an image for display based on the read image. For example, the display control unit 110 applies scaling processing to the image in accordance with the resolution of the display unit 111 to generate the image for display. The display control unit 110 outputs an image in which a display item (a display item generated by the superimposed image processing unit 115) is superimposed on the generated image for display to the display unit 111.

[0024] On the other hand, in the optical see-through state, for example, the display control unit 110 outputs the display item generated by the superimposed image processing unit 115 to the display unit 111 without outputting the image after development processing.

[0025] The display unit 111 displays the image output from the display control unit 110 .

[0026] The pupil measurement unit 112 detects the state of the pupil and the position (point of view; line of sight) at which the user wearing the HMD 100 is looking. The pupil measurement unit 112 also measures changes in the size of the user's pupil.

[0027] The storage device 113 is a storage medium that stores various data such as applications.

[0028] The short-range communication interface 114 is an interface used for communication with the controller. The user can input gestures to the HMD 100 by moving the controller they are holding. The user can also give instructions to the HMD 100 (user terminal) by operating buttons or a joystick provided on the controller.

[0029] 2A to 2C explain display control of the HMD 100 according to the first embodiment. Here, the explanation will be made with reference to the field of view of a car driver as an example. In the first embodiment, the CPU 101 controls the state of the optical system 104 (the state of the HMD 100) in accordance with the state of the user's pupil acquired by the pupil measurement unit 112, thereby performing control processing for the display of the HMD 100.

[0030] 2A shows the optical see-through state. The state switching unit 105 sets the state of the optical system 104 to the optical see-through state. Display items 201, 202 (images for superimposition, such as character strings or icons) generated by the superimposition image processing unit 115 are displayed on the display unit 111. In other words, the optical see-through state is a state in which the user sees the display items together with the field of view (the actual real space visible with the naked eye).

[0031] 2B is a diagram illustrating a display example of the HMD 100 when the display control process of the HMD 100 according to embodiment 1 is not performed. Backlighting due to the setting sun or the like occurs in the center of the driver's field of vision, and the display is dazzling to the user.

[0032] 2C is a diagram illustrating a case where the display control process of the HMD 100 according to the first embodiment is performed. The state switching unit 105 sets the state of the optical system 104 to the video see-through state. The display unit 111 displays display items 201 and 202 and an "image of the field of view captured and developed by the imaging processing unit 106 and the development processing unit 107." In other words, the video see-through state is a state in which the user is viewing an image obtained by combining an image of the driver's field of view with the display items.

[0033] (Regarding display switching processing) 3 is a flowchart showing the display switching process according to embodiment 1. The following process is realized by the CPU 101 controlling each component of the HMD 100 in accordance with a program stored in the ROM 102.

[0034] In step S301, CPU 101 controls state switching unit 105 to set the state of optical system 104 to the optical see-through state. In the optical see-through state, CPU 101 controls superimposed image processing unit 115 and display control unit 110 to display display items generated by superimposed image processing unit 115 on display unit 111. In this state, display unit 111 transmits light, so the user directly views the field of view (real space) through display unit 111. Therefore, by the processing of step S301, CPU 101 displays the display items on display unit 111 so that the user can visually recognize that the display items are arranged in the field of view.

[0035] In step S302, the CPU 101 controls the pupil measurement unit 112 to The pupil state of the user wearing the HDM 100 is measured. Then, based on the user's pupil state, CPU 101 determines whether the user is experiencing glare in the field of view (i.e., outside HDM 100). For example, CPU 101 determines whether the user is experiencing glare based on changes in pupil size and eyeball position. The correspondence between the measurement result of the pupil state and the user's feeling of glare is determined experimentally, and a table indicating the correspondence can be read from ROM 102. Therefore, CPU 101 can determine whether the user is experiencing glare by determining whether the measurement result of the pupil state is a specific pupil state corresponding to a specific glare pattern described in the table.

[0036] If it is determined that the user is feeling dazzled, the process proceeds to step S303. If it is determined that the user is not feeling dazzled, the process proceeds to step S301. In this case, CPU 101 continues the optical see-through state.

[0037] In step S303, the CPU 101 controls the state switching unit 105 to set (switch) the state of the optical system 104 to the video see-through state.

[0038] In step S304, CPU 101 controls imaging processing unit 106 and development processing unit 107 to generate an image (hereinafter referred to as a "field of view scene image") by capturing and developing the field of view scene (real space). Furthermore, CPU 101 controls superimposition image processing unit 115 and display control unit 110 to generate a composite image by superimposing the display item generated by superimposition image processing unit 115 on the field of view scene image. CPU 101 displays the composite image on display unit 111.

[0039] In step S305, CPU 101 measures (detects) the user's viewpoint position (the position where the user is looking) using pupil measurement unit 112. Then, CPU 101 acquires the brightness of the viewpoint position in the field of view scene image based on the field of view scene image and the viewpoint position by display control unit 110. Note that instead of the brightness of the viewpoint position in the field of view scene image, the user's pupil state may be acquired.

[0040] In step S306, the CPU 101 performs appropriate exposure correction so that the user does not perceive the visual field scene image (composite image) as being too dazzling. In the exposure correction, for example, the optical system 104 is controlled to adjust the aperture and shutter speed. The image capture processing unit 106 is also controlled to adjust the ISO sensitivity.

[0041] For this reason, CPU 101 calculates adjustment values ​​(adjustment values ​​for aperture, shutter speed, and ISO sensitivity) according to the brightness of the viewpoint position (or at least one of the brightness of the viewpoint position and the state of the user's pupil). Then, CPU 101 applies the adjustment values ​​to imaging processing unit 106 (imaging unit). The correspondence between the brightness of the viewpoint position (or at least one of the brightness of the viewpoint position and the state of the user's pupil) and appropriate exposure compensation is determined by verification, and a table indicating the correspondence can be read from ROM 102.

[0042] In step S307, the CPU 101 controls the imaging processing unit 106 and the development processing unit 107 to which the adjustment value has been applied to re-image and develop the user's field of view, in order to reflect the exposure correction in the field of view image. As a result, the CPU 101 updates the field of view image.

[0043] In step S308, similar to step S305, CPU 101 measures the user's viewpoint position (the position where the user is looking) by detecting pupil movement using pupil measurement unit 112. Then, CPU 101 acquires the brightness of the viewpoint position in the field of view scene image based on the field of view scene image and the viewpoint position. Note that, similar to step S305, Instead of the brightness at the viewpoint position in the image, the pupil measurement unit 112 may acquire the state of the user's pupils.

[0044] In step S309, CPU 101 performs appropriate display image processing so that the user does not perceive the visual field scene image as being too dazzling. In the display image processing, for example, CPU 101 calculates adjustment values ​​for at least one of brightness, contrast, and color processing according to the measurement results of the brightness of the viewpoint position in the visual field scene image (or at least one of the brightness of the viewpoint position and the state of the user's pupils). CPU 101 then controls development processing unit 107 to adjust at least one of the brightness, contrast, and color of the visual field scene image based on the adjustment values. The correspondence between the brightness of the viewpoint position and the appropriate display image processing is determined experimentally, and a table showing the correspondence can be read out from ROM 102.

[0045] In step S310, CPU 101 controls pupil measurement unit 112 to measure the state of the user's pupil. As a result, CPU 101 determines whether the user is feeling dazzled by what is ahead of their field of vision (i.e., the video see-through image displayed on display unit 111). For example, similar to step S302, CPU 101 determines whether the user is feeling dazzled based on changes in pupil size, eyeball position, etc. If it is determined that the user is feeling dazzled, the process proceeds to step S304. In this case, CPU 101 continues the video see-through state and further performs appropriate exposure correction and display image processing in steps S305 and onward. If it is determined that the user is not feeling dazzled, the process proceeds to step S311.

[0046] In step S311, CPU 101 acquires the "correction value for exposure correction in step S306 (hereinafter referred to as the 'exposure correction value')" and the "correction value for display image processing in step S309 (hereinafter referred to as the 'image correction value')."

[0047] In step S312, CPU 101 calculates a correction value (amount of adjustment related to the brightness of the field of view scene image) that comprehensively considers the exposure correction value and the image correction value, and determines whether the correction value is equal to or greater than threshold value Th1. The correspondence between the correction value and the brightness of real space is determined experimentally, and a table showing the correspondence can be read from ROM 102. If it is determined that the correction value that comprehensively considers the exposure correction value and the image correction value is equal to or greater than threshold value Th1, CPU 101 determines that the field of view scene image is in a state where it is perceived as dazzling to the user, because a large amount of adjustment has been made to the brightness of the field of view scene image. In this case, the process proceeds to step S304, and the video see-through state continues.

[0048] On the other hand, if it is determined that the correction value obtained by comprehensively considering the exposure correction value and the image correction value is less than the threshold value Th1 (less than the specific threshold value), the CPU 101 determines that the current visible scenery is not dazzling to the user. In this case, the process proceeds to step S301. In this case, in step S301, the CPU 101 controls the state switching unit 105 to set the state of the optical system 104 to the optical see-through state.

[0049] Through the above process, the HMD 100 automatically switches between the optical see-through state and the video see-through state according to the user's biological information (for example, changes in pupil state). This allows for more appropriate display switching according to the user's state than when switching states based on, for example, real-space information detected by a sensor.

[0050] <Embodiment 2> In the first embodiment, the HMD 100 switches between an optical see-through state and a video see-through state depending on whether the user is experiencing glare. However, unlike the video see-through state in which the colors of the background view image and the displayed items (superimposed images) can be distinguished, in the conventional optical see-through state, it may be difficult to distinguish the colors of the background and the displayed items. In this case, In some cases, after switching from the video see-through state to the optical see-through state, the difficulty in seeing the display items becomes apparent. For example, a case can be imagined in which a blue display item is displayed on a visual scene of a blue sky. Therefore, in the second embodiment, after switching from the video see-through state to the optical see-through state, if the visual scene of the user and the display item are the same color and difficult to see, the HMD 100 changes the color of the display item.

[0051] 4 shows a part of the hardware configuration of an HMD 400 according to embodiment 2. The HMD 400 includes a color component measurement unit 401 and a color conversion processing unit 402 in addition to the components of the HMD 100 according to embodiment 1.

[0052] A color component measurement unit 401 measures the color components of an image stored in the RAM 103. The measured color components are quantified, for example, using RGB.

[0053] The color conversion processing unit 402 performs color conversion on the image. One method of color conversion is to convert the color of each pixel using a lookup table, but the second embodiment is not limited to this method.

[0054] 5A and 5B are diagrams illustrating an example of a display on the HMD 400 according to the second embodiment. In FIGS. 5A and 5B, a user is viewing a visual field and a display item 501 in an optical see-through state. Here, the visual field of a car driver will be described as an example. FIG. 5A is a diagram illustrating a case where the processing according to the second embodiment is not performed. The display item 501 is navigation information from a navigation system. If the color of the user's visual field is blue sky and the color of the display item 501 is also blue, the user's visual field and the display item 501 have the same color, making it difficult for the user to see the display item 501. FIG. 5B is a diagram illustrating a case where the processing according to the second embodiment is performed. The color of the navigation information in the display item 501 is changed to a color different from the color of the user's visual field, making the display item 501 easier for the user to see.

[0055] Fig. 6 is a flowchart showing the process (the process of changing the color of a display item according to the second embodiment) executed between step S301 and step S302 in the flowchart of Fig. 3. The following process is realized by the CPU 101 controlling each component of the HMD 100 in accordance with a program stored in the ROM 102.

[0056] Between step S301 and step S302 in the flowchart of FIG. 3, new processing of steps S601 to S603 as shown in the flowchart of FIG. 6 is executed.

[0057] In step S601, the CPU 101 controls the imaging processing unit 106 and the development processing unit 107 in the background to capture and develop the field of view scene, thereby generating a field of view scene image. The CPU 101 sequentially stores the generated field of view scene images in the RAM 103.

[0058] In step S602, CPU 101 controls color component measurement unit 401 to measure the color components of "the display item generated by superimposed image processing unit 115 in step S301" and "the field of view scene image generated by development processing unit 107," and determines the difference between the two color components. In the example of FIG. 5A, CPU 101 determines the display position of display item 501 from the display coordinates of display item 501, and measures the color components of the field of view scene image at the display position (superimposed position) of display item 501. CPU 101 then determines the difference between the color components of display item 501 and the color components of the field of view scene image at the display position of display item 501. Note that CPU 101 may also determine the difference between the color components of display item 501 and the average color components of the field of view scene image.

[0059] It is determined that the difference in color components between the display item and the field of view scene image is equal to or greater than the threshold value Th2. If it is determined that the color of the field of view scene image is different from the color of the display item, the process proceeds to step S302. If it is determined that the difference in color component between the display item and the field of view scene image is less than the threshold value Th2, the process proceeds to step S603.

[0060] In step S603, CPU 101 controls color conversion processing unit 402 to perform color conversion processing of the display items. That is, if it is determined that the field of view scene image and the display items are the same color and difficult to see, CPU 101 controls color conversion processing unit 402 to perform color conversion processing of the display items. The contrast (correspondence) between the color of the field of view scene image and the color that makes the display items easy to see is determined by verification. The color conversion processing is performed by reading a table indicating the correspondence from ROM 102.

[0061] By the above process, after switching to the optical see-through state, if the background of the field of view and the displayed item are the same color and difficult to see, the HMD 100 changes the color of the displayed item, thereby enabling the HMD 100 to display the displayed item in a way that is easy for the user to recognize.

[0062] <Embodiment 3> In the second embodiment, the HMD 400 changes the color of the display item when the background of the field of view and the display item are the same color and difficult to see after switching to the optical see-through state. However, although the user can see the display item in the video see-through state, after switching to the optical see-through state, there is a problem that the display item becomes difficult to see when the background becomes a direct view.

[0063] For example, in a situation where the scene in real space is dazzling and the state is switched to video see-through, the user may be able to see bright display items even if they are superimposed because the field of view image is corrected in the video see-through state. On the other hand, in a situation where the scene in real space becomes slightly dark and the state is switched to optical see-through, the scene in real space is still bright, so the user may have difficulty seeing the display items.

[0064] As a specific example, if only the upper left corner of the field of view is bright due to road lighting at night, it is difficult for the user to see a display item superimposed at the same coordinate as the road lighting. Therefore, in the third embodiment, a method for partially displaying the field of view image when the user feels dazzled by part of the field of view will be described.

[0065] The hardware configuration diagram of the HMD 100 according to the third embodiment is the same as that of the first embodiment, and therefore a description thereof will be omitted.

[0066] 7A and 7B are diagrams illustrating an example of a display on the HMD 100 according to the third embodiment. FIGS. 7A and 7B illustrate a state in which a user is viewing a field of view and a display item in an optical see-through state. Here, an example will be described using the field of view of a car driver at night. FIG. 7A is a diagram illustrating a case in which processing according to the third embodiment is not performed. Display item 701 is navigation information from a navigation system. Because the display item 701 being viewed by the user overlaps with a road light in the field of view, the display item 701 is difficult for the user to see. FIG. 7B is a diagram illustrating a case in which processing according to the third embodiment is performed. Because the object on which the display item 701 is superimposed has been replaced with a field of view image, the display item 701 is easy for the user to see.

[0067] 8 is a flowchart showing a process (display control process according to the third embodiment) executed between step S301 and step S302 in the flowchart of FIG. 3. The following process is performed by the CPU 101 controlling each component of the HMD 100 in accordance with a program stored in the ROM 102. This is achieved by controlling the

[0068] Between step S301 and step S302 in the flowchart of FIG. 3, new processing of steps S801 to S807 as shown in the flowchart of FIG. 8 is executed.

[0069] In step S801, the CPU 101 controls the image capturing unit 106 and the development unit 107 in the background to generate an image that has been captured and developed. The CPU 101 sequentially stores the generated images in the RAM 103 as field of view images.

[0070] In step S802, CPU 101 controls pupil measurement unit 112 to measure the user's gaze position (the direction of the line of sight) and pupil state. Then, based on the user's gaze position and pupil state, CPU 101 determines whether the user is looking at a displayed item and is feeling dazzled. CPU 101 also acquires the coordinates of the user's gaze position from pupil measurement unit 112. Based on the coordinates of the gaze position and the coordinates of the displayed item, CPU 101 determines whether the user is looking at a displayed item. For example, similar to step S302, CPU 101 determines whether the user is feeling dazzled based on changes in pupil size, eyeball position, etc.

[0071] If it is determined in step S802 that the user is looking at the display item and is feeling dazzled, the process proceeds to step S803. If it is determined that the user is not looking at the display item or is not feeling dazzled, the process proceeds to step S302.

[0072] In step S803, CPU 101 performs appropriate exposure compensation so that the field of view image extracted in step S805 (described later) is not perceived as dazzling by the user. For example, exposure compensation involves controlling optical system 104 to adjust the aperture or shutter speed. For exposure compensation, image capture processing unit 106 is controlled to adjust the ISO sensitivity. CPU 101 calculates adjustment values ​​for aperture, shutter speed, ISO sensitivity, and the like, based on the pupil state measured by pupil measurement unit 112. The correspondence between the pupil state measurement results and appropriate exposure compensation is determined experimentally, and a table showing the correspondence can be read from ROM 102.

[0073] In step S804, the CPU 101 controls the imaging processing unit 106 and the development processing unit 107 to perform imaging and development processing of the field of view scene in order to reflect the exposure correction in the field of view scene image. In this way, the CPU 101 updates the field of view scene image.

[0074] In step S805, CPU 101 cuts out only the area corresponding to the display item from the field of view scene image generated by development processing unit 107 in step S804. As an example, area 702 shown in FIG. 7C is an area cut out from the field of view scene image. Hereinafter, the image of the area cut out from the field of view area image will be referred to as a "cut-out image."

[0075] In step S806, CPU 101 controls display control unit 110 to composite the display item onto the cropped image. As a result, in the optical see-through state, in area 702 shown in Fig. 7C, the display item is superimposed on the cropped image of the field of view scene image updated in step S804.

[0076] In step S807, the CPU 101 performs appropriate display image processing so that the user does not find the extracted image too dazzling. In the display image processing, for example, the development processing unit 107 is controlled to adjust at least one of the brightness, contrast, and color processing of the extracted image displayed on the display unit 111. The CPU 101 calculates an adjustment value for at least one of the brightness, contrast, and color processing of the extracted image according to the measurement result of the pupil state by the pupil measurement unit 112. The correspondence between the measurement results of the pupil state and the appropriate display image processing has been determined through verification, and a table showing the correspondence can be read out from the ROM 102.

[0077] In steps S801 to S807, CPU 101 performs the above-described processing for each frame of the field of view scene image.

[0078] By the above processing, when only a part of the field of view is perceived as dazzling in the optical see-through state and the displayed items are difficult to see, the HMD 100 displays a field of view image that has been subjected to exposure correction and image processing on a part of the display surface, thereby enabling the HMD 100 to display display items that are easy for the user to recognize.

[0079] If CPU 101 determines that there are multiple ranges (areas) in the field of view that the user finds dazzling, it may display the cut-out images so that they appear to overlap with each of the multiple ranges. Also, if CPU 101 determines that there are multiple ranges in the field of view that the user finds dazzling in step S802, it may display the cut-out images by prioritizing display items that the user looks at frequently (for example, areas such as navigation data).

[0080] In each embodiment, the state of the optical system 104 is switched according to the pupil state. However, the state of the optical system 104 may be switched according to any biological information that responds to brightness. For example, when a person feels dazzled, they may furrow their eyebrows. Therefore, the state of the eyebrows may be determined from an image of the space between the eyebrows, and the state of the optical system 104 may be switched based on the state of the space between the eyebrows. Furthermore, when a person feels dazzled, they may squint their eyes. Therefore, the state of the eyes (eye size) may be determined from an image of the eyes, and the state of the optical system 104 may be switched based on the size of the eyes.

[0081] Furthermore, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than or equal to B, proceed to step S2." Conversely, "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2" may be read as "If A is greater than (higher than) B, proceed to step S1; if A is less than (lower than) B, proceed to step S2." Therefore, unless a contradiction arises, "greater than or equal to A" may be read as "greater than (higher; longer; more) than A," and "less than or equal to A" may be read as "less than (lower; shorter; fewer) than A." Furthermore, "greater than (higher; longer; more) than A" may be read as "greater than or equal to A," and "less than (lower; shorter; fewer) than A" may be read as "less than or equal to A."

[0082] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0083] The processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays), CPLDs (Complex Programmable LSIs), and so on. BLE Logic Device).

[0084] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0085] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0086] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A display device that can be worn on a user's head, the display device having a first state in which the user visually recognizes a field of view image that is an image of real space, and a second state in which the user directly visually recognizes the real space, an acquisition means for acquiring biometric information of the user; a control means for switching a state of the display device between the first state and the second state based on the biometric information of the user; A display device comprising: (Configuration 2) The control means determining whether the user feels dazzled based on the biometric information of the user; When the state of the display device is the second state and it is determined that the user feels dazzled, the state of the display device is switched to the first state. 2. The display device according to configuration 1, (Configuration 3) When the state of the display device is the first state, the control means determines that the user does not feel dazzled, and the amount of adjustment related to the brightness of the field of view scene image is less than a first threshold, and switches the state of the display device to the second state. 3. The display device according to configuration 2. (Configuration 4) Further, the imaging means acquires the field of view image by imaging, When the state of the display device is the first state, the control means calculates an adjustment value of at least one of an aperture, a shutter speed, and an ISO sensitivity of the imaging means based on the biometric information of the user, and controls imaging of the imaging means based on the calculated adjustment value. 4. The display device according to any one of configurations 1 to 3. (Configuration 5) Further, the imaging means acquires the field of view image by imaging, When the state of the display device is the first state, the control means calculates an adjustment value of at least one of an aperture, a shutter speed, and an ISO sensitivity of the imaging means based on the brightness of the position where the user is looking in the field of view scene image, and controls imaging of the imaging means based on the calculated adjustment value. 5. The display device according to any one of configurations 1 to 4. (Configuration 6) When the state of the display device is the first state, the control means adjusts at least one of brightness, contrast, and color of the field of view scene image based on the biometric information of the user. 6. The display device according to any one of configurations 1 to 5. (Configuration 7) When the state of the display device is the first state, the control means adjusts at least one of the brightness, contrast, and color of the field of view scene image based on the brightness of the position in the field of view scene image where the user is looking. 7. The display device according to any one of configurations 1 to 6. (Configuration 8) The display device further comprises a display means for displaying the field of view image and display items when the state of the display device is the first state, and for displaying the display items so that the user can visually recognize that they are arranged in the real space when the state of the display device is the second state. 8. The display device according to any one of configurations 1 to 7. (Configuration 9) a measuring means for measuring color components of the display item and the view scene image; When the state of the display device is the first state and the difference between the color component of the field of view image and the color component of the display item is less than a second threshold, a color conversion processing means performs color conversion processing of the display item; 9. The display device according to configuration 8, further comprising: (Configuration 10) The color component of the field of view scene image is a color component of the field of view scene image at a position where the display item is superimposed. 10. The display device according to configuration 9, (Configuration 11) When the display device is in the second state, the control means causes the display means to display a partial range of the field of view scene image so as to overlap the display item when the user is looking at the display item and the biological information indicates a specific state. 11. The display device according to any one of configurations 8 to 10. (Configuration 12) the biological information includes at least one of information on a change in eyeball position and a change in pupil size; 12. The display device according to any one of configurations 1 to 11. (Configuration 13) A display device having a first state in which a user visually recognizes a field of view image, which is an image of real space, and a second state in which the user directly visually recognizes the real space, and a display control device that controls the display device, which is wearable on the user's head, an acquisition means for acquiring biometric information of the user; a control means for switching a state of the display device between the first state and the second state based on the biometric information of the user; A display control device comprising: (method) A method for controlling a display device that can be worn on a user's head, comprising: the display device has a first state in which the user visually recognizes a field of view image, which is an image of a real space, and a second state in which the user directly visually recognizes the real space; The control method includes: an acquisition step of acquiring biometric information of the user; The display is changed between the first state and the second state based on the biometric information of the user. a control step of switching the state of the display device; 1. A method for controlling a display device, comprising: (program) 13. A program for causing a computer to function as each means of the display device according to any one of configurations 1 to 12. [Explanation of symbols]

[0087] 100: HMD (display device), 101: CPU (control unit), 104: Optical system, 112: Pupil measurement section (acquisition section)

Claims

1. A display device that can be worn on a user's head, the display device having a first state in which the user visually recognizes a field of view image that is an image of real space, and a second state in which the user directly visually recognizes the real space, an acquisition means for acquiring biometric information of the user; a control unit that switches the state of the display device between the first state and the second state based on the biometric information of the user; A display device comprising:

2. The control means determining whether the user feels dazzled based on the biometric information of the user; When the state of the display device is the second state and it is determined that the user feels dazzled, the state of the display device is switched to the first state.

2. The display device according to claim 1.

3. When the state of the display device is the first state, the control means determines that the user does not feel dazzled, and the amount of adjustment related to the brightness of the field of view scene image is less than a first threshold, and switches the state of the display device to the second state.

3. The display device according to claim 2.

4. Further, the imaging means acquires the field of view image by imaging, When the state of the display device is the first state, the control means calculates an adjustment value of at least one of an aperture, a shutter speed, and an ISO sensitivity of the imaging means based on the biometric information of the user, and controls imaging by the imaging means based on the calculated adjustment value.

2. The display device according to claim 1.

5. Further, the imaging means acquires the field of view image by imaging, When the state of the display device is the first state, the control means calculates an adjustment value of at least one of an aperture, a shutter speed, and an ISO sensitivity of the imaging means based on the brightness of the position where the user is looking in the field of view scene image, and controls imaging of the imaging means based on the calculated adjustment value.

2. The display device according to claim 1.

6. When the state of the display device is the first state, the control means adjusts at least one of brightness, contrast, and color of the field of view scene image based on the biometric information of the user.

2. The display device according to claim 1.

7. When the state of the display device is the first state, the control means adjusts at least one of the brightness, contrast, and color of the field of view scene image based on the brightness of the position in the field of view scene image where the user is looking.

2. The display device according to claim 1.

8. a display means for displaying the field of view image and display items when the state of the display device is the first state, and for displaying the display items so that the user can visually recognize that they are arranged in the real space when the state of the display device is the second state; Furthermore, 2. The display device according to claim 1.

9. a measuring means for measuring color components of the display item and the view scene image; When the state of the display device is the first state and the difference between the color component of the field of view scene image and the color component of the display item is less than a second threshold, a color conversion processing means performs a color conversion process on the display item; 9. The display device according to claim 8, further comprising:

10. The color component of the field of view scene image is a color component of the field of view scene image at a position where the display item is superimposed.

10. The display device according to claim 9.

11. When the display device is in the second state, the control means causes the display means to display a partial range of the field of view scene image so as to overlap the display item when the user is looking at the display item and the biological information indicates a specific state.

9. The display device according to claim 8.

12. the biological information includes at least one of information on a change in eyeball position and a change in pupil size; 2. The display device according to claim 1.

13. A display device having a first state in which a user visually recognizes a field of view image, which is an image of real space, and a second state in which the user directly visually recognizes the real space, and a display control device that controls the display device, which is wearable on the user's head, an acquisition means for acquiring biometric information of the user; a control unit that switches the state of the display device between the first state and the second state based on the biometric information of the user; A display control device comprising:

14. A method for controlling a display device that can be worn on a user's head, comprising: the display device has a first state in which the user visually recognizes a field of view image, which is an image of a real space, and a second state in which the user directly visually recognizes the real space, The control method includes: an acquisition step of acquiring biometric information of the user; a control step of switching a state of the display device between the first state and the second state based on the biometric information of the user; 1. A method for controlling a display device, comprising:

15. A program for causing a computer to function as each of the means of the display device according to any one of claims 1 to 12.

Images