Display device, display method, and program

Abstract

To provide a display device, display method, and program that can reduce the psychological stress on users caused by visual input. [Solution] The display device according to the present disclosure includes: a transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light that is transmitted; a gaze detection unit that detects the user's gaze information based on first image information showing the user's eyeballs; an expression determination unit that determines the user's facial expression based on second image information showing the user's face; and an occlusion range setting unit that, when the facial expression determined by the expression determination unit is the facial expression to be controlled, sets an occlusion range that sets the light transmittance to a predetermined value based on the gaze information and the emotion score.

Description

Technical Field

[0006] , ,

[0005] ,

[0001] The present disclosure relates to a display device, a display method, and a program.

Background Art

[0002] People may feel psychological stress or trauma depending on what they see. Although it may be possible to reduce the stress by avoiding seeing objects or events that cause discomfort, not all people can voluntarily divert their line of sight. On the other hand, smart glasses that display various image information in the user's line of sight direction are known.

[0003] For example, Patent Document 1 below discloses a smart glass (see-through type head-mounted display) used for a videophone or the like, which includes a camera equipped with a fish-eye lens or a wide-angle lens, and fixing means such as an arm for connecting and fixing the camera to the smart glass, and is characterized in that the camera can be fixed at a position where it can photograph the upper part from the user's shoulder wearing the smart glass from the front.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, although the smart glass described in Patent Document 1 above can improve the convenience of the user, it cannot reduce the mental stress caused by the object being viewed by the user or improve the state of anxiety due to trauma based on the user's past experience.

[0006] In light of the above issues, this disclosure aims to provide a display device, a display method, and a program that can reduce the psychological stress on users caused by visual inspection. [Means for solving the problem]

[0007] To solve the above-mentioned problems and achieve the objective, the display device according to this disclosure includes: a transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light that is transmitted; a gaze detection unit that detects the user's gaze information based on first image information showing the user's eyeballs; an expression determination unit that determines the user's facial expression based on second image information showing the user's face; and an occlusion range setting unit that, when the facial expression determined by the expression determination unit is the facial expression to be controlled, sets an occlusion range that sets the light transmittance to a predetermined value based on the gaze information and the emotion score.

[0008] To solve the above-mentioned problems and achieve the objective, the display method according to the present disclosure is a display method using a display device comprising: a transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light transmitted; a step of detecting the user's gaze information based on first image information showing the user's eyeballs; a step of determining the user's facial expression based on second image information showing the user's face; and a step of setting the light transmittance and the shielding range of the transmittance adjustment unit based on the gaze information and the emotion score.

[0009] To solve the above-mentioned problems and achieve the objective, the program relating to this disclosure is a program that causes a computer of a display device, which includes a transmittance adjustment unit that adjusts the light transmittance indicating the proportion of light transmitted, to execute processing, and includes the steps of: detecting the user's gaze information based on first image information showing the user's eyeballs; determining the user's facial expression based on second image information showing the user's face; and setting the light transmittance and the shielding range of the transmittance adjustment unit based on the gaze information and the emotion score. [Effects of the Invention]

[0010] According to this disclosure, it is possible to provide a display device, a display method, and a program that can reduce the psychological stress on users caused by viewing. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a diagram illustrating the overview of the display device related to this disclosure. [Figure 2] Figure 2 shows an example configuration of the first embodiment of the display device according to this disclosure. [Figure 3] Figure 3 shows a first example of the shielding range according to this disclosure. [Figure 4] Figure 4 is a diagram illustrating in detail a first example of the shielding range relating to this disclosure. [Figure 5] Figure 5 shows a second example of the shielding range according to this disclosure. [Figure 6] Figure 6 is a flowchart showing the flow of the first embodiment of the display method according to this disclosure. [Figure 7] Figure 7 is a flowchart showing a modified example of the first embodiment of the display method according to this disclosure. [Figure 8] Figure 8 shows an example configuration of a second embodiment of the display device according to this disclosure. [Figure 9] Figure 9 is a flowchart showing the flow of a second embodiment of the display method according to this disclosure. [Figure 10] Figure 10 shows an example configuration of a third embodiment of the display device according to this disclosure. [Figure 11] Figure 11 is a flowchart showing the flow of a third embodiment of the display method according to this disclosure. [Figure 12] Figure 12 shows an example configuration of the fourth embodiment of the display device according to this disclosure. [Figure 13] Figure 13 shows an example of information display in the absence of a shielding area as described in this disclosure. [Figure 14] Figure 14 shows an example of information display when there is a shielding area related to this disclosure. [Figure 15]FIG. 15 is a flowchart showing the flow of the fourth embodiment of the display method according to the present disclosure.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present disclosure will be described in detail based on the drawings. Note that the present disclosure is not limited by the embodiments described below.

[0013] (Overview of the Display Device) First, an overview of the display device 100 according to the present disclosure will be described using FIG. 1. FIG. 1 is a diagram for explaining the overview of the display device according to the present disclosure. As shown in FIG. 1, the display device 100 according to the present disclosure may be a so-called smart glass that a user wears on the head and visually recognizes an image displayed at a location corresponding to the lens portion of the glasses.

[0014] The display device 100 is, for example, a glasses-type device that displays various virtual video information and image information such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). The display device 100 is not limited to the glasses type shown in FIG. 1, and may be a goggle-type device that covers the front of the user's eyes. Further, the display device 100 may be a head-mounted type device such as a VR head-mounted display. Note that the display device 100 shields a part of the visual field of the user wearing the display device 100.

[0015] As shown in FIG. 1, the display device 100 mainly includes a display unit 140, a transmittance adjustment unit 150, an imaging unit 160, and an acceleration sensor unit 170. The display device 100 also includes other components, which will be described later.

[0016] The display unit 140 is a display device that displays images by emitting light itself while transmitting ambient light emitted from the user's surrounding environment. The display unit 140 includes a display unit 140A for the right eye and a display unit 140B for the left eye. As shown in Figure 1, the display unit 140 is provided in front of the user's left and right eyes.

[0017] The transmittance adjustment unit 150 is located either behind or in front of the display surface of the display unit 140. The transmittance adjustment unit 150 is located parallel to the display surface of the display unit 140. The transmittance adjustment unit 150 can adjust the transmittance of light emitted from the user's surrounding environment for each region. The shielding range can be set in two dimensions, vertically and horizontally, or it can be divided into multiple regions horizontally and a part of these regions can be used as the shielding range. The shielding range of the transmittance adjustment unit 150 will be explained in detail later.

[0018] The first imaging unit 160A captures first image information, which is an image including the eyeball of the user wearing the display device 100. The first imaging unit 160A is located around the transmittance adjustment unit 150. The first imaging unit 160A may be implemented by, for example, a visible light camera or an infrared camera.

[0019] The second imaging unit 160B captures second image information, which is an image including the user's facial expression. The second imaging unit 160B may be positioned around the transmittance adjustment unit 150 and facing the user. The second imaging unit 160B may be implemented by, for example, a visible light camera or an infrared camera.

[0020] The acceleration sensor unit 170 detects the acceleration generated in the transmittance adjustment unit 150. The acceleration sensor unit 170 is an acceleration sensor that detects acceleration in three mutually orthogonal directions: the X-axis direction, the Y-axis direction, and the Z-axis direction.

[0021] As described above, the display device 100 estimates the user's gaze direction and the user's emotions based on the first image information obtained by the second imaging unit 160B and the second image information, and adjusts the light transmittance of the transmittance adjustment unit 150 corresponding to the user's gaze direction based on the estimated user emotions. The configuration of the display device 100 will be described in detail below.

[0022] (Display device configuration) (First Embodiment) Next, the configuration of the display device 100 according to this disclosure will be described with reference to Figure 2. Figure 2 is a diagram showing an example configuration of the first embodiment of the display device according to this disclosure. As shown in Figure 2, the display device 100 according to this disclosure comprises a communication unit 110, a storage unit 120, a control unit 130, a display unit 140, a transmittance adjustment unit 150, and an imaging unit 160. These configurations will be described in order below.

[0023] The communication unit 110 is responsible for transmitting and receiving various types of information between itself and external devices, either via wired or wireless connections. The communication unit 110 may, for example, be equipped with a GPS (Global Positioning System) antenna to receive radio waves from GPS satellites. In the case of a wired connection, the communication unit 110 may be equipped with interfaces such as a USB (Universal Serial Bus) terminal, an SDI (Serial Digital Interface) terminal, or an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal. In the case of a wireless connection, it may be implemented using an antenna that receives radio waves such as 5G or 6G, or by wireless LAN (Local Area Network) as defined in IEEE 802.11, Bluetooth (registered trademark), Wi-Fi (registered trademark), etc.

[0024] The memory unit 120 is a storage device that stores various types of information. The memory unit 120 comprises a main memory and an auxiliary storage device. The main memory may be implemented using semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), or flash memory. The auxiliary storage device may be implemented using a hard disk or SSD (Solid State Drive), for example.

[0025] The control unit 130 is a controller that performs various calculations and functions. The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs stored in the memory unit 120 using RAM as the working area. Alternatively, the control unit 130 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0026] As shown in Figure 3, the control unit 130 includes an acquisition unit 131, a gaze detection unit 132, an expression determination unit 133, an occlusion range setting unit 134, and a control signal transmission unit 135. The control unit 130 realizes these functions and performs these processes by reading and executing a program (software) from the storage unit 120. These functions of the control unit 130 may also be realized by electronic circuits. Furthermore, the control unit 130 may execute these processes with a single CPU, or it may have multiple CPUs and execute these processes in parallel with the multiple CPUs. These configurations will be described in order below.

[0027] The acquisition unit 131 acquires various types of information from devices inside or outside the display device 100. Specifically, the acquisition unit 131 acquires the first image information captured by the first imaging unit 160A and the second image information captured by the second imaging unit 160B from the imaging unit 160. Once the acquisition unit 131 has acquired the first image information and the second image information, it may store the acquired first image information and second image information in the storage unit 120.

[0028] The gaze detection unit 132 detects the direction of the gaze of the user wearing the display device 100 based on the first image information. For example, the gaze detection unit 132 detects the direction of the user's gaze by performing image recognition processing to detect objects on the first image information captured by the first imaging unit 160A and identifying the direction in which the user's eyeballs are pointing. Alternatively, the gaze detection unit 132 may detect the direction of the gaze using a three-dimensional eyeball model by detecting the user's pupil based on the first image information and the orientation of the user's face based on the second image information. Alternatively, the gaze detection unit 132 may irradiate the eyeball, including the fundus, with a laser to measure a first distance from the fundus and a second distance from the surface of the eyeball, and detect the direction of the user's gaze based on the measured first and second distances. Here, the gaze information may include a vector that shows the direction of the user's gaze in three dimensions, or it may include coordinate data of the user's point of fixation on a predetermined target surface.

[0029] The facial expression determination unit 133 determines the user's facial expression based on the second image information. The facial expression determination unit 133 detects six facial expressions (anger, sadness, fear, surprise, disgust, and joy) from the user's second image information based on Ekman theory. For example, in this case, fear and disgust are the facial expressions to be controlled. The facial expression determination unit 133 may determine whether an expression is a controlled expression if, for example, the controlled expression continues for a predetermined time (e.g., 2 seconds) or longer. Conversely, the facial expression determination unit 133 determines that an expression is not a controlled expression if the duration of the controlled expression is less than the predetermined time.

[0030] The facial expression determination unit 133 calculates an anger score, a fear score, a surprise score, a disgust score, and a joy score as multiple parameters of an emotion score that indicates the likelihood of the emotion expressed in the user's facial expression. Specifically, when annotating an image of a human face with six facial expressions, labels are assigned based on Ekman theory. The facial expression determination unit 133 may use the annotated image data as training data and classify the user's facial expression in the second image information using a trained model that has learned the characteristics of each facial expression using a CNN (Convolutional Neural Network) or the like, and calculate an emotion score that indicates the likelihood of each emotion.

[0031] The occlusion range setting unit 134 sets an occlusion range, which is the region in which the light transmittance is controlled. Specifically, if the light transmittance of the entire lens of the smart glasses is set low, it may obstruct the user's view while walking, potentially causing safety problems. Therefore, the occlusion range set by the occlusion range setting unit 134 may be set with a width W and a height H centered on the left and right lines of sight within the smart glasses, so as not to obstruct the entire field of view. Figure 4 shows a first example of an occlusion range set by the occlusion range setting unit 134. Figure 4 is a diagram showing a first example of an occlusion range according to this disclosure. As shown in Figure 4, the occlusion range may be set to an area centered on the user's left eye's line of sight direction Pa, with a width W to the left and a height H to the right and left. Similarly, the occlusion range for the user's right eye may be set to an area centered on the user's right eye's line of sight direction Pb, with a width W to the left and right and a height H to the right and left and right.

[0032] The occlusion range set by the occlusion range setting unit 134 will be explained in more detail with reference to Figure 4. Figure 4 is a diagram illustrating in detail a first example of the occlusion range according to this disclosure. Figure 4 shows an enlarged view of the occlusion range for the right eye shown in Figure 3. As shown in Figure 4, the occlusion range setting unit 134 sets the occlusion range at a position away from the upper and lower edges and the left and right edges of the area of ​​the transmittance adjustment unit 150 so as not to obstruct the user's entire field of vision. Specifically, the occlusion range setting unit 134 sets the occlusion range at a position away from the upper and lower edges of the area

[0033] The shielding area set by the shielding area setting unit 134 is not limited to this and may be set by a shape such as a circle. Figure 5 is a diagram showing a second example of the shielding area according to this disclosure. In Figure 5, the light transmittance is set radially with respect to the line of sight. The closer to the line of sight, the lower the light transmittance is set, and the further away from the line of sight, the higher the light transmittance is set. The shielding area setting unit 134 also controls the light transmittance of the liquid crystal elements of the transmittance adjustment unit 150. Here, the light transmittance of the liquid crystal elements is lowered to make it difficult for the user to see light from the outside. That is, the shielding area setting unit 134 lowers the light transmittance of a predetermined shielding area of ​​the smart glasses' glasses based on the user's line of sight direction obtained by the line of sight detection unit 132.

[0034] The control signal transmission unit 135 transmits a control signal to the transmittance adjustment unit 150 to control the transmittance adjustment unit 150. Specifically, the control signal transmission unit 135 transmits a control signal to the transmittance adjustment unit 150 that includes information on the shielding range set by the shielding range setting unit 134 and the light transmittance. As a result, the transmittance adjustment unit 150 can control the liquid crystal elements within the shielding range whose light transmittance has been appropriately adjusted to a predetermined value.

[0035] The display unit 140 is a display device that transmits ambient light. In other words, the display unit 140 is a transparent display that allows the user to see through to the area in front of their field of vision by transmitting light. The display unit 140 may be a self-emissive display such as an organic EL (Electro-Luminescence) display. The user can see objects within their field of vision through the display unit 140.

[0036] The transmittance adjustment unit 150 switches between transmitting and blocking incident light by controlling the transmittance of the incident light that has passed through the display unit 140. The transmittance adjustment unit 150 may, for example, be equipped with a liquid crystal element to enable transmittance adjustment. In this case, the transmittance adjustment unit 150 changes the orientation state of the liquid crystal by the applied voltage, thereby changing the transmittance of incident light to the display device 100 from transmission to blocking. The transmittance may be adjusted by adjusting the proportion of liquid crystal that changes the orientation state within a region, or by adjusting the orientation state of the liquid crystal. The transmittance adjustment unit 150 can change the orientation state of the liquid crystal at any location. That is, the transmittance adjustment unit 150 can control incident light from transmission to blocking at any location. The transmittance adjustment unit 150 may also be implemented, for example, by electrophoretic liquid crystal elements (Electrochromic Devices).

[0037] The imaging unit 160 captures an image. The imaging unit 160 includes the first imaging unit 160A and the second imaging unit 160B described above. The imaging unit 160 comprises an optical element and an image sensor. The optical element is, for example, an element that constitutes an optical system, such as a lens, mirror, prism, or filter. The image sensor is an element that converts light incident through the optical element into an image signal, which is an electrical signal. The image sensor is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0038] Furthermore, before setting the light transmittance of the shielding range set by the shielding range setting unit 134 to a low value, text information to inform the user of this may be displayed on the display unit 140. An LED (Light Emitting Diode) or the like installed within the user's field of view above the display unit 140 may be illuminated, or a speaker may be installed inside the display unit 140 to emit a beep or other sound to notify the user. If the user's line of sight shifts significantly within a predetermined time after notifying the user, the shielding area setting and light transmittance may be returned to their original settings.

[0039] As described above, the display device 100 can be configured to prevent the object causing the emotion from appearing in the user's field of vision when the user's facial expression is one of the controlled expressions such as fear or disgust. Therefore, the display device 100 can be provided that can reduce the user's psychological stress caused by viewing such an object.

[0040] (Regarding display methods) (First Embodiment) Next, a first embodiment of the display method relating to this disclosure will be described with reference to Figure 6. Figure 6 is a flowchart showing the flow of the first embodiment of the display method relating to this disclosure. The display method relating to this disclosure will be described step by step below, following the flow shown in Figure 6.

[0041] First, the acquisition unit 131 acquires the user's image information (step S101). Next, the facial expression determination unit 133 determines whether or not the facial expression is one of the controlled objects (step S102). Next, the gaze detection unit 132 detects the user's gaze direction based on the user's image information (step S103). Next, the occlusion range setting unit 134 sets the occlusion area and transmittance based on the user's gaze direction (step S104). Next, the transmittance adjustment unit 150 controls the transmittance of the lens in the occlusion area (step S105).

[0042] In step S102, if the expression is not the one being controlled (step S102: No), the shielding area and transmittance are restored to their original values.

[0043] According to the display method described above, if the user's facial expression, determined based on image information, matches the expression of the target object, the occlusion area and its transmittance can be set based on the user's gaze direction, and the transmittance of the lens in the occlusion area can be controlled. As a result, the user's gaze direction is obscured, preventing them from viewing objects that cause discomfort. Therefore, this display method can provide a way to reduce the user's mental stress.

[0044] In addition, similar to the modified example of the first embodiment of the display device 100 described above, the first embodiment of the display method may be as shown in Figure 7. Figure 7 is a flowchart showing a modified example of the first embodiment of the display method according to this disclosure. Below, a modified example of the first embodiment of the display method according to this disclosure will be described step by step in accordance with the flowchart shown in Figure 7.

[0045] First, the acquisition unit 131 acquires the user's image information (step S201). Next, the facial expression determination unit 133 determines whether the user's facial expression is the facial expression of the control target (step S202). If the user's facial expression is the facial expression of the control target (step S202: Yes), the gaze detection unit 132 acquires the user's gaze direction based on the user's image information (step S203). Next, the occlusion range setting unit 134 notifies the user that an occlusion area has been set (step S204). Next, the gaze detection unit 132 acquires the user's gaze direction based on the user's image information (step S205). Next, the gaze detection unit 132 determines whether the amount of movement of the user's gaze exceeds a predetermined value (step S206). If the amount of movement of the user's gaze exceeds a predetermined value (step S206: Yes), the occlusion range setting unit 134 sets the occlusion area and transmittance based on the gaze direction (step S207). Next, the transmittance adjustment unit 150 controls the transmittance of the lens in the shielding area (step S208).

[0046] In step S202, if the user's facial expression is not the facial expression of the controlled object (step S202: No), the shielding range setting unit 134 returns the shielding area and transmittance to their original values ​​(step S209).

[0047] Furthermore, in step S206, if the amount of movement of the user's gaze exceeds a predetermined value (step S206: Yes), the occlusion range setting unit 134 returns the occlusion area and transmittance to their original values ​​(step S209). Here, step S206 is defined as when the amount of movement of the user's gaze exceeds a predetermined value, but for example, it may also be defined as when the time the user keeps their left and right eyelids closed exceeds a predetermined time.

[0048] As described above, a modified version of the first embodiment of the display method can be used to notify the user before setting the occlusion range, thereby preventing the user from becoming disoriented due to a sudden obstruction of their view. Therefore, it is possible to provide a display device 100 that can reduce the psychological stress on the user caused by visual disturbances.

[0049] (Display device configuration) (Second embodiment) Next, the configuration of the second embodiment of the display device 100 according to this disclosure will be described with reference to Figure 8. Figure 8 is a diagram showing an example of the configuration of the second embodiment of the display device according to this disclosure. As shown in Figure 8, the second embodiment of the display device 100 according to this disclosure comprises a communication unit 110, a storage unit 120, a control unit 130, a display unit 140, a transmittance adjustment unit 150, and an imaging unit 160. Note that in the second embodiment of the display device 100, the control unit 130 comprises a confusion level estimation unit 136 and a shielding time setting unit 137, and the processing of the shielding range setting unit 134 differs from that of the first embodiment of the display device 100. Therefore, in the following, only the confusion level estimation unit 136, the shielding time setting unit 137, and the shielding range setting unit 134 of the control unit 130 will be described, and the description of the other components will be omitted.

[0050] The confusion level estimation unit 136 calculates the level of confusion by combining multiple user emotion scores obtained from the facial expression determination unit 133. The confusion level estimation unit 136 may calculate the level of confusion using, for example, one of the following two formulas. Confusion level = Fear score + Disgust score. Confusion level = Fear score + Disgust score - Joy. The confusion score can be set to a value between 0 and 100, for example.

[0051] The shielding time setting unit 137 sets the shielding time, which is the time for which the transmittance of the shielding range is maintained at a predetermined value. For example, the shielding time setting unit 137 sets the shielding time to increase in proportion to the degree of confusion. That is, the shielding time setting unit 137 lengthens the shielding time as the degree of confusion increases, and shortens the shielding time as the degree of confusion decreases. Here, the proportionality constant with respect to the degree of confusion may be arbitrarily set so that the shielding time is appropriate.

[0052] The shielding range setting unit 134 adjusts the shielding range and light transmittance according to the degree of confusion calculated by the confusion estimation unit 136 described above. For example, the shielding range setting unit 134 widens the shielding range as the degree of confusion increases, and narrows the shielding range as the degree of confusion decreases. Also, the shielding range setting unit 134 sets the light transmittance lower as the degree of confusion increases, and sets the light transmittance higher as the degree of confusion decreases.

[0053] According to the second embodiment of the display device 100 described above, the scores of multiple facial expressions of the user are taken into consideration when evaluating the degree of user discomfort. By controlling the display to make objects or events that the user finds unpleasant less visible, taking into account the degree of user discomfort, it is possible to more appropriately reduce mental stress and trauma. Therefore, it is possible to provide a display device 100 that can reduce the psychological stress of the user caused by what they see.

[0054] (Regarding display methods) (Second embodiment) Next, a second embodiment of the display method relating to this disclosure will be described with reference to Figure 9. Figure 9 is a flowchart showing the flow of the second embodiment of the display method relating to this disclosure. The second embodiment of the display method relating to this disclosure will be described step by step below, following the flow shown in Figure 9.

[0055] First, the acquisition unit 131 acquires the user's image information (step S301). Next, the facial expression determination unit 133 determines whether the user's facial expression is the controlled facial expression (step S302). If the user's facial expression is the controlled facial expression (step S302: Yes), the gaze detection unit 132 acquires the user's gaze direction based on the user's image information (step S303). Next, the occlusion range setting unit 134 sets the occlusion area and transmittance based on the user's gaze direction (step S304). Next, the confusion level estimation unit 136 estimates the user's level of confusion (step S305). Next, the occlusion time setting unit 137 sets the occlusion time based on the level of confusion (step S306). Next, the occlusion range setting unit 134 adjusts the occlusion area and transmittance based on the level of confusion (step S307). Next, the transmittance adjustment unit 150 controls the transmittance of the lens in the occlusion area (step S308). Next, the shielding time setting unit 137 determines whether or not the shielding time has elapsed (step S309).

[0056] In step S309, if the shielding time has elapsed (step S309: Yes), the shielding range setting unit 134 restores the shielding area and transmittance to their original values ​​(step S310).

[0057] If the shielding time has not elapsed in step S309 (step S309: No), the display device 100 returns to step S301 and executes the subsequent processes.

[0058] According to the second embodiment of the display method described above, the scores of multiple facial expressions of the user are taken into consideration when evaluating the degree of user discomfort. By controlling the visibility of objects or events that the user finds unpleasant, taking into account the degree of user discomfort, it is possible to more appropriately reduce psychological stress and trauma. Therefore, a second embodiment of the display method can be provided that can reduce the psychological stress and other issues caused by the user's perception.

[0059] (Display device configuration) (Third embodiment) Next, the configuration of the third embodiment of the display device 100 according to this disclosure will be described with reference to Figure 10. Figure 10 is a diagram showing an example of the configuration of the third embodiment of the display device according to this disclosure. As shown in Figure 10, the third embodiment of the display device 100 according to this disclosure comprises a communication unit 110, a storage unit 120, a control unit 130, a display unit 140, a transmittance adjustment unit 150, and an imaging unit 160. Of these, the communication unit 110, the storage unit 120, the control unit 130, the display unit 140, the transmittance adjustment unit 150, and the imaging unit 160 may be the same as those of the first embodiment of the display device 100, so their explanation will be omitted. The third embodiment of the display device 100 differs from the first embodiment of the display device 100 in that the control unit 130 includes a walking speed measurement unit 138 and an acceleration sensor unit 170. Therefore, the walking speed measurement unit 138, the shielding range setting unit 134, and the acceleration sensor unit 170 of the control unit 130 will be described below.

[0060] The walking speed measurement unit 138 measures the user's walking speed. The walking speed measurement unit 138 may calculate the user's walking speed based on the measured values ​​of acceleration in the X, Y, and Z axes measured by the acceleration sensor unit 170. Specifically, the walking speed may be calculated by determining the correlation between the sum of the squares of the accelerations in the X, Y, and Z axes and the user's walking speed based on the data, and then multiplying the aforementioned sum of the squares of the accelerations by the correlation coefficient.

[0061] The shielding range setting unit 134 adjusts and sets the shielding range and light transmittance of the transmittance adjustment unit 150 according to the walking speed. For example, the shielding range setting unit 134 may set the shielding range to be narrower when the walking speed is fast, and to be wider when the walking speed is slow. Also, the shielding range setting unit 134 may set the transmittance to be high when the walking speed is fast, and to be low when the walking speed is slow.

[0062] The acceleration sensor unit 170 measures the user's three-axis acceleration. The acceleration sensor unit 170 may be a capacitive acceleration sensor that measures acceleration using the relationship between the change in capacitance due to the movement of the movable electrode and the resulting acceleration, for example, by creating a movable electrode and a fixed electrode using MEMS (Micro Electro Mechanical Systems). The acceleration sensor unit 170 may also be installed in a wearable device other than smart glasses, such as the user's smartphone, smartwatch, or smart shoes, and the acceleration measured by these wearable devices may be received via the communication unit 110.

[0063] According to the third embodiment of the display device 100 described above, the walking speed of the user wearing the display device 100 is taken into consideration when evaluating the degree of user discomfort. By controlling the display to make uncomfortable objects less visible, taking into consideration the walking speed of the user wearing the display device 100, it is possible to reduce mental stress and trauma more safely. Therefore, a third embodiment of the display device 100 can be provided that can reduce the psychological stress of the user caused by visibility.

[0064] (Regarding display methods) (Third embodiment) Next, a third embodiment of the display method relating to this disclosure will be described using Figure 11. Figure 11 is a flowchart showing the flow of the third embodiment of the display method relating to this disclosure. The third embodiment of the display method relating to this disclosure will be described step by step below, following the flow shown in Figure 11.

[0065] First, the acquisition unit 131 acquires the user's image information (step S401). Next, the facial expression determination unit 133 determines whether the user's facial expression is the controlled facial expression (step S402). If the user's facial expression is the controlled facial expression (step S402: Yes), the gaze detection unit 132 acquires the user's gaze direction based on the user's image information (step S403). Next, the occlusion range setting unit 134 sets the occlusion area and transmittance based on the user's gaze direction (step S404). Next, the walking speed measurement unit 138 measures the walking speed (step S405). Next, the occlusion range setting unit 134 sets the occlusion area and transmittance based on the walking speed (step S406). Next, the transmittance adjustment unit 150 controls the transmittance of the lens in the occlusion area (step S407).

[0066] In step S402, if the user's facial expression is not the facial expression of the controlled object (step S402: No), the shielding range setting unit 134 returns the shielding area and transmittance to their original values ​​(step S408). Then, the display device 100 terminates the processing of the third embodiment of the display method.

[0067] According to the third embodiment of the display method described above, the walking speed of the user wearing the display device 100 is taken into consideration when evaluating the degree of user discomfort. By controlling the visibility of objects that cause discomfort, taking into consideration the walking speed of the user wearing the display device 100, it is possible to reduce mental stress and trauma more safely. Therefore, a third embodiment of the display method can be provided that can reduce the psychological stress on the user caused by visual perception.

[0068] (Display device configuration) (Fourth embodiment) Next, the configuration of the fourth embodiment of the display device 100 according to this disclosure will be described with reference to Figure 12. Figure 12 is a diagram showing an example of the configuration of the fourth embodiment of the display device according to this disclosure. As shown in Figure 12, the fourth embodiment of the display device 100 according to this disclosure comprises a communication unit 110, a storage unit 120, a control unit 130, a display unit 140, a transmittance adjustment unit 150, and an imaging unit 160. Of these, the communication unit 110, the storage unit 120, the control unit 130, the display unit 140, the transmittance adjustment unit 150, and the imaging unit 160 may be the same as those of the first embodiment of the display device 100, so their description will be omitted. The fourth embodiment of the display device 100 differs from the first embodiment of the display device 100 in that the control unit 130 comprises an information display control unit 139, so the information display control unit 139 of the control unit 130 will be described below.

[0069] The information display control unit 139 displays various types of information in the information display area of ​​the display unit 140. For example, it may display weather forecasts, advertisements, videos, browser information, etc. The information display control unit 139 obtains this information from an external server device or the like via the communication unit 110. The information display area is usually set to a predetermined position. The information display control unit 139 may set the size of the shielding area and the information display area to be the same. If they are not the same, the information display control unit 139 moves them so that their center points coincide. In addition, if a shielding area is set, the information display control unit 139 moves the information display area into the shielding area.

[0070] The process when no shielding area is set in the information display control unit 139 will be specifically explained using Figure 14. Figure 14 is a diagram showing an example of information display when there is no shielding area according to this disclosure. As shown in Figure 14, when no shielding area is set, the information display control unit 139 sets the information display area so that the information display area A is in the user's line of sight direction Pb and displays the information.

[0071] The processing when a shielding area is set in the information display control unit 139 will be specifically explained using Figure 15. Figure 15 is a diagram showing an example of information display when a shielding area is set according to the present disclosure. As shown in Figure 15, when a shielding area is set in the information display control unit 139, it sets the information display area so that it overlaps with the shielding area and displays the information. That is, if a shielding area is set in the information display control unit 139 while it is displaying information when no shielding area is set, it moves the information display area to the shielding area.

[0072] This allows the displayed information to be moved to an obscured area based on the user's facial expression and in response to the user's discomfort, thereby preventing the user from seeing the information they find unpleasant. Therefore, a fourth embodiment of the display device 100 can be provided that can reduce the psychological stress on the user caused by viewing the information.

[0073] (Regarding display methods) (Fourth embodiment) Next, a fourth embodiment of the display method relating to this disclosure will be described with reference to Figure 13. Figure 13 is a flowchart showing the flow of the fourth embodiment of the display method relating to this disclosure. The fourth embodiment of the display method relating to this disclosure will be described step by step below, following the flow shown in Figure 13.

[0074] First, the information display control unit 139 displays information in the information display area of ​​the display unit 140 (step S501). Next, the acquisition unit 131 acquires the user's image information (step S502). Next, the facial expression determination unit 133 determines whether the user's facial expression is the controlled facial expression (step S503). If the user's facial expression is the controlled facial expression (step S503: Yes), the gaze detection unit 132 acquires the user's gaze direction based on the user's image information (step S504). Next, the occlusion range setting unit 134 sets the occlusion area and transmittance based on the user's gaze direction (step S505). Next, the transmittance adjustment unit 150 controls the transmittance of the lens in the occlusion area (step S506). Next, the information display control unit 139 moves the information display area inside the occlusion area (step S507). Next, the display device 100 returns to step S503 and executes the subsequent processing.

[0075] In step S503, if the user's facial expression is not the facial expression of the controlled object (step S503: No), the shielding range setting unit 134 returns the shielding area and transmittance to their original values ​​(step S508). Then, the display device 100 terminates the processing of the fourth embodiment of the display method.

[0076] Therefore, based on the user's facial expression, the displayed information can be moved to an obscured area according to the user's level of discomfort, making it impossible for the user to see the information they find unpleasant. Thus, a fourth embodiment of the display method can be provided that can reduce the psychological stress on the user caused by viewing the information.

[0077] (Structure and effect) The display device 100 according to this disclosure includes a transmittance adjustment unit 150 that adjusts the light transmittance, which indicates the proportion of light that is transmitted; a gaze detection unit 132 that detects the user's gaze information based on first image information showing the user's eyeballs; an expression determination unit 133 that determines the user's facial expression based on second image information showing the user's face; and an occlusion range setting unit 134 that, if the facial expression determined by the expression determination unit 133 is the facial expression to be controlled, sets an occlusion range that sets the light transmittance to a predetermined value based on the gaze information and the emotion score.

[0078] With this configuration, the display device 100 can set an obscuring area when the user's facial expression is one of the controlled expressions such as fear or disgust, so that the object that caused that emotion is not displayed in the user's field of vision. Therefore, it is possible to provide a display device 100 that can reduce the psychological stress on the user caused by viewing.

[0079] The display device 100 according to this disclosure further comprises an acceleration sensor unit 170 that measures acceleration generated by the user's movement, and a walking speed measuring unit 138 that measures the user's walking speed based on the acceleration measured by the acceleration sensor unit 170, and the shielding range setting unit 134 sets the shielding range based on the calculated user's walking speed.

[0080] This configuration allows for a safer reduction of psychological stress and trauma by controlling the display device 100 to make unpleasant objects less visible, taking into account the walking speed of the user wearing it. Therefore, it is possible to provide a display device 100 that can reduce the psychological stress and other issues caused by visual perception.

[0081] The facial expression determination unit 133 of the display device 100 according to this disclosure calculates a fear score representing the degree of the user's fear and a disgust score representing the degree of the user's aversion based on the second image information, and the occlusion range setting unit 134 sets the occlusion range of the transmittance adjustment unit 150 based on the fear score and the disgust score.

[0082] This configuration allows the occlusion range to be set based on the user's fear score and aversion score. Therefore, the occlusion range can be appropriately set according to the user's emotions. Thus, it is possible to provide a display device 100 that can reduce the user's psychological stress caused by viewing.

[0083] The display method relating to this disclosure is a display method using a display device 100 that includes a transmittance adjustment unit 150 for adjusting the light transmittance which indicates the proportion of light transmitted, and includes the steps of: detecting the user's gaze information based on first image information including the user's eyeballs; calculating an emotion score including a plurality of parameters representing the user's emotions based on second image information including the user's facial expressions; and setting the light transmittance and the shielding range of the transmittance adjustment unit 150 based on the emotion score.

[0084] This configuration allows for the creation of an obscuring area when the user's facial expression is one of the controlled emotions, such as fear or disgust, preventing the object that caused that emotion from appearing in the user's field of vision. Therefore, it is possible to provide a display method that can reduce the psychological stress on the user caused by viewing the object.

[0085] The program relating to this disclosure is a program that causes a computer of a display device 100, which includes a transmittance adjustment unit 150 that adjusts the light transmittance indicating the proportion of light transmitted, to execute processing, and includes the steps of: detecting the user's gaze information based on first image information including the user's eyeballs; calculating an emotion score including a plurality of parameters representing the user's emotions based on second image information including the user's facial expressions; and setting the light transmittance and shielding range of the transmittance adjustment unit 150 based on the emotion score.

[0086] With this configuration, if the user's facial expression is one of the controlled emotions, such as fear or disgust, an occlusion area can be set to prevent the object that caused that emotion from appearing in the user's field of vision. Therefore, it is possible to provide a program that can reduce the psychological stress on the user caused by visual perception.

[0087] Although embodiments of the present invention have been described above, the embodiments are not limited to those described herein. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the embodiments described above. [Explanation of symbols]

[0088] 100 display device 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 Eye-line detection unit 133 Facial expression determination section 134 Shielding range setting section 135 Control signal transmission unit 136 Perplexity Estimation Department 137 Shielding time setting section 138 Walking speed measurement unit 139 Information Display Control Unit 140 Display section 150 Transmittance adjustment section 160 Imaging Unit 160A First Imaging Unit 160B Second imaging unit 170 Acceleration sensor section A Information display area H Height Pa,Pb View direction W width

Claims

1. A transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light that is transmitted, A gaze detection unit detects the user's gaze information based on the first image information showing the user's eyeballs, A facial expression determination unit determines the user's facial expression based on second image information showing the user's face, The system includes a shielding range setting unit that, when the facial expression determined by the facial expression determination unit is the facial expression to be controlled, sets a shielding range in which the light transmittance is set to a predetermined value based on the gaze information and emotion score. Display device.

2. An acceleration sensor unit that measures acceleration generated by the user's movements, The system further includes a walking speed calculation unit that calculates the user's walking speed based on the acceleration measurement value from the acceleration sensor unit, The shielding range setting unit sets the shielding range based on the calculated walking speed of the user. The display device according to claim 1.

3. The facial expression determination unit calculates a fear score representing the degree of the user's fear and a disgust score representing the degree of the user's disgust based on the second image information. The shielding range setting unit sets the shielding range of the transmittance adjustment unit based on the fear score and the aversion score. The display device according to claim 1 or claim 2.

4. A display method using a display device comprising a transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light transmitted, A step of detecting the user's gaze information based on the first image information showing the user's eyeballs, A step of determining the user's facial expression based on second image information showing the user's face, The process includes the step of setting the light transmittance and shielding range of the transmittance adjustment unit based on the aforementioned gaze information and emotion score. Display method.

5. A program that causes a computer of a display device, which includes a transmittance adjustment unit that adjusts the light transmittance, which indicates the proportion of light transmitted, to perform processing, A step of detecting the user's gaze information based on the first image information showing the user's eyeballs, A step of determining the user's facial expression based on second image information showing the user's face, The process includes the step of setting the light transmittance and shielding range of the transmittance adjustment unit based on the aforementioned gaze information and emotion score. program.

Images