Display controller and display control method

Abstract

To make it easier to recognize its presence, even when an object approaches, as compared with prior art.SOLUTION: A display controller comprises: a UI element information acquisition unit 201 which acquires UI element information indicating a UI element which is a display target to real space; an object information acquisition unit 202 which acquires object information indicating presence / absence of an object which approaches a region on which the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is displayed; and a transparency correction unit 203 which corrects transparency, on the basis of the object information acquired by the object information acquisition unit 202, with respect to the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a display control device and a display control method for holographically projecting a user interface element into real space. [Background technology]

[0002] Generally, in MR (Mixed Reality) technology, a user looks at user interface elements (hereinafter referred to as UI elements) such as windows or buttons projected as holograms into real space through glasses and performs operations such as moving or enlarging windows or pressing buttons (see, for example, Patent Document 1). Basic operation methods include hand gestures, hand ray, voice input, gaze input, etc. A user performs some operation on a UI element by using any of the above operation methods. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-57659 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional devices, UI elements are projected into the real world as holograms, so it may be difficult to recognize the presence of an object (a person or other moving object) even if it approaches. In addition, because conventional devices project UI elements into real space as holograms, the visibility of the UI elements can be poor depending on the brightness of the real space, which can lead to reduced work efficiency. In other words, when conventional devices are used in bright places such as outdoors, the hologram-projected UI elements appear faint and are difficult to recognize. Furthermore, in conventional devices, the user basically needs to operate all UI elements by himself, which places a heavy burden on the user when there are many UI elements to display.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a display control device that makes it easier to recognize the presence of an object even if the object is approaching, compared to conventional devices. [Means for solving the problem]

[0006] The display control device according to the present disclosure is characterized by comprising a user interface element information acquisition unit that acquires user interface element information indicating user interface elements to be displayed in real space, an object information acquisition unit that acquires object information indicating the presence or absence of an object approaching an area in which the user interface elements indicated by the user interface element information acquired by the user interface element information acquisition unit are displayed, and a transparency correction unit that corrects the transparency of the user interface elements indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the object information acquired by the object information acquisition unit. Effect of the Invention

[0007] According to the present disclosure, with the above-mentioned configuration, it becomes easier to recognize the presence of an object even if the object approaches, as compared to the conventional technology. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram showing a configuration example of a display system according to a first embodiment. [Diagram 2] 1 is a diagram illustrating an example of a configuration of a display control device according to a first embodiment. [Diagram 3] 4 is a flowchart showing an operation example of the display control device according to the first embodiment. [Figure 4]Figures 4A and 4B are diagrams showing an example of the operation of a display control device in embodiment 1, where Figure 4A is a diagram showing an example of the state before transparency correction by the display control device, and Figure 4B is a diagram showing an example of the state after transparency correction by the display control device. [Diagram 5] FIG. 11 is a diagram illustrating an example of the configuration of a display control device according to a second embodiment. [Figure 6] 11 is a flowchart showing an operation example of the display control device according to the second embodiment. [Figure 7] Figures 7A and 7B are diagrams showing an example of the operation of a display control device according to embodiment 2, where Figure 7A is a diagram showing an example of the state before correction of at least one of hue and saturation and brightness by the display control device, and Figure 7B is a diagram showing an example of the state after correction of at least one of hue and saturation and brightness by the display control device. [Figure 8] FIG. 11 is a diagram illustrating an example of the configuration of a display control device according to a third embodiment. [Figure 9] 13 is a flowchart showing an operation example of the display control device according to the third embodiment. [Figure 10] Figures 10A and 10B are diagrams showing an example of the operation of a display control device relating to embodiment 3, where Figure 10A is a diagram showing an example of the state before an enlarged screen is created by the display control device, and Figure 10B is a diagram showing an example of the state after an enlarged screen is created by the display control device. [Figure 11] 11A and 11B are diagrams illustrating an example of a hardware configuration of a display control device according to embodiments 1-3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of a display system according to the first embodiment. The display system is a system that holographically projects user interface elements (hereinafter referred to as UI elements) into real space. Examples of UI elements include windows and buttons. This display system includes a head-mounted display device 1 and a display control device 2.

[0010] 1 illustrates a case where the display control device 2 is provided outside the device 1. However, the present invention is not limited to this, and the display control device 2 may be provided inside the device 1.

[0011] The device 1 is a device worn by a user on the head. The device 1 has a distance information acquisition unit and a display unit. The distance information acquisition unit acquires distance information. This distance information is information indicating the distance from the device 1 to an object existing in an external area including an area in which UI elements are displayed. An example of the distance information acquisition unit is a depth sensor or a sensor having a space scanning function. The display unit displays the UI elements obtained from the display control device 2 by holographic projection. As this device 1, an existing device can be used.

[0012] The display control device 2 controls UI elements that are to be displayed in real space. As shown in FIG. 2, the display control device 2 includes a UI element information acquisition unit 201, an object information acquisition unit 202, and a transparency correction unit 203.

[0013] The UI element information acquisition unit 201 acquires UI element information. The UI element information is information that indicates UI elements to be displayed in real space.

[0014] The object information acquisition unit 202 acquires object information. The object information is information indicating the presence or absence of an object approaching an area displaying a UI element indicated by the UI element information acquisition unit 201, that is, information indicating the presence or absence of an object whose distance from the area is shorter than a threshold and which is approaching the area. Examples of the object include a person or other moving object. The threshold can be set appropriately. At this time, the object information acquisition unit 202 may acquire the object information by detecting the presence or absence of an object approaching the area displaying the UI element based on the distance information acquired by the distance information acquisition unit. Also, for example, the object information acquisition unit 202 may acquire the object information by cooperating with a positioning system that monitors the movement of objects and detecting the presence or absence of an object approaching the area displaying the UI element based on information indicating the movement of the object acquired by the positioning system. Furthermore, the object information acquisition unit 202 may acquire the object information itself input from outside.

[0015] In addition, the object information acquired by the object information acquisition unit 202 may include at least one of information indicating the distance from an area where an object approaching the area displays a UI element indicated by the UI element information acquisition unit 201, and information indicating the movement speed of the object.

[0016] The transparency correction unit 203 corrects the transparency of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 based on the object information acquired by the object information acquisition unit 202 . At this time, for example, when the object information acquired by object information acquisition unit 202 indicates that there is an object approaching the area where the UI element is displayed, transparency correction unit 203 corrects the transparency of the UI element to be higher than normal. Furthermore, thereafter, when the object information acquired by object information acquisition unit 202 indicates that there is no longer an object approaching the area where the UI element is displayed, transparency correction unit 203 returns the transparency of the UI element to normal.

[0017] In addition, if the object information includes at least one of information indicating the distance from the object area and information indicating the moving speed, the transparency correction unit 203 may correct the transparency of the UI element taking these parameters into consideration. At this time, for example, the transparency correction unit 203 corrects the transparency of the UI element so as to increase continuously or in stages as the distance from the object region decreases or the moving speed of the object increases.

[0018] In addition, if the display control device 2 is capable of linking with a positioning system capable of monitoring the movement of objects within a specified area, the transparency correction unit 203 may also correct the transparency of UI elements taking into account movement information from this positioning system.

[0019] Then, when the transparency of the UI element is corrected by the transparency correcting unit 203, the device 1 displays the UI element after the correction.

[0020] Next, an example of the operation of the display control device 2 according to the first embodiment shown in FIGS. 1 and 2 will be described with reference to FIG. The UI element information acquisition unit 201 acquires UI element information indicating UI elements to be displayed in real space.

[0021] In an operation example of the display control device 2 according to the first embodiment shown in FIGS. 1 and 2, first, as shown in FIG. 3, the object information acquisition unit 202 acquires object information indicating the presence or absence of an object approaching an area displaying a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 (step ST101). At this time, the object information acquisition unit 202 may acquire the object information by detecting the presence or absence of an object approaching an area displaying a UI element based on the distance information acquired by the distance information acquisition unit. Also, for example, the object information acquisition unit 202 may cooperate with a positioning system that monitors the movement of an object, and acquire the object information by detecting the presence or absence of an object approaching an area displaying a UI element based on information indicating the movement of an object obtained by the positioning system. Also, the object information acquisition unit 202 may acquire the object information itself input from the outside.

[0022] In addition, the object information acquired by the object information acquisition unit 202 may include at least one of information indicating the distance from an area where an object approaching the area displays a UI element indicated by the UI element information acquisition unit 201, and information indicating the movement speed of the object.

[0023] Next, transparency correction unit 203 determines whether or not a correction of transparency is required for the UI element indicated by the UI element information acquired by UI element information acquisition unit 201, based on the object information acquired by object information acquisition unit 202 (step ST102). That is, transparency correction unit 203 determines whether or not the object information indicates that there is an object approaching the area displaying the UI element, thereby determining whether or not a correction of transparency is required for the UI element.

[0024] At this time, if the transparency correction unit 203 determines in step ST102 that the transparency correction for the UI element is not necessary, the sequence ends.

[0025] On the other hand, in step ST102, when it is determined that the transparency correction unit 203 needs to correct the transparency of the UI element, the transparency correction unit 203 corrects the transparency of the UI element (step ST103). At this time, as shown in FIG. 4, for example, the transparency correction unit 203 corrects the transparency of the UI element to be higher than in the normal case. Note that FIG. 4A is a diagram showing an example of the normal case, that is, the state before the transparency correction by the transparency correction unit 203. In FIG. 4A, the reference numeral 41a indicates the UI element before the correction. Also, the diagram of FIG. 4B is a diagram showing an example of the state after the transparency correction by the transparency correction unit 203. In FIG. 4B, the reference numeral 41b indicates the UI element after the correction. FIG. 4B shows a case where a person approaches the area in which the UI element is displayed.

[0026] In addition, if the object information includes at least one of information indicating the distance from the object area and information indicating the moving speed, the transparency correction unit 203 may correct the transparency of the UI element taking these parameters into consideration. At this time, for example, the transparency correction unit 203 corrects the transparency of the UI element so as to increase continuously or in stages as the distance from the object region decreases or the moving speed of the object increases.

[0027] In addition, if the display control device 2 is capable of linking with a positioning system capable of monitoring the movement of objects within a specified area, the transparency correction unit 203 may also correct the transparency of UI elements taking into account movement information from this positioning system.

[0028] Thereafter, if the object information acquired by the object information acquisition unit 202 indicates that there is no longer an object approaching the area in which the UI element is displayed, the transparency correction unit 203 returns the transparency of the UI element to normal.

[0029] In this way, the display control device 2 according to the first embodiment adjusts the transparency of a UI element based on the presence or absence of an object approaching from behind the UI element displayed in the user's field of view. This makes it easier for the user to recognize an object approaching from behind a UI element projected with a hologram, even if such an object is present. As a result, the user can more easily avoid a collision with the object, which can increase safety.

[0030] As described above, according to the first embodiment, the display control device 2 includes a UI element information acquisition unit 201 that acquires UI element information indicating UI elements to be displayed in real space, an object information acquisition unit 202 that acquires object information indicating the presence or absence of an object approaching an area in which the UI element indicated by the UI element information acquisition unit 201 is displayed, and a transparency correction unit 203 that corrects the transparency of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201, based on the object information acquired by the object information acquisition unit 202. As a result, the display control device 2 according to the first embodiment makes it easier to recognize the presence of an object even if it is approaching, compared to conventional devices.

[0031] Embodiment 2 The display control device 2 according to the first embodiment has been described as correcting the transparency of a UI element depending on the presence or absence of an object approaching the area in which the UI element is displayed. In contrast, the display control device 2 according to the second embodiment has been described as correcting at least one of the hue and saturation, and the brightness of a UI element depending on at least one of the hue and saturation, and the brightness of a background in the area in which the UI element is displayed.

[0032] Fig. 5 is a diagram showing an example of the configuration of a display control device 2 according to embodiment 2. In the display control device 2 according to embodiment 2 shown in Fig. 5, a first background information acquisition unit 204, a second background information acquisition unit 205, a first color correction unit 206, and a second color correction unit 207 are added to the display control device 2 according to embodiment 1 shown in Fig. 2. The other example of the configuration of the display control device 2 according to embodiment 2 shown in Fig. 5 is similar to the example of the configuration of the display control device 2 according to embodiment 1 shown in Fig. 2, and the same reference numerals are used and only the different parts will be described.

[0033] The device 1 includes an external image acquisition unit that acquires an external image. The external image acquisition unit acquires an external image. The external image is an image obtained by capturing an image of an external area including an area in which UI elements are displayed.

[0034] The first background information acquisition unit 204 acquires the first background information. The first background information is information indicating at least one of the hue and saturation of the background in the area displaying the UI element indicated by the UI element information acquisition unit 201. Note that the range of the background referred to here may be, for example, the entire visual field area that is the area visible to the user, or may be only a part of the visual field area, for example, an area overlapping with the area displaying the UI element. At this time, the first background information acquisition unit 204 may acquire the first background information by detecting at least one of the hue and saturation of the background in the area in which the UI element is displayed based on the external image acquired by the external image acquisition unit. Furthermore, the first background information acquisition unit 204 may acquire the first background information itself input from outside. For example, when a sensor capable of acquiring the first background information is provided inside or outside the device 1, the first background information acquisition unit 204 may acquire the first background information from this sensor. An example of this sensor is an optical sensor (color sensor).

[0035] The second background information acquisition unit 205 acquires second background information. The second background information is information indicating the brightness of the background in an area displaying a UI element indicated by the UI element information acquisition unit 201. Note that the range of the background referred to here may be, for example, the entire visual field area that is an area visible to the user, or may be only a part of the visual field area, for example, an area overlapping with an area displaying a UI element. At this time, the second background information acquisition unit 205 may acquire the second background information by detecting the brightness of the background in the area where the UI element is displayed, based on the external image acquired by the external image acquisition unit. Furthermore, second background information acquiring section 205 may acquire the second background information itself input from outside.

[0036] The first color correction unit 206 corrects at least one of the hue and saturation of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201, based on the first background information acquired by the first background information acquisition unit 204. At this time, for example, first color correction unit 206 compares at least one of the hue of the background and the hue of the UI element, and the saturation value of the background or a representative value of saturation such as an average value thereof and the saturation value of the UI element or a representative value of saturation such as an average value thereof. Then, first color correction unit 206 corrects at least one of the hue and saturation of the UI element according to at least one of the difference in the hue and the difference in the representative value of saturation. At this time, for example, first color correction unit 206 may correct the saturation of the UI element to be higher when the representative value of the saturation of the background is higher than the representative value of the saturation of the UI element, and may correct the saturation of the UI element to be lower when the representative value of the saturation of the background is lower than the representative value of the saturation of the UI element. Also, for example, first color correction unit 206 may increase contrast by correcting the hue of the UI element to be a complementary color of the hue of the background when the hue of the background and the hue of the UI element are similar colors.

[0037] The second color correction unit 207 corrects the brightness of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201, based on the second background information acquired by the second background information acquisition unit 205. At this time, for example, when the brightness of the background is higher than a threshold, the second color correction unit 207 corrects the brightness of the UI element to be lower than in a normal case. Note that the threshold can be set appropriately.

[0038] Then, when at least one of the hue and saturation of a UI element is corrected by the first color correction unit 206, or when the brightness of a UI element is corrected by the second color correction unit 207, the device 1 displays the corrected UI element.

[0039] Next, an example of the operation of the display control device 2 according to the second embodiment shown in Fig. 5 will be described with reference to Fig. 6. In Fig. 6, only the operation related to the first color correction based on at least one of the hue and saturation of the background and the second color correction based on the lightness of the background by the display control device 2 according to the second embodiment will be described.

[0040] In an operation example of the display control device 2 according to the second embodiment shown in Fig. 5, first, as shown in Fig. 6, the first background information acquisition unit 204 acquires first background information indicating at least one of the hue and saturation of the background in an area in which the UI elements indicated by the UI element information acquired by the UI element information acquisition unit 201 are displayed (step ST201). Note that the range of the background here may be, for example, the entire visual field area which is an area visible to the user, or may be only a part of the visual field area, for example, an area overlapping with an area in which the UI elements are displayed. At this time, the first background information acquisition unit 204 may acquire the first background information by detecting at least one of the hue and saturation of the background in the area displaying the UI element based on the external image acquired by the external image acquisition unit. The first background information acquisition unit 204 may also acquire the first background information itself input from outside.

[0041] Further, the second background information acquisition unit 205 acquires second background information indicating the brightness of the background in the area displaying the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 (step ST202). Note that the range of the background referred to here may be, for example, the entire visual field area which is the area visible to the user, or may be only a part of the visual field area, for example, an area overlapping with the area displaying the UI element. In this case, the second background information acquisition unit 205 may acquire the second background information by detecting the brightness of the background in the area where the UI element is displayed based on the external image acquired by the external image acquisition unit. The second background information acquisition unit 205 may also acquire the second background information itself input from outside.

[0042] Next, the first color correction unit 206 determines whether or not at least one of the hue and saturation needs to be corrected for the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201, based on the first background information acquired by the first background information acquisition unit 204 (step ST203). That is, for example, the first color correction unit 206 compares at least one of the hue of the background indicated by the first background information and the hue of the UI element, and the saturation value of the background indicated by the first background information or a representative value of saturation such as an average value thereof and a representative value of saturation such as an average value thereof of the UI element. Then, the first color correction unit 206 determines whether or not at least one of the difference between the hues and the difference between the representative values ​​of saturation is equal to or greater than a threshold value, thereby determining whether or not at least one of the hue and saturation needs to be corrected for the UI element.

[0043] In step ST203, if the first color correction unit 206 determines that correction of at least one of the hue and saturation for the UI element is not necessary, the sequence proceeds to step ST205.

[0044] On the other hand, in step ST203, when it is determined that at least one of the hue and saturation of the UI element needs to be corrected, the first color correction unit 206 corrects at least one of the hue and saturation of the UI element (step ST204). At this time, for example, the first color correction unit 206 corrects at least one of the hue and saturation of the UI element according to at least one of the difference in the hue and the difference in the representative value of the saturation. At this time, for example, the first color correction unit 206 may correct the saturation of the UI element to be higher when the representative value of the saturation of the background is higher than the representative value of the saturation of the UI element, and may correct the saturation of the UI element to be lower when the representative value of the saturation of the background is lower than the representative value of the saturation of the UI element. Also, for example, when the hue of the background and the hue of the UI element are similar colors, the first color correction unit 206 may correct the hue of the UI element to be a complementary color of the hue of the background to increase the contrast.

[0045] Further, second color correction unit 207 determines whether or not brightness correction is required for the UI element indicated by the UI element information acquired by UI element information acquisition unit 201, based on the second background information acquired by second background information acquisition unit 205 (step ST205). That is, for example, second color correction unit 207 determines whether or not brightness correction is required for the UI element by determining whether the brightness value of the background indicated by the second background information is a threshold value.

[0046] In step ST205, if the second color correction unit 207 determines that the brightness correction for the UI element is not necessary, the sequence ends.

[0047] On the other hand, in step ST205, when second color correction unit 207 determines that the brightness of the UI element needs to be corrected, it corrects the brightness of the UI element (step ST206). At this time, second color correction unit 207 corrects the brightness of the UI element according to the above value. For example, when the brightness of the background is high, second color correction unit 207 corrects the brightness of the UI element to be lower.

[0048] 7 is a diagram for explaining correction of at least one of the hue and saturation and the brightness of a UI element based on at least one of the hue and saturation and the brightness of the background by the display control device 2 according to the second embodiment. FIG. 7A is a diagram showing an example of a state before correction by the first color correction unit 206 and the second color correction unit 207. In FIG. 7A, reference numeral 71a indicates a UI element before correction. Also, FIG. 7B is a diagram showing an example of a state after correction by the first color correction unit 206 and the second color correction unit 207. In FIG. 7B, reference numeral 71b indicates a UI element after correction.

[0049] In this way, the display control device 2 according to the second embodiment adjusts at least one of the hue and saturation, and the brightness of a UI element based on at least one of the hue and saturation, and the brightness of the background of the UI element displayed in the user's field of view. This makes it easier to view a UI element projected as a hologram, regardless of at least one of the hue and saturation, and the brightness of the background of the UI element.

[0050] In addition, in the display control device 2 of embodiment 2, at least one of the hue and saturation of the background is used in the area overlapping with the area in which the UI element is displayed, rather than at least one of the hue and saturation of the background in the entire viewing area, thereby enabling a first color correction according to the location of the area in which the UI element is displayed. Similarly, in the display control device 2 of embodiment 2, the background brightness is determined not by the background brightness of the entire viewing area, but by the background brightness of the area overlapping with the area in which the UI element is displayed, thereby enabling a second color correction according to the location of the area in which the UI element is displayed.

[0051] As described above, according to the second embodiment, the display control device 2 includes a first background information acquisition unit 204 that acquires first background information indicating at least one of the hue and saturation of the background in an area where a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is displayed, a second background information acquisition unit 205 that acquires second background information indicating the brightness of the background in an area where a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is displayed, a first color correction unit 206 that corrects at least one of the hue and saturation of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 based on the first background information acquired by the first background information acquisition unit 204, and a second color correction unit 207 that corrects the brightness of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 based on the second background information acquired by the second background information acquisition unit 205. This makes it possible for the display control device 2 according to the second embodiment to improve the visibility of UI elements compared to the conventional case. That is, the display control device 2 according to the second embodiment can improve the visibility of UI elements regardless of the brightness of the real space, and can suppress a decrease in work efficiency, compared to the conventional case. For example, even if the display control device 2 according to the second embodiment is used in a bright place such as outdoors, it can suppress a situation where the holographically projected UI elements appear faint and are difficult to recognize.

[0052] In the above, a case has been described in which the display control device 2 according to the second embodiment corrects at least one of the hue and saturation and the brightness of a UI element based on at least one of the hue and saturation and the brightness of the background. However, the present invention is not limited to this, and the display control device 2 may correct at least one of the hue and saturation or the brightness of a UI element based on at least one of the hue and saturation or the brightness of the background, and the same effect as above can be obtained.

[0053] For example, when the display control device 2 corrects only at least one of the hue and saturation of a UI element based on at least one of the hue and saturation of a background, the second background information acquisition unit 205 and the second color correction unit 207 shown in Fig. 5 are not necessary. In this case, the display control device 2 further includes a first background information acquisition unit 204 that acquires first background information indicating at least one of the hue and saturation of a background in an area in which a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is displayed, and a first color correction unit 206 that corrects at least one of the hue and saturation of a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 based on the first background information acquired by the first background information acquisition unit 204, in comparison with the display control device 2 according to the first embodiment shown in Fig. 2.

[0054] Also, for example, when the display control device 2 corrects only the brightness of the UI element based on the brightness of the background, the first background information acquisition unit 204 and the first color correction unit 206 shown in Fig. 5 are not necessary. In this case, the display control device 2 further includes, in addition to the display control device 2 according to the first embodiment shown in Fig. 2, a second background information acquisition unit 205 that acquires second background information indicating the brightness of the background in an area in which a UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is displayed, and a second color correction unit 207 that corrects the brightness of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201, based on the second background information acquired by the second background information acquisition unit 205.

[0055] Embodiment 3 In the display control device 2 according to the second embodiment, the transparency of the UI element is corrected according to the presence or absence of an object approaching the area in which the UI element is displayed, and at least one of the hue and saturation and the brightness of the UI element is corrected according to at least one of the hue and saturation and the brightness of the background in the area in which the UI element is displayed. In contrast, in the display control device 2 according to the third embodiment, in addition to the above, a configuration is shown for creating an enlarged screen in which the area of ​​the UI element that the user is gazing at is enlarged.

[0056] Fig. 8 is a diagram showing a configuration example of a display control device 2 according to embodiment 3. In the display control device 2 according to embodiment 3 shown in Fig. 8, gaze direction information acquisition unit 208, gaze state determination unit 209, and enlarged screen creation unit 210 are added to the display control device 2 according to embodiment 2 shown in Fig. 5. The other configuration example of the display control device 2 according to embodiment 3 shown in Fig. 8 is similar to the configuration example of the display control device 2 according to embodiment 2 shown in Fig. 5, and the same reference numerals are assigned and only different parts will be described.

[0057] The device 1 includes a gaze direction detection unit that detects the gaze direction of the user by eye tracking or the like.

[0058] The gaze direction information acquisition unit 208 acquires gaze direction information. The gaze direction information is information indicating the gaze direction of the user. At this time, the gaze direction information acquisition unit 208 acquires gaze direction information which is information indicating the gaze direction detected by the gaze direction detection unit.

[0059] The gaze state determination unit 209 determines whether the user is gazing at a minute area in the area displaying the UI elements, based on the gaze direction information acquired by the gaze direction information acquisition unit 208. In this case, when the gaze state determination unit 209 determines that the user is gazing at a minute area in the area displaying the UI elements, the gaze state determination unit 209 determines that the user is gazing at the minute area. Note that the minute area is an area smaller than the area displaying the UI elements. For example, a circular area with a radius of 5 cm can be set as the minute area. Also, for example, 3 seconds can be set as the predetermined time.

[0060] Based on the gaze direction information acquired by gaze direction information acquisition unit 208 and the determination result by gaze state determination unit 209, enlarged screen creation unit 210 creates a screen in which a part of the UI elements indicated by the UI element information acquisition unit 201 is enlarged. At this time, if gaze state determination unit 209 determines that the user is gazing at a minute area, enlarged screen creation unit 210 creates a screen in which a specific area including the gaze point, which is a point on the user's gaze, is cut out and enlarged. Note that the specific area is smaller than the area in which the UI elements are displayed and larger than the minute area. For example, the specific area may be an area 400 cm around the gaze point. 2 It is possible to set a square area of ​​about 100 mm.

[0061] Then, when an enlarged screen is created by the enlarged screen creating unit 210, the device 1 displays the enlarged screen in addition to the UI elements.

[0062] Next, an example of the operation of the display control device 2 according to the third embodiment shown in Fig. 8 will be described with reference to Fig. 9. In Fig. 8, only the operation of the display control device 2 according to the third embodiment regarding the creation of an enlarged screen based on the line of sight will be described.

[0063] In the operation example of the display control device 2 according to the third embodiment shown in Fig. 8, first, the gaze direction information acquisition unit 208 acquires gaze direction information indicating the gaze direction of the user (step ST301) as shown in Fig. 9. At this time, the gaze direction information acquisition unit 208 acquires the gaze direction information indicating the gaze direction detected by the gaze direction detection unit.

[0064] Next, gaze state determination unit 209 determines whether the user is gazing at a minute area in the area displaying the UI element based on the gaze direction information acquired by gaze direction information acquisition unit 208 (step ST302). At this time, when gaze state determination unit 209 determines that the user's gaze remains on a minute area for a predetermined time in the area displaying the UI element, it determines that the user is gazing at the minute area.

[0065] In this step ST302, if the gaze state determination section 209 determines that the user is not gazing at the minute region, the sequence ends.

[0066] On the other hand, when the gaze state determination unit 209 determines that the user is gazing at a minute region, the enlarged screen creation unit 210 creates a screen in which a part of the UI element indicated by the UI element information acquired by the UI element information acquisition unit 201 is enlarged based on the gaze direction information acquired by the gaze direction information acquisition unit 208 (step ST303). At this time, as shown in FIG. 10, for example, when the gaze state determination unit 209 determines that the user is gazing at a minute region, the enlarged screen creation unit 210 creates a screen in which a specific region including a gaze point, which is the gaze of the user, is cut out and enlarged. Note that FIG. 10A is a diagram showing an example of a state before the screen is created by the enlarged screen creation unit 210, and is a diagram showing a state in which only UI elements are displayed. In FIG. 10A, the reference numeral 101 indicates a UI element, the reference numeral 102 indicates a gaze point, and the reference numeral 103 indicates a specific region to be cut out. Also, FIG. 10B is a diagram showing an example of a state after the screen is created by the enlarged screen creation unit 210, and is a diagram showing a state in which the UI element and the enlarged screen are displayed. In FIG. 10B, reference numeral 104 denotes an enlarged screen.

[0067] In this way, the display control device 2 according to the third embodiment automatically creates a screen in which a predetermined area centered on the user's gaze point is enlarged. As a result, the display control device 2 according to the third embodiment has improved operability compared to conventional devices, and can reduce the burden on the user even when there are many UI elements to be displayed.

[0068] In the above description, the gaze direction information acquisition unit 208, the gaze state determination unit 209, and the enlarged screen creation unit 210 are added to the display control device 2 according to embodiment 2. However, the present invention is not limited to this, and the gaze direction information acquisition unit 208, the gaze state determination unit 209, and the enlarged screen creation unit 210 may be added to the display control device 2 according to embodiment 1 or to a configuration of the display control device 2 according to embodiment 2 that performs only the first color correction of UI elements based on at least one of the hue and saturation of the background or a configuration of the display control device 2 according to embodiment 2 that performs only the second color correction of UI elements based on the brightness of the background.

[0069] As described above, according to the third embodiment, the display control device 2 includes the gaze direction information acquisition unit 208 that acquires gaze direction information indicating the gaze direction of the user, the gaze state determination unit 209 that determines whether the user is gazing at a small area in an area displaying UI elements, based on the gaze direction information acquired by the gaze direction information acquisition unit 208, and the enlarged screen creation unit 210 that creates a screen in which a part of the UI elements indicated by the UI element information acquisition unit 201 is enlarged, based on the gaze direction information acquired by the gaze direction information acquisition unit 208 and the determination result by the gaze state determination unit 209. This enables the display control device 2 according to the third embodiment to improve the operability and visibility of UI elements compared to conventional ones.

[0070] Finally, a hardware configuration example of the display control device 2 according to the embodiments 1 to 3 will be described with reference to Fig. 11. Hereinafter, a hardware configuration example of the display control device 2 according to the embodiment 1 will be described, but the same applies to the hardware configuration examples of the display control device 2 according to the embodiments 2 and 3. Each function of the UI element information acquisition unit 201, the object information acquisition unit 202, and the transparency correction unit 203 in the display control device 2 is realized by a processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in Fig. 11A, or may be a CPU (also called a Central Processing Unit, a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in Fig. 11B.

[0071] When the processing circuit 51 is a dedicated hardware, the processing circuit 51 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. The functions of each of the UI element information acquisition unit 201, the object information acquisition unit 202, and the transparency correction unit 203 may be realized by the processing circuit 51 individually, or the functions of each unit may be realized by the processing circuit 51 collectively.

[0072] When the processing circuit 51 is a CPU 52, the functions of the UI element information acquisition unit 201, the object information acquisition unit 202, and the transparency correction unit 203 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the display control device 2 includes a memory 53 for storing a program that, when executed by the processing circuit 51, results in the execution of each step shown in FIG. 3, for example. It can also be said that these programs cause a computer to execute the procedures and methods of the UI element information acquisition unit 201, the object information acquisition unit 202, and the transparency correction unit 203. Here, examples of memory 53 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, or DVD (Digital Versatile Disc).

[0073] Note that the functions of UI element information acquisition unit 201, object information acquisition unit 202, and transparency correction unit 203 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the functions of UI element information acquisition unit 201 can be realized by processing circuit 51 as dedicated hardware, and the functions of object information acquisition unit 202 and transparency correction unit 203 can be realized by processing circuit 51 reading and executing a program stored in memory 53.

[0074] Thus, the processing circuitry 51 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

[0075] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0076] Various aspects of the present disclosure are summarized below as appendices.

[0077] (Appendix 1) a user interface element information acquisition unit that acquires user interface element information indicating user interface elements to be displayed in real space; an object information acquiring unit that acquires object information indicating the presence or absence of an object approaching an area displaying the user interface element indicated by the user interface element information acquiring unit; a transparency correction unit that corrects transparency of a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the object information acquired by the object information acquisition unit; A display control device comprising: (Appendix 2) a first background information acquisition unit that acquires first background information indicating at least one of a hue and a saturation of a background in an area that displays a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit; a first color correction unit that corrects at least one of hue and saturation for the user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the first background information acquired by the first background information acquisition unit. 2. A display control device according to claim 1. (Appendix 3) a second background information acquisition unit that acquires second background information indicating a brightness of a background in an area that displays a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit; a second color correction unit that corrects brightness of the user interface elements indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the second background information acquired by the second background information acquisition unit. 3. The display control device according to claim 1 or 2. (Appendix 4) a gaze direction information acquisition unit that acquires gaze direction information indicating a gaze direction of a user; a gaze state determination unit that determines whether a user is gazing at a small area within an area displaying user interface elements based on the gaze direction information acquired by the gaze direction information acquisition unit; an enlarged screen creation unit that creates a screen in which a part of the user interface element indicated by the user interface element information acquisition unit is enlarged based on the gaze direction information acquired by the gaze direction information acquisition unit and the determination result by the gaze state determination unit. 4. A display control device according to claim 1, further comprising: (Appendix 5) A step in which a user interface element information acquisition unit acquires user interface element information indicating user interface elements to be displayed in real space; an object information acquiring unit acquiring object information indicating the presence or absence of an object approaching an area displaying a user interface element indicated by the user interface element information acquired by the user interface element information acquiring unit; a transparency correction unit correcting transparency of the user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit based on the object information acquired by the object information acquisition unit; A display control method comprising: [Explanation of symbols]

[0078] 1 device, 2 display control device, 51 processing circuit, 52 CPU, 53 memory, 201 UI element information acquisition unit, 202 object information acquisition unit, 203 transparency correction unit, 204 first background information acquisition unit, 205 second background information acquisition unit, 206 first color correction unit, 207 second color correction unit, 208 gaze direction information acquisition unit, 209 gaze state determination unit, 210 enlarged screen creation unit.

Claims

1. a user interface element information acquisition unit that acquires user interface element information indicating user interface elements to be displayed in real space; an object information acquiring unit that acquires object information indicating the presence or absence of an object approaching an area displaying a user interface element indicated by the user interface element information acquired by the user interface element information acquiring unit; a transparency correction unit that corrects transparency of a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the object information acquired by the object information acquisition unit; A display control device comprising:

2. a first background information acquisition unit that acquires first background information indicating at least one of a hue and a saturation of a background in an area that displays a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit; a first color correction unit that corrects at least one of hue and saturation for the user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the first background information acquired by the first background information acquisition unit.

2. The display control device according to claim 1.

3. a second background information acquisition unit that acquires second background information indicating a brightness of a background in an area that displays a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit; a second color correction unit that corrects brightness of the user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit, based on the second background information acquired by the second background information acquisition unit.

3. The display control device according to claim 1 or 2.

4. a gaze direction information acquisition unit that acquires gaze direction information indicating a gaze direction of a user; a gaze state determination unit that determines whether a user is gazing at a small area within an area displaying user interface elements based on the gaze direction information acquired by the gaze direction information acquisition unit; an enlarged screen creation unit that creates a screen in which a part of the user interface element indicated by the user interface element information acquisition unit is enlarged based on the gaze direction information acquired by the gaze direction information acquisition unit and the determination result by the gaze state determination unit.

2. The display control device according to claim 1.

5. A step in which a user interface element information acquisition unit acquires user interface element information indicating user interface elements to be displayed in real space; An object information acquisition unit acquires object information indicating the presence or absence of an object approaching an area displaying a user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit; a transparency correction unit correcting transparency of the user interface element indicated by the user interface element information acquired by the user interface element information acquisition unit based on the object information acquired by the object information acquisition unit; A display control method comprising:

Images