Glasses-type display device and program

Abstract

To provide an eyewear display device and a program, which can suppress changes in direction of a user's line of sight and body direction in order to view a plurality of display areas.SOLUTION: An eyewear display device includes: display means 17 for displaying one or more display areas 21 to 25 in front of eyes of a user 100; line-of-sight recognition means 13 for recognizing a line of sight of the user 100; and control means 11 for identifying a display area intended by the user based on a movement of the user's line of sight recognized by the line-of-sight recognition means 13.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This invention relates to a glasses-type display device and a program that display one or more display areas in front of the user's eyes while being worn by the user.

Background Art

[0002] Conventionally, when working using multiple displays in an office or the like, the operator has looked at each display by changing the line of sight for each display. Also, depending on the operator, the body orientation has also been adjusted to the display being viewed.

[0003] In recent years, glasses-type display devices have been proposed, enabling the display and operation of a three-dimensional keyboard and multiple displays through the glasses. Also, it has become possible to arrange multiple display areas in a preferred number at preferred positions in an AR (Augmented Reality) space.

[0004] In Patent Document 1, as a glasses-type display device that realizes accurate display operation control according to the user's situation (will, visual state, physical condition, etc.), various images can be visually recognized on a display unit immediately in front of the eyes, providing captured images, reproduced images, received images, etc. At this time, regarding control related to various display operations such as on / off of the display operation, display operation mode, source switching, etc., information regarding the user's operation or physical condition is acquired, the user's will or situation is determined from that information, and appropriate control is performed based on the determination result.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, with conventional glasses-type display devices like those described above, even if multiple display areas are placed in the AR space, in order to view each display area, the user needs to change the direction of their gaze and body orientation to face each display area, just as if multiple displays were physically arranged. This causes fatigue and makes the device inconvenient to use.

[0007] This invention has been made in view of the above technical background, and aims to provide a glasses-type display device and program that can suppress changes in the direction of the user's gaze and body orientation when viewing multiple display areas. [Means for solving the problem]

[0008] The above objectives will be achieved by the following means. (1) A display means that displays multiple display areas, including a display area at a specific position, in front of the user's eyes, A gaze recognition means for recognizing the user's gaze, A determination means for determining whether the user's gaze, as recognized by the gaze recognition means, has moved from a display area at a specific location to a display area other than the specific location, If the aforementioned determination means determines that the user's gaze has moved from the display area at the specific location to a display area other than the specific location, Without moving the entire screen, Display area at the aforementioned specific location itself Move it to a position other than the specific one, and the display area at the destination of the gaze. itself A control means for moving the above-mentioned specific position, A spectacle-type display device characterized by having the following features. (2) The spectacle-type display device described in paragraph 1 above, wherein the display area includes not only a display area that is within the user's field of view, but also a display area that is located in three-dimensional space and is outside the field of view and is not linked to the movement of the spectacle-type display device. (3) The glasses-type display device according to paragraph 1 or 2, wherein the specific position is the front position relative to the glasses-type display device worn by the user or a preset front position. (4) A storage means for storing the position of a display area in a three-dimensional space set by the user, The glasses-type display device and the determination means for determining the relative position of the display area in three-dimensional space, Equipped with, The control means identifies the display area to which the user's gaze is directed based on the determination result of the determination of the relative position of the glasses-type display device and the display area in three-dimensional space by the determination means and the movement of the user's gaze recognized by the gaze recognition means, and moves the identified display area to the identified position, as described in paragraph 1 above. (5) The glasses-type display device according to paragraph 1 or 2, wherein the control means moves the display area based on the user's gaze recognized by the gaze recognition means, as well as trigger conditions set according to the user. (6) A display means that displays one or more display areas in front of the user's eyes, A gaze recognition means for recognizing the user's gaze, A control means that identifies the display area intended by the user based on the user's eye movements recognized by the eye-gaze recognition means, and changes the position of the identified display area. The system includes means for acquiring user information to obtain information about the user's actions or physical condition, Based on the information acquired by the user information acquisition means, if it is determined that the user's gaze has moved along with their face, or if it is determined that the direction of their face or body has been changed to face the display area, the control means This is a spectacle-type display device characterized by not changing the position of the display area. (7) The system is equipped with user information acquisition means for acquiring information about the user's actions or physical condition, Based on the information acquired by the user information acquisition means, if it is determined that the user's gaze has moved along with their face, or if it is determined that the direction of their face or body has been changed to face the display area, the control means This is a glasses-type display device as described in item 1 or 2 above, which does not move the display area. (8) The glasses-type display device according to paragraph 5, further comprising setting means for setting combinations of trigger conditions based on user settings. (9) When multiple display areas exist in the line of sight, the determination means identifies the display area where the user's eyes were focused as the display area to which the line of sight is directed, as described in paragraph 1 or 2 above. (10) The glasses-type display device according to paragraph 1 or 2, wherein the plurality of display areas include not only the display areas above, below, to the left and right of the display area at the specific position, but also the display areas in front and behind. (11) The glasses-type display device described in paragraph 1 or 2 above, wherein the multiple display areas also include multiple window areas displayed within a single screen. (12) The glasses-type display device according to paragraph 1 or 2 above, wherein the movement of the display area is performed by directly swapping the display area at the destination of the gaze with the original display area at a specific position, or by sliding multiple display areas including the original display area at a specific position. (13) The glasses-type display device according to paragraph 1 or 2, wherein the specific position can be changed at will. (14) When the specific position is changed, the control means displays the display area that was displayed at the specific position before the change at the specific position after the change, as described in paragraph 13 above. (15) A computer for a glasses-type display device equipped with display means that displays multiple display areas including a display area at a specific position in front of the user's eyes, A gaze recognition step that recognizes the user's gaze, A determination step to determine whether the user's gaze, as recognized by the gaze recognition step, has moved from the display area at the specific location to a display area other than the specific location, If the above determination step determines that the user's gaze has moved from the display area at the specific location to a display area other than the specific location, Without moving the entire screen, Display area at the aforementioned specific location itself Move it to a position other than the specific one, and the display area at the destination of the gaze. itself A control step of moving the object to the aforementioned specific position, A program to execute. [Effects of the Invention]

[0009] According to the glasses-type display device of the present invention, a plurality of display areas including a display area at a specific position are displayed in front of the eyes of the user wearing the glasses-type display device, and it is determined whether the line of sight of the user recognized by the line-of-sight recognition means has moved from the display area at the specific position to a display area other than the specific position. When it is determined that the line of sight of the user has moved from the display area at the specific position to a display area other than the specific position, Without moving the entire screen, the display area at a specific position itself is moved to a position other than the specific position, and the display area at the destination of the line of sight itself is moved to the specific position. Therefore, the display area that the user wants to view can be displayed at the specific position. For this reason, the user does not need to change the direction of the line of sight or the body in the direction of each display area to view the plurality of display areas, so stable use can be achieved without causing fatigue.

[0010] Also, according to the program of the present invention, on a computer of a glasses-type display device including display means for displaying a plurality of display areas including a display area at a specific position in front of the eyes of the user, a line-of-sight recognition step for recognizing the line of sight of the user, and a determination step for determining whether the line of sight of the user recognized in the line-of-sight recognition step has moved from the display area at the specific position to a display area other than the specific position. When it is determined in the determination step that the line of sight of the user has moved from the display area in the front to a display area other than the front, Without moving the entire screen, the display area at a fixed position itself is moved to a position other than the specific position, and the display area at the destination of the line of sight itself is moved to the specific position, and the computer can be made to execute a control step.

Brief Description of the Drawings

[0011] [Figure 1] It is a block diagram showing the configuration of a glasses-type display device according to an embodiment of the present invention. [Figure 2] It is a flowchart showing an example of the operation of a glasses-type display device worn by a user. [Figure 3]This is an explanatory diagram illustrating an example of changing the position of the display area. [Figure 4] This table shows an example of trigger conditions. [Figure 5] This flowchart shows the operation of the glasses-type display device when trigger conditions are set. [Figure 6] This is an illustrative diagram of another example of changing the position of the display area. [Figure 7] This is an explanatory diagram illustrating yet another example of changing the position of the display area. [Figure 8] (A) and (B) are diagrams illustrating how to change the position of the display area to which the gaze is directed, to the position of the display area directly in front. [Figure 9] (A) and (B) are diagrams illustrating that when the user's orientation changes by 90 degrees, the front position also changes. [Modes for carrying out the invention]

[0012] Hereinafter, embodiments of this invention will be described based on the drawings.

[0013] Figure 1 is a block diagram showing the configuration of a glasses-type display device 1 according to one embodiment of this invention.

[0014] The glasses-type display device 1 comprises a control unit 11, an imaging unit 12, a gaze recognition unit 13, a motion detection unit 14, a storage unit 15, a determination unit 16, a display unit 17, an image processing unit 18, a communication unit 19, etc., and has a glasses shape that can be worn by the user.

[0015] The control unit 11 provides overall control for the glasses-type display device 1 and includes a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), and other components. The CPU executes processing by operating according to the operating program stored in the ROM, etc. RAM is memory that provides a workspace for the CPU when it operates according to the operating program. ROM is memory that holds the CPU's operating program and other data. The specific operation of the control unit 11 will be described later.

[0016] The imaging unit 12 is a camera that captures images of the area in front of the glasses-type display device 1 worn by the user, as well as the user's eye movements, etc.

[0017] Under the control of the control unit 11, the gaze recognition unit 13 monitors the gaze of the user wearing the glasses-type display device 1 from the user's eye movements captured by the imaging unit 12, and recognizes the movement of the gaze, focal point, etc.

[0018] The motion detection unit 14 is equipped with a motion capture sensor, an acceleration sensor, a gyroscope sensor, etc., and detects and acquires information about the user's movements or physical condition, such as changes in the direction of the user's face or body.

[0019] The memory unit 15 stores various data, such as the front position (coordinates), the three-dimensional spatial positions (coordinates) of one or more display areas displayed by the display unit 17, trigger conditions for changing the position of the display areas (described later), images captured by the imaging unit 12, and application software.

[0020] The determination unit 16 determines whether the trigger conditions for moving a specific display area among the multiple display areas displayed by the display unit 17 have been met, determines the relative position of the glasses-type display device 1 and the display area in three-dimensional space, and performs other determinations. The determination unit 16 may be configured as part of the functions of the control unit 11.

[0021] The display unit 17 displays one or more display areas in front of the user wearing the glasses-type display device 1.

[0022] The image processing unit 18 performs image processing on the image in the display area before it is displayed on the display unit 17.

[0023] The communication unit 19 performs data communication with external devices via a network, short-range wireless communication, etc.

[0024] Next, an example of the operation of the glasses-type display device 1 worn by the user will be explained with reference to the flowchart in Figure 2. This operation, and the operations described thereafter, are performed by the CPU of the glasses-type display device 1 operating according to an operation program stored in ROM or the like.

[0025] In this embodiment, the display area, which is located in a position other than directly in front of the user, is configured to change to a position directly in front of the user in response to the user's eye movements.

[0026] In step S01, the control unit 11 determines, based on the gaze recognition by the gaze recognition unit 13, whether the gaze has moved to an area other than the display area directly in front. If the gaze has not moved (NO in step S01), nothing is done in step S02. In other words, the position of the display area is not changed.

[0027] If the gaze shifts (YES in step S01), step S03 determines whether a display area exists in the front position. If it does not exist (NO in step S03), step S04 identifies the display area the gaze was directed to, i.e., the display area intended for the user to see, and moves this identified display area to the front position. If a display area exists in the front position (YES in step S03), step S05 swaps the display area the gaze was directed to with the front display area.

[0028] For example, as shown in Figure 3, a display area 21 is displayed in the front position, and four display areas 22 to 25 are displayed above, below, to the left and right of this front display area 21. When a user 100 wearing glasses-type display device 1 shifts their gaze from the front display area 21 to the upper left display area 22, the upper left display area 22 is identified as the display area intended by the user, and the positions of the upper left display area 22 and the front display area 21 are swapped and displayed as indicated by arrow A1.

[0029] Thus, the display area 22 to which the user 100 wearing the glasses-type display device 1 directs their gaze can be considered the display area that the user 100 is trying to see. By changing the position of this display area 22 to the position of the display area 21 in front, the display area that the user 100 is trying to see will always be in front of them. Therefore, the user 100 does not need to change the direction of their gaze or body orientation to view multiple display areas, enabling stable use without causing fatigue.

[0030] In the example described above, the position of the display area 22 where the user's gaze was directed was changed to the frontal position. However, in a configuration where the display area is changed solely based on the movement of the user's gaze, the position of the display area may be changed against the user's intention.

[0031] Therefore, in addition to eye movement, other conditions may be added as trigger conditions for changing the position of the display area. An example of trigger conditions is shown in the table in Figure 4. In the example shown in this table, the display area is set to move to the front only if the eye moves and the user gazes at the destination display area for 1 second, or if the eye moves and the user blinks twice. If it is determined that user 100's gaze has moved along with their face, or if it is determined that they have changed the direction of their face or body to face the display area, the system will decide that they want to see both the front display area and the display area at the point of their gaze, and will not switch positions. Note that trigger conditions may be set for each user 100.

[0032] In this way, by adding other conditions in addition to eye movement, it is possible to prevent situations where the position of the display area is changed against the user's (100) intention.

[0033] Figure 5 is a flowchart showing the operation of the glasses-type display device 1 when the trigger conditions described above are set.

[0034] In step S11, the control unit 11 determines, based on the gaze recognition by the gaze recognition unit 13, whether the gaze has moved outside the display area directly in front of the user. If the gaze has not moved (NO in step S11), nothing is done in step S12. In other words, the position of the display area is not changed.

[0035] If the gaze shifts (YES in step S11), step S13 determines whether the set trigger condition has been met. If it has not been met (NO in step S13), nothing happens in step S14 (the position of the display area is not changed).

[0036] If the trigger condition is met (YES in step S13), in step S15, it is determined whether or not a display area exists in the front position. If it does not exist (NO in step S15), in step S16, the display area that the gaze was directed to is identified and this identified display area is moved to the front position. If a display area exists in the front position (YES in step S15), in step S17, the display area that the gaze was directed to and the front display area are swapped.

[0037] In this embodiment, when changing the display area to which the gaze is directed (for example, display area 22 in Figure 3) to the front position or swapping it with the front display area (for example, display area 21 in Figure 3), "front" refers to at least one of the following: the position of the center of the front of the glasses-type display device 1, which changes in accordance with the orientation of the glasses-type display device 1, or the position indicated by coordinates (direction) set in advance by the user 100.

[0038] Furthermore, the display areas may be fixed in position in three-dimensional space. For example, as shown in the schematic plan view of Figure 6, one display area 31 is positioned in front, and a display area 34 is positioned at a distance from the back of it. Display areas 32 and 33 are positioned between these two display areas 31 and 34, on either side of them.

[0039] When user 100 shifts their gaze from the display area 31 on the front side to the display area on the back side, as indicated by arrow A2, the display area 34 on the back side and the display area 31 on the front side may be swapped, as indicated by arrow A3. In this case, it may be detected that the focus of the gaze has shifted to the display area 34 on the back side, and the display area 34 on the back side may be identified as the display area intended by the user. Furthermore, if there are multiple display areas in the line of sight, not just the front and back display areas 31 and 34, it may be difficult to identify the display area intended by the user based solely on the movement of the gaze, so the display area that is in focus may be identified.

[0040] Furthermore, if user 100 fixes and sets the position (coordinates) of the display area in three-dimensional space, the display area to which user 100's gaze moves can be identified as follows.

[0041] In other words, the position of the display area in the set three-dimensional space is stored in the storage unit 15, the relative position of the glasses-type display device 1 and the display area in three-dimensional space is determined, the display area intended by the user is identified from the determined relative position and the movement of the user's gaze recognized by the gaze recognition unit 13, and the position of the identified display area is changed to the front position. As a result, even if the orientation of the glasses-type display device 1 is changed, the display area in three-dimensional space that is located where the gaze is moving can be easily identified.

[0042] Furthermore, the display area may be located outside the user's field of view, not just within the user's 100's field of view, but also in a three-dimensional space and not linked to the movement of the glasses-type display device 1. Even if the display area is outside the field of view, the user can identify the display area at the destination of their gaze, and then simply move this identified display area to the front.

[0043] Figure 7 illustrates another example of a display area. As shown in Figure 3 (display areas 21-25) and Figure 6 (display areas 31-34), the display areas may be independent of each other, but as shown in Figure 7, it also includes one or more window areas 41-45 displayed within a single screen 4. In the example in Figure 7, in addition to the window area 41 in the front position, a total of four window areas 42-45 are displayed around the front window area 41. In this case as well, when the user 100 moves their gaze from the front window area 41 to, for example, window area 42, the system can identify window area 42 as the display area the user wants to see from the movement of their gaze, and then swap this window area 42 with the front window area 41 as indicated by arrow A4. Also, a single screen 4 may be a flat surface or a curved surface, etc.

[0044] Furthermore, one or more window areas 41-45 within one screen 4 and one or more window areas within another screen may be displayed simultaneously, or one or more window areas within one screen 4 or multiple screens and an independent display area outside the screen may be displayed simultaneously. In addition, an independent display area may be set in a position outside the user's field of view.

[0045] Figures 8(A) and 8(B) illustrate how to change the position of the display area to which the user's gaze is directed, to the position of the display area directly in front of them. In Figures 8(A) and 8(B), four display areas 51 to 54 are displayed side by side. The display area directly in front of the user is designated as display area 52, and the display area to which the user 100 wearing the glasses-type display device 1 directs their gaze is designated as display area 54.

[0046] In Figure 8(A), the display area 54 at the point of gaze and the display area 52 in front are directly swapped, as shown by arrows A5 and A6. In Figure 8(B), the display area 54 at the point of gaze is moved to the position of the front display area 52, as shown by arrow A7, and the swap is performed by sliding each display area from the front display area 52 to the display area 53 adjacent to the display area 54 at the point of gaze one position, as shown by arrows A8 and A9.

[0047] In the above embodiment, the display area where the gaze is directed is changed to the front position. For example, as shown in Figure 9(A), when a user 100 sitting at an L-shaped desk 7 turns 90 degrees from a state where they were facing forward, as shown in Figure 9(B), the front position also changes accordingly, and the display area 61 that was displayed in the original front position is displayed as a display area 62 in the front position after the change of direction. If the user 100 has set a position in three-dimensional space as the front, and the user 100 changes direction, the relative position of the set front position and the glasses-type display device 1 should be determined, and the set front position should be moved so that the relative position after the change of direction is the same.

[0048] Furthermore, the display area where the user's gaze is directed may be changed from the frontal position to a pre-set position other than the frontal position. [Explanation of Symbols]

[0049] 1. Glasses-type display device 11 Control Unit 12 Imaging Unit 13 Gaze recognition section 14 Motion detection unit 15 Storage section 16 Judgment section 17 Display section 18 Image Processing Unit 19 Communications Department 21~25, 31~34, 51~54, 61, 62 display area 4 screens 41-45 Window area (display area) 7 L-shaped desk 100 users

Claims

1. A display means that displays multiple display areas, including a display area at a specific position, in front of the user's eyes, A gaze recognition means for recognizing the user's gaze, A determination means for determining whether the user's gaze, as recognized by the gaze recognition means, has moved from a display area at a specific location to a display area other than the specific location, If the determination means determines that the user's gaze has moved from the display area at the specific location to a display area other than the specific location, the control means moves the display area at the specific location itself to a location other than the specific location, without moving the entire screen, and moves the display area to which the gaze has moved itself to the specific location. A spectacle-type display device characterized by having the following features.

2. The spectacle-type display device according to claim 1, wherein the display area includes not only a display area within the user's field of view, but also a display area located in three-dimensional space and outside the field of view that is not linked to the movement of the spectacle-type display device.

3. The glasses-type display device according to claim 1 or 2, wherein the specific position is a frontal position relative to the glasses-type display device worn by the user or a preset frontal position.

4. A storage means for storing the position of a display area in a three-dimensional space set by the user, The glasses-type display device and the determination means for determining the relative position of the display area in three-dimensional space, Equipped with, The glasses-type display device according to claim 1, wherein the control means identifies a display area to which the user's gaze is directed based on the determination result of the determination of the relative position of the glasses-type display device and the display area in three-dimensional space by the determination means and the movement of the user's gaze recognized by the gaze recognition means, and moves the identified display area to the identified position.

5. The glasses-type display device according to claim 1 or 2, wherein the control means moves the display area based on the user's gaze recognized by the gaze recognition means, as well as trigger conditions set according to the user.

6. A display means that displays one or more display areas in front of the user's eyes, A gaze recognition means for recognizing the user's gaze, A control means that identifies the display area intended by the user based on the user's eye movements recognized by the eye-gaze recognition means, and changes the position of the identified display area. The system includes means for acquiring user information to obtain information about the user's actions or physical condition, A glasses-type display device characterized in that, based on the information acquired by the user information acquisition means, if it is determined that the user's gaze has moved along with their face, or that the user has changed the direction of their face or body to face the display area, the control means does not change the position of the display area.

7. The system includes means for acquiring user information to obtain information about the user's actions or physical condition, The glasses-type display device according to claim 1 or 2, wherein, based on the information acquired by the user information acquisition means, if it is determined that the user's gaze has moved along with their face, or if it is determined that the direction of their face or body has been changed to face the display area, the control means does not move the display area.

8. The glasses-type display device according to claim 5, further comprising setting means capable of setting a combination of trigger conditions based on the user's setting operations.

9. The spectacle-type display device according to claim 1 or 2, wherein, when multiple display areas exist in the line of sight, the determination means identifies the display area where the user's eyes were focused as the display area to which the line of sight has moved.

10. The spectacle-type display device according to claim 1 or 2, wherein the plurality of display areas include not only the display areas above, below, to the left and right of the display area at the specific position, but also the display areas in front and behind.

11. The glasses-type display device according to claim 1 or 2, wherein the multiple display areas also include multiple window areas displayed within a single screen.

12. The glasses-type display device according to claim 1 or 2, wherein the movement of the display area is performed by directly swapping the display area at the destination of the gaze with the original display area at a specific position, or by sliding a plurality of display areas including the original display area at a specific position.

13. The glasses-type display device according to claim 1 or 2, wherein the aforementioned specific position can be arbitrarily changed.

14. The spectacle-type display device according to claim 13, wherein when the specified position is changed, the control means displays the display area that was displayed at the specified position before the change at the specified position after the change.

15. A computer for a glasses-type display device equipped with display means that displays multiple display areas, including a display area at a specific position, in front of the user's eyes, A gaze recognition step that recognizes the user's gaze, A determination step to determine whether the user's gaze, as recognized by the gaze recognition step, has moved from the display area at the specific location to a display area other than the specific location, If the determination step determines that the user's gaze has moved from the display area at the specific location to a display area other than the specific location, the control step involves moving the display area at the specific location itself to a location other than the specific location, without moving the entire screen, and moving the display area to which the gaze has moved itself to the specific location. A program to execute.

Images