Display device and display method

The head-mounted display device addresses visual obstruction in three-dimensional arrangements by detecting gaze points and adjusting the display mode of obstructing objects, ensuring clear visibility of desired objects.

JP2024161159AActive Publication Date: 2024-11-15MAXELL LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2024146593
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-08-28
Publication Date
2024-11-15
Estimated Expiration
2040-12-10

AI Technical Summary

Technical Problem

Conventional head-mounted information processing devices face issues with visual obstruction when real and virtual objects overlap, making it difficult for users to see desired objects due to occlusion, especially in three-dimensional arrangements, and existing solutions do not adequately address this problem.

Method used

A head-mounted display device that detects the user's gaze point and identifies target objects, changing the display mode of obstructing objects to eliminate or reduce visibility obstruction by adjusting transparency, position, size, or duplicating the objects to ensure the user can see the desired object without effort.

Benefits of technology

The device effectively eliminates or reduces visual obstruction, allowing users to view desired objects in three-dimensional arrangements by adjusting the display mode of obstructing objects, enhancing user experience and ease of use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024161159000001_ABST
    Figure 2024161159000001_ABST
Patent Text Reader

Abstract

To provide a technology which can eliminate or reduce visual obstruction with respect to a display device when an object such as a real object or a virtual object which a user wants to visually recognize is visually obstructed by masking or the like due to another object.SOLUTION: A display device according to an embodiment comprises a display and a processor and displays at least three-dimensionally arranged virtual objects out of individual entity objects cut out from entities in an external world and the three-dimensionally arranged virtual objects, as objects on the display and settles an object which a user wants to gaze, as a target object and detects an object which obstructs the user visually recognizing the target object, as an obstructive object and, if the obstructive object is present, changes a display mode of at least one of the target object and the obstructive object so as to eliminate or reduce obstruction to visual recognition of the target object due to the obstructive object.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to technology for display devices and information processing devices, and to technology for displaying images of virtual objects and the like. [Background technology]

[0002] In recent years, display devices capable of displaying images of virtual objects and the like have come into widespread use. One example is a head-mounted information processing device (so-called head-mounted display: HMD) that is worn on the user's head and includes a display unit, a camera unit, and the like. The head-mounted information processing device displays real objects and virtual objects in real space, seamlessly blending the real world and the virtual world in real time, and allows the user to experience as if a virtual object were present in the real world. Display methods include the so-called video see-through type and optical see-through type. In the case of the video see-through type, images corresponding to the real objects and virtual objects are generated and displayed on a display unit in front of the head. In the case of the optical see-through type, the real object in front of the user is made visible, and an image of the virtual object is displayed on the display unit superimposed on it.

[0003] An example of prior art related to the above display device is JP 2015-090635 A (Patent Document 1). Patent Document 1 states that "information is appropriately displayed while ensuring the user's field of vision" and the following: In an information display system having a see-through head mounted display, a control unit detects a user's gaze point based on imaging data of both eyes of the user, determines whether the user is gazing at a virtual screen or at a background beyond the virtual screen based on the gaze point, determines whether the user's gaze area overlaps with a display position of an object on the virtual screen, and changes the display position and / or display form of the object based on the determination result when the gaze point moves. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-090635 A Summary of the Invention [Problem to be solved by the invention]

[0005] In display devices such as conventional head-mounted information processing devices, when real objects in real space and virtual objects are displayed overlapping each other, depending on their relative positions and the user's line of sight, the real object or virtual object that the user wants to view may be blocked by other real objects or virtual objects, making it difficult to view or obstructing viewing.

[0006] Patent Document 1 describes that in an HMD that allows a real object to be seen through realistically and displays a virtual object on a virtual screen, it is determined based on the user's gaze point whether the user is gazing at the virtual screen or the background, and whether the line of sight overlaps an object on the virtual screen, and the display position and / or display form of the virtual object is changed according to both determination results. Patent Document 1 describes that when an object that the user is gazing at is overlapped and covered by a virtual object, the information display system changes the display position or the display form of the virtual object depending on the transparency and displays it. However, Patent Document 1 considers eliminating visual obstruction in the line of sight, but that is all, and does not consider obstruction to the range that the user wants to view. Patent Document 1 also does not suggest any display that reflects the occlusion relationship when real objects and virtual objects are arranged in three dimensions (Three-Dimensional: 3D).

[0007] The object of the present invention is to provide a technology for a display device such as a head-mounted information processing device capable of displaying virtual objects arranged in three dimensions, which can eliminate or reduce the visual obstruction caused by occlusion by other objects in the visual range of an object such as a real object or virtual object that the user wishes to view, thereby enabling the user to visually view the entire object in an optimal manner, and can realize such a function with less effort for the user and in a convenient manner. Problems and effects other than those mentioned above are described in [Mode for carrying out the invention]. [Means for solving the problem]

[0008] A representative embodiment of the present invention has the following configuration: A display device according to the embodiment includes a display device for displaying an image and a processor for controlling the display of the image, and displays at least one of an individual real object cut out from a real object in the external world and a virtual object arranged in three dimensions on the display device as an object, determines the object at which a user wishes to gaze as a target object, detects the object that interferes with the user's viewing of the target object as an obstructing object, and, if there is an obstructing object, changes the display mode of at least one of the target object and the obstructing object so as to eliminate or reduce the obstruction caused by the obstructing object in viewing the target object. Effect of the Invention

[0009] According to a representative embodiment of the present invention, with respect to technology for a display device such as a head-mounted information processing device capable of displaying virtual objects arranged in a three-dimensional manner, when a user wishes to view an object such as a real object or a virtual object and the viewing range of the object is obstructed by another object, the obstruction to viewing can be eliminated or reduced, allowing the user to conveniently view the entire object, and such a function can be realized in an easy-to-use manner with little effort for the user. [Brief description of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration and a display example of a head-mounted information processing device (HMD) which is a display device according to a first embodiment of the present invention. [Diagram 2] In the first embodiment, classification of objects, occlusion relationships, and categories are described. [Diagram 3] In the first embodiment, a display example in the case of adjusting the transparency will be shown. [Figure 4] In the first embodiment, a display example in the case of adjusting the transparency will be shown. [Diagram 5] In the first embodiment, a display example in the case of reduction / enlargement will be shown. [Figure 6] In the first embodiment, a display example in the case of moving the display position will be shown. [Figure 7] In the first embodiment, a display example in the case of moving the display position will be shown. [Figure 8] In the first embodiment, a display example in the case of duplicate display will be shown. [Figure 9] In the first embodiment, the main processing flow will be described. [Figure 10] In the first embodiment, a functional block configuration example is shown. [Figure 11] In the first embodiment, a display example will be shown. [Figure 12] In the first embodiment, a display example will be shown. [Figure 13] In the first embodiment, a processing flow of an operation example will be shown. [Figure 14] In the first embodiment, a display example will be shown. [Figure 15] In the first embodiment, a display example will be shown. [Figure 16] In the first embodiment, a display example will be shown. [Figure 17] In the first embodiment, a display example will be shown. [Figure 18] In the first embodiment, a display example will be shown. [Figure 19] In the first embodiment, a display example will be shown. [Figure 20] 13 shows a display example in the display device according to the second embodiment of the present invention. [Figure 21] A display example will be shown in the second embodiment. [Figure 22] In the second embodiment, a supplementary explanatory diagram is shown. [Figure 23] In the second embodiment, a processing flow of an operation example will be shown. [Figure 24] In each embodiment, an example of object data is shown. [Diagram 25] 13 shows a first example of sharing in the display device according to the third embodiment of the present invention. [Figure 26] 13 shows a second example of sharing in the display device according to the third embodiment of the present invention. [Figure 27] A display example will be shown in the third embodiment. [Figure 28] A display example will be shown in the third embodiment. [Figure 29] A display example will be shown in the third embodiment. [Diagram 30] A display example will be shown in the third embodiment. [Diagram 31] A display example will be shown in the third embodiment. [Diagram 32] 13 shows a display example in a display device according to a fourth embodiment of the present invention. [Diagram 33] 13 shows a display example in a modified example of the fourth embodiment. [Diagram 34] 13 shows a display example in a display device according to a fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings, the same parts are generally given the same reference numerals, and repeated explanations are omitted. In the drawings, the representation of each component may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention, and the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. Unless otherwise limited, each component may be singular or plural. For the purpose of explanation, when describing processing by a program, the program, function, processing unit, etc. may be described as the main body, but the main body of hardware for them is a processor, or a controller, device, computer, system, etc., configured with the processor, etc. The computer executes processing according to the program read onto the memory while appropriately using resources such as memory and communication interface by the processor. This realizes a predetermined function, processing unit, etc. The processor is configured with a semiconductor device such as a CPU or GPU. The processor is configured with a device or circuit capable of performing a predetermined calculation. The processing is not limited to software program processing, and can also be implemented with a dedicated circuit. The dedicated circuit can be an FPGA, ASIC, etc. The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer and installed. The program source may be a program distribution server on a communication network, or a non-transient computer-readable storage medium. The program may be composed of multiple program modules. For the purpose of explanation, various data and information may be described using expressions such as tables and lists, but are not limited to such structures and formats. Furthermore, data and information for identifying various elements may be described using expressions such as identification information, identifiers, IDs, names, numbers, etc., but these expressions are interchangeable.

[0012] <Embodiment 1> A display device and a display method according to a first embodiment of the present invention will be described with reference to Fig. 1 and other figures. The display device according to the first embodiment is a virtual object display device, and is applied to a head-mounted information processing device (hereinafter referred to as HMD). The display method according to the first embodiment is a method having steps executed by the display device according to the first embodiment.

[0013] The display device of the first embodiment includes a display device (in other words, a display) capable of displaying virtual objects, and a processor for controlling the display of the virtual objects on the display device, and displays at least virtual objects on the display surface of the display device, which are real objects of the outside world including, as parts thereof, individual real objects cut out or recognized from real objects of the outside world, and virtual objects arranged in three dimensions. In the case of a video see-through type, the individual real objects and virtual objects can be displayed as images that are objects. In the case of an optical see-through type, a virtual object can be displayed as an object so as to match the real object.

[0014] The display device of the first embodiment determines and determines an individual real object or virtual object that is an object that the user wishes to gaze at as a target object, and detects an individual real object or virtual object that is an object that interferes with the user's viewing of the target object as an obstructing object. When the display device of the embodiment detects the presence of an obstructing object, it changes the display mode of at least one of the target object and the obstructing object so as to eliminate or reduce the obstruction of the obstructing object with respect to the viewing of the target object.

[0015] [1. Display device (HMD)] FIG. 1 shows an outline of the configuration and a display example of a head-mounted information processing device (HMD) 1, which is a display device of the first embodiment. FIG. 1 shows a schematic configuration of the appearance when a user U1 wears the HMD 1 on his head. FIG. 1 also shows how the user U1 views an image of a three-dimensional object displayed in a field of view 101 by the HMD 1. FIG. 1 also shows an example of a change in the display mode of the object in the field of view 101. (a) is a display example before the change, showing a case where there is an occlusion obstruction relationship between objects "A" and "B". (b) is a display example after the change, showing a state where the occlusion obstruction relationship between objects "A" and "B" is temporarily resolved.

[0016] The HMD1 is worn on the head of the user U1, and displays an image of an object or the like within a visual field 101 of the user U1. The visual field 101 is associated with a display surface 11 of a display device provided in the HMD1. The object is an individual real object that is a part of a real object, or a virtual object arranged in three dimensions. The user U1 can view, for example, objects 102 and 103 within the visual field 101. For example, the object 102 is a virtual object with a rectangular parallelepiped shape and marked "B". The object 103 is a virtual object with a rectangular parallelepiped shape and marked "A". As viewed from the user U1, the object 102 is located behind the object 103 located in front. The object 103 in front covers at least a part of the object 102 in the rear, thereby obstructing the viewing of the object 102, in other words, making it difficult to view. For the sake of explanation, such objects 102 and 103 (a pair of two objects) are described as objects in a "covering obstruction relationship".

[0017] The lines of sight of both eyes of the user U1 include a line of sight 104 of the left eye and a line of sight 105 of the right eye. A gaze point 106, which is a position where the user U1 is gazing in a three-dimensional space, can be calculated based on the directions of the lines of sight 104 and 105 of the user U1's eyes. An object located near the gaze point 106, for example, the object "B" 102, is associated with a desired object that the user U1 gazes at and visually recognizes as a target or goal. The HMD1 determines and determines such an object as a target object based on the lines of sight and the gaze points of both eyes. In this example, in (a), the object "B" 102 where the gaze point 106 is located is determined as the target object. The HMD1 sets a target visual range 107 for the target object. The target visual range 107 is a range that is related to the target object and is estimated to be desired to be visually recognized by the user U1.

[0018] On the other hand, the "A" object 103 in front occludes a part (e.g., the lower left part) of the target visual range 107 of the "B" object 102, which is the target object that the user U1 intends to view. In this case, the occluding "A" object 103 prevents the user U1 from viewing the entire target visual range 107 of the "B" object 102, which is the target object. The HMD1 judges and detects such an object that obstructs visibility as an obstructing object. The HMD1 grasps the relationship between objects such as "A" and "B" as an "occlusion obstruction relationship."

[0019] When such an obstruction relationship exists, the HMD1 changes the display mode of these objects. For example, the HMD1 changes the display mode of the object 103 of "A", which is an obstructing object obstructing the target visual range 107. Specifically, in this example, the HMD1 changes the display position of the object 103 of "A" to a position outside the target visual range 107 within the field of view 101. In other words, the HMD1 moves the object 103 to an empty position outside the target visual range 107 and replaces it with the state of the object 103a after the movement. By this change, the HMD1 makes the entire target visual range 107 unobstructed. This allows the user U1 to view the entire target visual range 107 of the object 102 of "B", which is the target object. When moving the object 103 of "A", which is an obstructing object, the HMD1 may determine the display position after the movement so as not to be far from the original display position of the object 103 and the target object as much as possible. The above example of changing the display mode of the object in the occlusion obstruction relationship is an example of changing the display position on the obstruction object side, but the present invention is not limited to this and various change methods described later are possible.

[0020] The information and data of the virtual objects and the like may be generated within the HMD 1, or may be generated outside the HMD 1, for example, by an information server 120, and supplied to the HMD 1 via an external network. The information server 120 can handle large amounts of information, and can generate and hold, for example, high-quality, high-definition virtual objects. The external device may also be a user's mobile information terminal, a home device, or the like.

[0021] [Target object] In the first embodiment, a gaze point 106 in a three-dimensional space that can be calculated from two gaze directions 104 and 105 in FIG. 1 is used as a means for specifying, determining, and confirming a target object that is an object that the user U1 wants to gaze at. For example, the HMD1 can determine an object that is closest to the position of the gaze point 106 as the target object. This means is not limited to this, and various means can be applied. Other means include a pointer by a remote controller or the like, voice input, and recognition of a gesture by hand. When using a pointer, the user U1 operates the pointer displayed on the display surface by a remote controller or the like. The HMD1 may determine, as the target object, an object where the pointer is located within the field of view 101, or an object specified by an on operation of the pointer. In the case of voice input, the user U1 inputs information identifying a displayed object by voice. The HMD1 recognizes the input voice, and when it recognizes, for example, "B", it may determine the object 102 of "B" as the target object.

[0022] [Object] Further explanation of terminology will be given with reference to FIG. 2. FIG. 2(A) shows classification of "objects". In the first embodiment, there are two types of objects that the HMD 1 displays on the display surface 11. The two types of objects are described as "individual entity objects" and "virtual objects". These objects are elements that can constitute an occlusion obstruction relationship. In the HMD 1 of the first embodiment, these objects are objects that can be arranged three-dimensionally in the field of view 101 corresponding to the display surface 11. That is, these objects are objects that can be arranged in front of or behind the field of view 101 in the depth direction as viewed from the viewpoint of the user U1. Objects arranged in front of or behind each other may overlap each other, resulting in an occlusion obstruction relationship.

[0023] This object is not necessarily an image (referring to an image generated by a display device). An "individual real object" is an object based on a real object (in other words, a real image). In the case of a video see-through type, an "individual real object" is an image of an individual real object cut out from the real object. In the case of an optical see-through type, an "individual real object" is an individual real object cut out from the real object (in other words, recognized) and is not an image. A "virtual object" is an image of any virtual object generated by a display device in association with or independent of a real object.

[0024] FIG. 2B shows a pattern of the occlusion obstruction relationship of objects in the first embodiment. There are four patterns in which the above two types of objects are arranged one behind the other. In the first pattern, an individual entity object is arranged in front of an individual entity object in the back. In the second pattern, a virtual object is arranged in front of an individual entity object in the back. In the third pattern, an individual entity object is arranged in front of a virtual object in the back. In the fourth pattern, a virtual object is arranged in front of a virtual object in the back. With some exceptions, the HMD 1 of the first embodiment can apply display mode changes to each of these patterns.

[0025] [Example] 3 to 8 show various examples of changes in the display mode as display examples in the field of view 101 corresponding to the display surface 11 of the HMD 1. FIG.

[0026] [Display example (1)] FIG. 3A shows an example of a display mode change in which the transparency (in other words, the transparency) of the object 103 of "A" which is an obstructing object on the front side is increased when there is an obstructing relationship between the objects "A" and "B" as shown in FIG. 1A. This allows the obstructing object to be transparent, making it easier to view the target visual range 107 of the object 102 of "B" which is a partially obstructed target object. This allows the user U1 to view the entire target visual range 107 of the target object. In this example, the HMD1 increases the transparency of only the portion 103X of the image area of ​​the object 103 which is an obstructing object on the front side and obstructs the target visual range 107 on the rear side, making it closer to transparent. This increase in transparency adjustment reduces the degree of obstruction to visibility.

[0027] FIG. 3B shows another example, in which the HMD 1 adjusts the transparency of only the portion 103X of the front obstructing object that is blocking the target visible range 107 to the maximum state. In other words, the portion 103X with maximum transparency is made invisible. In terms of appearance, the target visible range 107 of the rear "B" object 102 appears to be temporarily in front of the "A" object 103. As a result, the target visible range 107 is not blocked at all, and visibility obstruction can be eliminated.

[0028] FIG. 4A shows another example in which the same transparency increase adjustment is performed for all of the obstruction objects "A" 103. This makes all of the obstruction objects transparent, making it easier to see the target visual recognition range 107. At the same time, the obstruction objects are displayed with the same transparency, making them easier to see.

[0029] FIG. 4B shows another example in which all of the obstructing objects "A" 103 are hidden with maximum transparency. In this case, the obstructing objects are completely invisible, making it easy to confirm the entire target visible range 107. As shown in the examples of FIG. 1 to FIG. 4 above, the obstruction of visibility of the target object's target visible range by the obstructing objects can be eliminated or the degree of obstruction can be reduced by changing the display mode.

[0030] [Display example (2)] FIG. 5 shows another example of a display mode change. The change from (a) to (b) in FIG. 5 shows a case where the object 103 of "A", which is an obstructing object, is reduced and the transparency is adjusted to be increased. After the change, the object 103 of "A" is replaced with the object 103b. In this way, the HMD 1 changes the size of the obstructing object so as to make it smaller relative to the target object. This can further reduce the degree of visual obstruction caused by the obstructing object. In addition, the obstructing object may be reduced only, and the effect of making it easier to confirm the target visual range can be obtained. Similarly, as another method, the change from (a) to (c) in FIG. 5 shows a case where the object 102 of "B", which is an obstructing object of "A", is enlarged. After the change, the object 102 of "B" is replaced with the enlarged object 102c. Even in this case, the effect of making it easier to confirm the target visual range can be obtained.

[0031] [Display example (3)] Fig. 6 shows an example of changing the display mode of the target object rather than the obstruction object. For example, the target object is a virtual object, and the obstruction object is a virtual object or an individual entity object that is less suitable for increasing the transparency or changing the display position than the target object in terms of visual discomfort of the user U1. In this case, changing the display mode of the target object as shown in Fig. 6 is useful.

[0032] FIG. 6 shows a case where the display position of a target object is changed. In the state before the change in (a), the "C" object 109 is arranged in the front, and the "B" object 102, which is a virtual object, is arranged in the back. The target object is the "B" object 102. A part of the target visibility range 107 of the "B" target object is occluded by the "C" object 109. The "C" object 109 in the front is a virtual object or an individual real object that is not suitable for adjusting the transparency up or changing the display position. (b) shows the state after the change. The HMD 1 moves the display position of the "B" object 102, which is the target object, to outside the occlusion range of the "C" object 109, which is an obstructing object. The "B" object 102 and the target visibility range 107 are replaced by the moved object 102b and the target visibility range 107b. As a result, the entire target visibility range 107b is visible. With this movement, the gaze point of the user U1 moves, for example, from the gaze point 106 to the gaze point 106b. This allows the user U1 to view the entire target visual range 107b of the moved object 102b located at the gaze point 106b. This is equivalent to viewing the entire target visual range 107 of the original object 102.

[0033] When moving a target object, the HMD 1 moves it to an empty position within the field of view 101, that is, to a position that does not obstruct the viewing of other objects. In this example, since the left side of objects "B" and "C" is empty, the object is moved to the left.

[0034] FIG. 7 shows another example of a change in the display mode. For objects "A" and "B" that are in an obstructing relationship, the HMD 1 may move both the target object and the obstructing object. Moving both objects is effective when the display has a small angle of view. In the change from (a) to (b), the target object "B" object 102 and the obstructing object "A" object 103 are moved in directions away from each other (left and right in this example). This makes the entire target visual range 107 visible.

[0035] [Display example (4)] FIG. 8 shows a method of displaying a duplicate object instead of moving an object as yet another method of changing the display mode. In the change from (a) to (b), the HMD1 displays the object 102 of "B" which is a target object partially blocked by the object 103 of "A" which is an obstructing object as is. Furthermore, the HMD1 generates a duplicate object 102r of the object 102 of "B" and displays it in an empty position (for example, a position on the left side). The HMD1 may also display information informing the user U1 that the duplicate object 102r is a duplicate along with the display of the duplicate object 102r. The HMD1 makes the entire target visual range 107r of the duplicate object 102r visible. This allows the user U1 to view the entire target visual range 107r of the duplicate object 102r at the gaze point 106r after the movement. This is equivalent to viewing the entire target visual range 107 of the original object 102. In this method, the user U1 can visually confirm the entirety of the target object using the duplicate object, and can also grasp the positional relationship between the original "B" object 102 and "C" object 103 while maintaining it as is.

[0036] As described above, the HMD1 of the first embodiment changes the display mode of at least one of the objects, such as the display position, transparency, size, or duplication, when at least a part of the target visual range of the target object is occluded by an obstructing object. Each change method can also be applied in combination. This can eliminate the visual obstruction of the target object by the obstructing object, or reduce the degree of visual obstruction. The HMD1 determines the details of the display mode change, taking into account the details of the obstruction relationship. For example, when changing the display mode of the obstructing object is not appropriate, the HMD1 changes the display mode of the target object.

[0037] [Transmission of display mode change status] When there is an occlusion obstruction relationship between the objects, the HMD1 temporarily changes the display mode of the object as in the above example. At this time, the HMD1 may output, using a GUI or the like, to inform the user U1 in an easily understandable manner that the display mode is being temporarily changed. For example, the HMD1 may display an image on the display surface indicating that the display mode is being changed. Image 130 in (B) of FIG. 3 is an example of such an image. Also, for example, when changing the display position of an object, the HMD1 may express the state of the change using animation, effects, or the like, or may display the changed object in a specific color or the like.

[0038] In addition, the HMD 1 may temporarily lock the gaze point determination process during the above-mentioned display mode change process. This makes it possible to prevent erroneous determination of the target object when the gaze point 106 moves in conjunction with a change in the object display position, such as that shown in FIG. 6.

[0039] [Processing flow] FIG. 9 shows a main process flow for explaining the basic operation of the HMD1 of the first embodiment. The flow in FIG. 9 has steps S1 to S8. In step S1, the HMD1 detects the gaze point 106 at which the user U1 is gazing in space based on the detection of the line of sight (104, 105) of both eyes of the user U1 in FIG. 1. The HMD1 determines and confirms a target object that is presumed to be a desired object that the user U1 is trying to view based on the position of the detected gaze point 106. Since the HMD1 grasps the position of each object in a three-dimensional space and the position of the gaze point 106, it can compare these positions and determine and confirm, for example, an object located closest to the position of the gaze point 106 as the target object. Note that the target object is determined using the gaze point 106 here, but a modified example will be described later.

[0040] Next, in step S2, the HMD1 selects and determines a target visual range that is presumably intended to be viewed by the user U1 for the determined target object. For example, in the example of FIG. 1 etc., the target visual range is selected as the same image area as the apparent display range of the target object (an image area with pixels along the shape). Not limited to this, the target visual range may be selected as an image area (e.g., a circumscribing rectangle or a circumscribing ellipse, etc.) that includes the target object. Alternatively, the target visual range may be a rectangle, ellipse, or other area of ​​a predetermined size centered on the gaze point.

[0041] In step S3, the HMD1 judges whether an obstructing object exists that obstructs the target visual range of the determined target object. For example, the HMD1 may judge that an obstructing object exists when a range of a predetermined percentage or more of the target visual range is obstructed by an object in front. If an obstructing object exists (Y), the process proceeds to step S4, and if not (N), the process skips step S4.

[0042] In step S4, the HMD1 changes the display mode of the object so as not to block the target visible range of the target object. As in the above example, the display mode change method can be selected from a suitable method such as display position, transparency, size, and duplication for at least one of the obstructing object and the target object. In particular, when the obstructing object is less suitable for display mode change than the target object, the HMD1 selects the method for changing the display mode of the target object.

[0043] In step S5, when the display mode is changed, the HMD1 maintains the state after the display mode change for a certain period of time. This allows the user U1 to view the entire target visible range of the target object in that state. If there is no obstructing object (S3-N), the user U1 can view the entire target visible range of the target object even if the display mode is not changed.

[0044] In step S6, the HMD1 judges whether the gaze point of the user U1 has moved outside the target visual range of the target object. If the gaze point remains within the target visual range (S6-N), the process returns to step S5. This maintains the state of the display mode change, and maintains the state in which the target visual range can be viewed.

[0045] On the other hand, if the gaze point has moved outside the target visual range (S6-Y), the process proceeds to step S7. In step S7, the HMD 1 restores the display mode change state of the target object and the obstructing virtual object in the above-mentioned obstruction relationship to the original state before the change.

[0046] After that, in step S8, the HMD 1 checks whether to continue or end the control process based on, for example, the state of gaze. If it continues (N), the process returns to step S1, and the detection of a new gaze point and the like are repeated in the same manner. If it ends (Y), this flow ends.

[0047] Based on the above processing flow, in the first embodiment, when a target object is occluded by an obstructing object, the visibility obstruction in the target visible range can be eliminated or the degree of visibility obstruction can be reduced by changing the display mode of the object.

[0048] In the above example, the display mode change state is maintained for a certain period of time depending on the state of the gaze point, but this is not limited to the above. The display mode change may be terminated when the user U1 inputs a specified operation, or when it is detected that the line of sight or gaze point has reached a specified state.

[0049] [Function block] 10 shows a functional block configuration example of the HMD1 which is the display device of the first embodiment. The configuration is basically similar for other types of display devices. In this configuration example, the components are implemented in one device, but this is not limiting, and some components may be implemented separately in different devices.

[0050] In Figure 10, the HMD1 is configured appropriately using a processor 410, a memory unit 420, a camera unit 431, a distance measurement sensor 440, a left eye gaze detection unit 432, a right eye gaze detection unit 433, a display processing unit 434, an operation input unit 435, a microphone 436, headphones 437, a vibration generating unit 438, and a communication unit 439, and each component is connected to each other via a bus 450.

[0051] The processor 410 is composed of a CPU, a ROM, a RAM, etc., and constitutes a controller of the HMD 1. The processor 410 executes processing in accordance with an operating system (OS) 422 and an application program 423 for operation control, which are stored as a control program 421 in a memory unit 420. In this way, the processor 410 controls each component unit and realizes functions such as the OS, middleware, and application, as well as other functions.

[0052] The memory unit 420 is composed of a non-volatile storage device or the like, and stores various programs 421 handled by the processor 410 or the like, and information data 424. As the information data 424, gaze point information 425 indicating the position of the gaze point gazed at by the user U1, target object information 426 indicating the shape and position of a target object visually recognized by the user U1, virtual object information 427 indicating the shape and position of a virtual object, and the like are stored.

[0053] The camera unit 431 photographs the field of view and visual field conditions around the front of the HMD 1, and acquires images by converting light incident from a lens into an electrical signal using an image sensor. In the case of an optical see-through type HMD, the user U1 visually recognizes actual objects within the field of view and visual field around the front with his or her own eyes. In the case of a video see-through type HMD, the camera unit 431 photographs actual objects within the field of view and visual field around the front, and the image of the photographed actual object is displayed on the display device of the display processing unit 434.

[0054] The distance measurement sensor 440 is a sensor that measures the distance between the HMD 1 and an external real object. The distance measurement sensor 440 may use a TOF (Time Of Flight) type sensor, a stereo camera, or other methods. The HMD 1 grasps three-dimensional arrangement information of the external real object using the distance measurement sensor 440 and the arrangement data, and displays an object that reflects the occlusion relationship between the individual real object and the virtual object. The HMD 1 may refer to the arrangement data of the external real object, including the occluded object, based on some feature points of the external real object. This arrangement data may be created or held by the HMD 1, or may be acquired from an external information server 120 or the like.

[0055] The left eye line of sight detection unit 432 and the right eye line of sight detection unit 433 detect the line of sight (104, 105) by capturing the movement and direction of the left eye and the right eye, respectively. Note that this line of sight detection processing can utilize well-known technology that is generally used as eye tracking processing. For example, as a method using corneal reflex, a technology is known in which infrared light is irradiated from an infrared LED (Light Emitting Diode) to the face, an image is taken with an infrared camera, and the position on the cornea of ​​the reflected light produced by the irradiation is used as a reference point, and the line of sight is detected based on the position of the pupil relative to the position of the corneal reflex. In addition, a method is also known in which the eye is photographed with a visible light camera, and the reference point is the inner corner of the eye, the moving point is the iris, and the line of sight is detected based on the position of the iris relative to the inner corner of the eye. The intersection of the line of sight 104 of the left eye detected by the left eye line of sight detection unit 432 and the line of sight 105 of the right eye detected by the right eye line of sight detection unit 433 is detected as the gaze point 106 at which the user U1 is gazing.

[0056] The display processing unit 434 is composed of a display device and a part that performs display processing. In the case of an optical see-through type HMD, the display processing unit 434 has, for example, a projection unit that projects light corresponding to virtual objects and notification information to the user, and a transparent half mirror that forms an image of the projected light in front of the user's eyes. In this case, the display surface 11 in FIG. 1 corresponds to the half mirror. This allows the user U1 to view both the actual object in the field of view and field of view range in front of the user's eyes and the image of the formed virtual object, etc., as if they were floating in real space. In addition, in the case of a video see-through type, the display processing unit 434 has a display device such as a liquid crystal display panel that displays the image of the actual object in front of the user's eyes (including the cut-out individual actual object) photographed by the camera unit 431 and the image of the generated virtual object, etc. together. In this case, the display surface 11 corresponds to a screen such as a liquid crystal display panel. With these, the user U1 can view the actual object in the field of view in front of the user's eyes and the virtual object, etc., in a state where they are superimposed on each other using the HMD1.

[0057] The operation input unit 435 is an input means such as a keyboard, key buttons, touch keys, etc., and allows the user U1 to set and input information that he / she wants to input. The operation input unit 435 is provided in a position and form that allows the user U1 to easily perform input operations on the HMD1. Alternatively, the operation input unit 435 may be provided in a form that is separated from the HMD1 body and connected by wire or wirelessly, like a remote controller. The HMD1 may display a graphical user interface (GUI) such as an input operation screen on the display surface 11 of the display processing unit 434, and may capture input operation information according to the position on the input operation screen to which the gaze detected by the left eye gaze detection unit 431 and the right eye gaze detection unit 432 is directed. The HMD1 may display a pointer on the input operation screen, and the user U1 may operate the pointer using the operation input unit 435 to capture the input operation information. The HMD1 may also collect the voice representing the input operation uttered by the user U1 using the microphone 436, and capture the input operation information.

[0058] The microphone 436 collects external sounds and the user's own voice. The HMD1 can take in instruction information by voice uttered by the user U1 and conveniently execute an operation in response to the instruction information. The headphones 437 are worn on the ears of the user U1 and output sound such as notification information to the user U1. The vibration generating unit 438 generates vibrations under the control of the processor 410 and converts the notification information to the user U1 transmitted by the HMD1 into vibrations. The vibration generating unit 438 can reliably convey a notification to the user U1 by, for example, generating vibrations on the head of the user U1 to which the HMD1 is closely attached. Examples of notification information to the user U1 include a notification when an obstructing object occurs, a notification informing of a change in the display mode, a notification of a method for changing the display mode, and a notification of the presence of a shared user, which will be described later. Such notifications can further improve usability.

[0059] The communication unit 439 is a part having a communication interface for performing wireless communication with other nearby HMDs, information processing terminals such as smartphones, or external devices such as the information server 120 in FIG. 1 by short-distance wireless communication, wireless LAN, or base station communication, and includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Note that, for example, short-distance wireless communication may be communication using an electronic tag, but is not limited thereto, and may be communication that allows the HMD 1 to wirelessly communicate with other nearby information processing terminals. Examples of such communication interfaces include Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or wireless LAN such as Wi-Fi (registered trademark). Note that, for base station communication, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used.

[0060] The communication unit 439 may apply other means such as optical communication or sonic communication as a wireless communication means. In that case, instead of a transmitting / receiving antenna, a light emitting / receiving unit and a sonic output / input unit are used, respectively. Also, when handling high definition video, the amount of data is dramatically large. In this case, if a high-speed, large-capacity communication network such as 5G (5th Generation: 5th generation mobile communication system) or local 5G is used for wireless communication, usability can be dramatically improved.

[0061] [Placement data] The HMD 1 of the first embodiment may acquire and use placement data (in other words, spatial data) of external entities in the outside world from an external device such as the information server 120 in FIG. 1 through communication. This placement data is data that indicates the placement (including positions, shapes, etc.) of individual entities in a three-dimensional space. For example, this placement data is data that includes various facilities and the like as individual entities in a space on a map. In addition, this placement data may have attribute information and related information (for example, the name and description of the facility, etc.) for each individual entity object. In another example, this placement data is data that includes individual entities such as walls and objects in a space of a building. When such placement data is available, it is generally easy to grasp the relationship between objects in a three-dimensional space, such as overlapping between objects. Therefore, in the HMD 1, using the placement data, it becomes easier to determine the boundaries of entities in the field of view, and it becomes easier to cut out and recognize individual entities.

[0062] [Component] The components realized based on processing by processor 410 in Figure 10 include a virtual object generation processing unit 411, a gaze point detection processing unit 412, a target object target visible range identification processing unit 413, an obstruction object discrimination processing unit 414, an object category processing unit 415, and an object display manner control processing unit 416.

[0063] The virtual object generation processing unit 411 generates a virtual object, which is an object in a virtual space different from the real space. Note that the HMD 1 may use data of a virtual object generated by an external device such as the information server 120 by wireless communication.

[0064] The gaze point detection processing unit 412 three-dimensionally calculates and detects the gaze point 106, which is the intersection of the gaze directions of both eyes in Figure 1 and the gaze point of user U1, from the left eye gaze 104 detected by the left eye gaze detection unit 432 and the right eye gaze detected by the right eye gaze detection unit 433.

[0065] The target object target visible range identification processing unit 413 determines the target object which is the object on which the gaze point is located, in other words, the object closest to the gaze point, and identifies and determines the target visible range 107 (Figure 1), which is the range in which the user U1 is presumed to intend to view the target object.

[0066] The obstructing object discrimination processing unit 414 discriminates obstructing objects that overlap with the target visible range of the target object in the depth direction as seen from the user U1 and obstruct the target visible range by obstructing the target visible range.

[0067] The object category processing unit 415 classifies objects into predetermined categories (in other words, types) according to the degree of restriction and tolerance for changes in the display mode of the object. The HMD 1 determines the method and details of changing the display mode according to the category of the object. The number and details of the categories are not limited.

[0068] The object display mode control processing unit 416 performs control processing to change the display mode of an object in an occlusion obstruction relationship. The display mode change is at least one of moving the display position, adjusting the transparency, changing the size (shrinking / enlarging), displaying a duplicate object, etc.

[0069] When an obstructing object determined by the obstructing object determination processing unit 414 obstructs at least a part of the target visible range of the target object identified by the target object target visible range identification processing unit 413, the HMD 1 controls the change of the display mode of the object in the obstructing relationship by the object display mode control processing unit 416. The object display mode control processing unit 416 changes the display mode of at least one of the obstructing object or the target object so as to eliminate or reduce the obstruction of the target object by the obstructing object. The object display mode control processing unit 416 determines the object to be changed and the display mode change method, etc., taking into account the categories, etc. of the objects before and after the obstructing object in the obstructing relationship.

[0070] For example, when the obstructing object is a virtual object (pattern 2 / pattern 4 of FIG. 2) and the obstructing object has a lower degree of restriction than the target object, the object display mode control processing unit 416 changes the display position of the obstructing object or adjusts the transparency up so as to eliminate or reduce the obstruction of the target object by the obstructing object. Also, when the obstructed target object is a virtual object (pattern 3 / pattern 4) and the target object has a lower degree of restriction than the obstructing object, the object display mode control processing unit 416 changes the display position of the target object so as to eliminate or reduce the obstruction of the target object by the obstructing object. As a result, the obstruction of the target visibility range of the target object by the obstructing object can be eliminated or the degree of the obstruction of visibility can be reduced.

[0071] [detail] The details of the processing and display in the first embodiment will be described with reference to FIG. 11 and subsequent figures. FIG. 11 shows a display example in the field of view 101 of the HMD1, and shows an example of an individual real object, a virtual object, and a target visual range. In (A) of FIG. 11, as an example of a real object, there is a landscape seen by the user U1 from a high place, for example, which includes a tower 508 and a building 500. The HMD1 recognizes, for example, the tower 508, etc., as an individual real object from this landscape. In the case of a video see-through type, the HMD1 cuts out the part of the tower 508 from the image of the landscape as an individual real object. In the case of an optical see-through type, the HMD1 recognizes the part of the tower 508 from the landscape as an individual real object. The above-mentioned arrangement data may be used when recognizing the tower 508, etc.

[0072] For example, when the HMD 1 focuses on a tower 508 that is an individual entity object, the HMD 1 generates an explanation panel 503 and a guide map 504 as examples of virtual objects related to the tower 508, and displays them as illustrated so as to be superimposed on a landscape including the tower 508. The explanation panel 503 is a virtual object that displays explanatory information about the tower 508 (e.g., its height of 634 m) as, for example, a speech bubble-shaped panel. The explanation panel 503 is disposed on the right side so that the origin of the speech bubble is in contact with the tower 508. The guide map 504 is a virtual object that provides guidance to the location of the tower 508 on the map. The guide map 504 is disposed in the upper left of the field of view 101.

[0073] The gaze points 501, 502, and 507 are examples of gaze points of the user U1 with respect to this landscape. The gaze point 507 is a case where the gaze is directed at a tower 508 that is an individual entity object. The HMD1 may display an explanatory panel 503 or the like that is a virtual object in response to the gaze on the tower 508. When the gaze point is located on an entity such as the tower 508 as the gaze point 507, the HMD1 cuts out or recognizes the part of the entity, the tower 508, from the landscape as an individual entity object based on the analysis and arrangement data. Then, the HMD1 determines the display range of the individual entity object, the tower 508, indicated by the dashed line, as a target visible range 509.

[0074] The gaze point 501 is for gazing at an explanation panel 503, and the gaze point 502 is for gazing at a guide map 504. The HMD1 sets a target visible range of a target object, with the object at which the gaze point of the user U1 is located as the target object. When the gaze point is located at a virtual object, the HMD1 determines the display range of the virtual object (corresponding image area) as the target visible range. For example, when the gaze point 501 is on the explanation panel 503, the display range indicated by the dashed line on the explanation panel 503 becomes the target visible range 505. When the gaze point 502 is on the guide map 504, the display range indicated by the dashed line on the guide map 504 becomes the target visible range 506.

[0075] In this example, each target visible range indicated by a dashed line is the same range according to the shape and area of ​​the object on the display, but this is not limiting. The target visible range may be a range larger than the object or a range smaller than the object. The target visible range may be a predefined size or shape (e.g., a rectangle or an ellipse). For example, when a building 500 is the target object, the target visible range 511 indicates a case where an ellipse roughly including the building 500 is set as the target visible range.

[0076] (B) of FIG. 11 shows another example of setting the target visual range. The HMD 1 may control to include related objects (virtual objects or individual entity objects) for an object (virtual object or individual entity object) where the gaze point is located together in one target visual range. In this example, for an individual entity object called a tower 508 where the gaze point 507 is located, the explanation panel 503 is a related virtual object that is suitable for displaying together with the tower 508. In this case, the HMD 1 sets a display range shown by a dashed line in the figure, which is a combination of the target visual range 509 of the tower 508 in (A) and the target visual range 505 of the explanation panel 503, as one target visual range 510 for the two related objects (508, 503).

[0077] FIG. 12 shows another display example. Of the objects (virtual objects and individual entity objects) overlapping in the line of sight of the user U1, the relationship between the depth direction position of the gaze point and the object position is unclear, and it may be difficult or impossible to determine the target object (object close to the gaze point, etc.) where the gaze point is located. In the example of FIG. 12, an individual entity object, which is a tower 508, and a virtual object, which is a guide map 504, overlap in the line of sight corresponding to the gaze point 507, and the guide map 504 hides a part of the tower 508. For example, the gaze point 507 is located near the middle between the position of the tower 508 and the position of the guide map 504 in the depth direction of the gaze point 507. For this reason, it is assumed that the HMD1 cannot determine which object is the target object. In this case, the HMD1 selects and confirms the target object based on the visibility value (in other words, importance) of the object for the user U1, etc. For example, HMD1 compares multiple candidate objects (508, 504), prioritizes them in terms of visibility and importance, determines the object with the highest priority as the target object, and sets the display range of the target object as the target visibility range.

[0078] In this example, the criterion for determining the priority order based on the visibility value is to prioritize individual entity objects over virtual objects. In addition, a general visibility value (e.g., the prominence of a facility on a map) is applied to compare individual entity objects. As a result, in this example, the HMD1 determines that the tower 508 has a higher priority than the guide map 504, and sets the individual entity object, the tower 508, as the target object and sets the target visibility range 509. This makes it possible to optimally select and determine the target visibility range of the target object that the user U1 wishes to view.

[0079] In FIG. 1 and the like, the gaze point 106 is information shown for explanation purposes, and is not actually displayed on the display surface 11. As a modified example, the HMD 1 may display an image such as a mark representing the gaze point on the display surface 11 in accordance with the position of the gaze point 106. The image such as the gaze point mark may be an image different from the pointer for operation, or may be an image having the same function. The pointer is, for example, information for position designation by the OS or an application. The selection of a target object may use an image such as a gaze point mark, or may use a pointer.

[0080] [Object Category] In the first embodiment, objects are classified into three categories as attributes of objects used to control the change of display mode. The three categories are shown in (C) of FIG. 2. The first category is an object that has the highest restriction on the change of display mode, and is an object that causes a sense of incongruity due to the change of display mode, or an individual entity object. Examples of objects that cause a sense of incongruity due to the change of display mode include an entity object to which a virtual object is fixed, or an entity object to which a virtual object is embedded and processed to be integrated and deformed, and the like. In the case of an optical see-through type, an entity object or an individual entity object is classified as the first category because it is difficult to change the display mode. An example of a virtual object being fixed or embedded in an individual entity object is a case in which a hole is represented as a virtual object and fixed or embedded in a part of a real wall (corresponding individual entity object) in AR (augmented reality) or a video game. This wall and hole should be treated as one unit and not separated, so they are treated as one related object and classified as the first category with the highest restriction.

[0081] The second category is an object that is subject to some degree of constraints regarding changes in display mode, but has a lower degree of restriction and a higher degree of tolerance than the first category. The second category includes, for example, virtual objects such as the explanation panel 503 (FIG. 11) that is displayed in association with the first category virtual objects and individual entity objects.

[0082] The third category is an object with even lower restriction and higher tolerance than the second category, in other words, the least restrictive of the three. The third category includes, for example, virtual objects such as a guide map 504 (FIG. 11) that has no or low restrictions on the relationship of display position, etc., with respect to real objects or other virtual objects. The third category includes independent virtual objects and objects that can be moved naturally to a display position where the user U1 can view an object that he or she wants to view.

[0083] 11(A), tower 508, which is an individual entity object, is in the first category. Explanation panel 503, which is a virtual object, explains tower 508 and has some display position constraints such as being suitable for displaying in a position close to tower 508, so it is in the second category. Guide map 504, which is a virtual object, is an object that does not look unnatural even if it is moved, so it is in the third category.

[0084] The HMD1 may perform a display mode change process according to the category classification of the object by the object category processing unit 415. For objects in an occlusion obstruction relationship, the HMD1 compares the categories of the target object and the obstruction object according to the restriction level regarding the display mode change. Based on the comparison result, the HMD1 determines the object to be changed and the method and details of the display mode change.

[0085] For example, when the target object is in a category that is not less restrictive than the obstruction object (i.e., the same or higher), the object display mode control processing unit 416 changes the display mode of the obstruction object. On the other hand, when the target object is in a category that is less restrictive than the obstruction object, the object display mode control processing unit 416 changes the display mode of the target object. As a result, the HMD 1 can eliminate or reduce the obstruction of visibility of the target visible range of the target object in an optimal manner according to the restriction level for each object. Furthermore, the HMD 1 can minimize the discomfort in visibility caused by the change in the display mode of both the target object and the obstruction object.

[0086] In the above method and criteria, when the restriction level of the target object and the restriction level of the obstruction object are the same, the display mode of the obstruction object is changed. Not limited to this, other methods and criteria may be used to change the display mode of the target object when the restriction level of the target object and the restriction level of the obstruction object are the same. In this case, the method gives priority to maintaining the display mode of the obstruction object in front, which is closer to the user U1.

[0087] [First pattern] Here, a processing example of a case where an entity object occludes an entity object (first pattern in FIG. 2) will be described. First, when there is no appearance information of the occluded part of the entity object on the rear side, the HMD 1 processes it as no occlusion. In this case, since no occlusion obstruction relationship occurs, no change in the display mode occurs. In terms of the flow in FIG. 9, as an exception processing, in step S3, it is treated as no occlusion (N). Also, when there is appearance information of the occluded part of the entity object on the rear side, for example, when the appearance information is obtained from the above-mentioned arrangement data, the HMD 1 processes it as occlusion exists. In terms of the flow in FIG. 9, it is treated as occlusion exists (Y) in step S3. That is, the HMD 1 sets the individual entity object corresponding to the part of the entity object on the rear side as a target object. In this case, the HMD 1 generates a duplicate object by duplicating the appearance of the individual entity object that is the target object, as in FIG. 8, as a display mode change, and displays the duplicate object in an empty position. This allows the user U1 to see the part of the actual object that is occluded by looking at the duplicate object.

[0088] When the HMD1 prioritizes displaying the occluded target object, it may use a method of displaying a duplicate object at the occluded position. In this case, the duplicate object is displayed superimposed in front of the actual object that is the occluding obstructing object. This is the same as the method of increasing the transparency of the occluding obstructing object (Figure 3).

[0089] On the other hand, when HMD1 prioritizes maintaining the display of the obstructing object in front closer to user U1, it cannot move the individual real object, which is the target object in the rear, so it uses a method of displaying a duplicate object in another vacant position (Figure 8).

[0090] In either case, when an entity object is displayed as a video image in a video see-through type, the HMD1 may change the display mode of the individual entity object by processing the individual entity object cut out from the video image and treating it as a virtual object.

[0091] [Example of operation (1)] An operation example of the HMD1 of the first embodiment will be described with reference to FIG. 13 and the like. FIG. 13 shows a processing flow for the operation example of FIG. 11 and the like. FIG. 13 shows a more detailed processing example than FIG. 9, and includes steps S601 to S613. FIG. 13 shows the details of steps S2 and S4 in FIG. 9 in particular. In step S601, the HMD1 detects the gaze point of the user U1 by the attention point detection processing unit 412, and judges whether there is an object located at the gaze point. If there is an object located at the gaze point, in other words, if one object within a predetermined distance range is determined (Y), in step S602, the HMD1 determines that object as a target object.

[0092] Next, the target object target visible range identification processing unit 413 of step S2 selects the target visible range of the target object. This process is composed of steps S603 to S609. In step S603, the HMD 1 judges whether there is an object overlapping in the line of sight of the gaze point, and if there is not (N), proceeds to step S604, and if there is (Y), proceeds to step S609. In step S604, the HMD 1 regards the object overlapping in the line of sight of the gaze point as a target object, and judges whether the gazed target object is an actual object (corresponding individual actual object) or a virtual object. If the target object is an actual object (A), proceeds to step S605, and if the target object is a virtual object (B), proceeds to step S606.

[0093] In step S605, the HMD 1 identifies and selects an individual real object, which is individually cut out or recognized from the real object, as a target visible range of the target object. In step S606, the HMD 1 identifies and selects a display range of the virtual object as a target visible range of the target object.

[0094] Here, in step S607, the HMD1 determines whether there is an object related to the target object, which is a real object (S605) or a virtual object (S606). As described above, the related object is a virtual object or the like whose display position should be linked. If there is a related object (Y), in step S608, the HMD1 identifies and selects the target object and its related object as the target visible range of one target object (FIG. 11B).

[0095] Meanwhile, in step S609, the HMD1 selects one object based on a predetermined criterion from among the multiple objects overlapping in the line of sight of the fixation point, sets it as a target object, and identifies and selects the target visible range of the target object. At this time, in the first embodiment, the above-mentioned visibility value and importance are used. The HMD1 sets the object with the highest visibility value and importance from among the multiple overlapping objects as the target object, and identifies and selects the display range of the target object as the target visible range. As a result of the above step S2, the target visible range of the target object is determined.

[0096] After that, in step S3, the HMD 1 determines whether or not there is a virtual object (sometimes referred to as an "obstructing virtual object") that is an obstructing object blocking the target visible range of the target object, using the obstructing object determination processing unit 414. If there is an obstructing virtual object (Y), the process proceeds to step S4, and if there is no obstructing virtual object (N), the process skips step S4. In the first embodiment, if there is a virtual object that blocks at least a part of the target visible range, the HMD 1 determines it as an obstructing virtual object and proceeds to step S4.

[0097] Step S4 includes steps S611 to S613. In step S611, the HMD 1 determines whether the target object has a higher restriction level than the obstructing virtual object, that is, whether the target object has a higher category than the obstructing virtual object, by the object category processing unit 415. For example, if the target object is in the first category and the obstructing virtual object is in the second category, the former is in the higher category. If the target object is in a higher category than the obstructing virtual object (Y), the process proceeds to step S612, and if not (N), the process proceeds to step S613. In step S612, the HMD 1 performs the above-mentioned display position movement or transparency adjustment as a display mode change of the obstructing virtual object by the object display mode control processing unit 416. In step S613, the HMD 1 performs the above-mentioned display position movement or transparency adjustment as a display mode change of the target object by the object display mode control processing unit 416. As a result, the entire target visual recognition range is made visible. Then, the process proceeds to the above-mentioned step S5.

[0098] [Example (2)] 14 shows an example of operation in the case of the second pattern. In the state before the change in (a), the target object is a tower 508 which is an individual real object of the first category, and the obstructing object is a guide map 504 which is a virtual object of the third category, in terms of the obstruction relationship. The target visible range 509 of the tower 508 where the gaze point 507 is located is partially obstructed by the guide map 504. In this case, the HMD 1 adjusts the transparency up, for example, with the guide map 504 which has a lower restriction and is in a lower category as the change target. As a result, in the state after the change in (b), the guide map 504 becomes transparent, and the entire target visible range 509 of the tower 508 which is the target object can be seen.

[0099] Fig. 15 shows another operation example in which the display position is moved as a display mode change. In the same case as Fig. 14(a), the HMD 1 moves the display position of the guide map 504, which is in a lower category, to a position outside the target visible range 509 of the tower 508. This allows the entire target visible range 509 of the tower 508, which is the target object, to be visible without any obstruction.

[0100] [Example (3)] Fig. 16 shows an example of operation in the case of the third pattern. Fig. 16 shows a case opposite to Fig. 14 etc., where the target object is a guide map 504 which is a virtual object of the third category, and the obstructing object is a tower 508 which is an individual real object of the first category. In (a), the target visible range 506 of the guide map 504 where the gaze point 502 is located is partially blocked by the tower 508. In this case, as in (b), the HMD1 moves the guide map 504 of the lower category to a position outside the tower 508 so that the tower 508 does not overlap within the target visible range 506. This makes it possible to view the entire target visible range 506 of the guide map 504 which is the target object, without any obstruction.

[0101] [Example (4)] 17 shows an example of operation in the case of the fourth pattern. The target object is an explanatory panel 503, which is a virtual object of the second category, and the obstructing object is a guide map 504, which is a virtual object of the third category. In (a), the target visible range 505 of the explanatory panel 503 is partially blocked by the guide map 504. In this case, the HMD 1 adjusts the transparency of the guide map 504, which is of a lower category, to be increased, as shown in (b). This makes the guide map 504 transparent, and the entire target visible range 505 of the explanatory panel 503, which is the target object, can be seen.

[0102] Fig. 18 shows another example of the operation in the case of moving the display position. In the same case as Fig. 17(a), the HMD 1 moves the display position of the guide map 504 of the lower category to a position outside the target visible range 505. This makes it possible to view the entire target visible range 505 of the explanation panel 503, which is the target object, without any obstruction.

[0103] [Example (5)] Fig. 19 shows another operation example. Fig. 19 shows a case opposite to that of Fig. 17, in which the target object is a guide map 504 that is a virtual object of the third category, and the obstructing object is an explanation panel 503 that is a virtual object of the second category. In (a), the target visible range 506 of the guide map 504 is partially obstructed by the explanation panel 503. In this case, the HMD1 moves the guide map 504, which is a lower category, to a position where the explanation panel 503 and other objects do not overlap within the target visible range 506. This allows the entire target visible range 506 of the guide map 504, which is the target object, to be visible without any obstruction.

[0104] When the target object is in the first category and the obstruction object is in the second category, the same control as when the target object is in the first category and the obstruction object is in the third category can be applied.

[0105] [Effects etc. (1)] As described above, according to the first embodiment, in the HMD1 capable of displaying virtual objects arranged in a three-dimensional manner, when there is a visual obstruction due to occlusion by other objects in the visual range of an object such as a real object or a virtual object that the user U1 wants to view, the visual obstruction can be eliminated or reduced by changing the display mode, and the user U1 can preferably view the entirety of the object. Moreover, such a function can be realized with less user effort and ease of use. According to the first embodiment, even if there is an obstruction relationship between objects, the user can preferably view the entirety of the target visual range of the target object that the user wants to gaze at. According to the first embodiment, the display mode can be automatically changed according to the obstruction relationship to support the user's viewing, so that such a function can be realized with less user effort and ease of use.

[0106] In Patent Document 1, when there is an object that obstructs the visibility of the background in the line of sight, the display mode of the object is changed. In contrast, in the first embodiment, when there is an obstructing object that obstructs the visibility of the target visible range of a target object due to a shielding relationship between objects arranged in three dimensions, the display mode of the obstructing object or the target object is changed so that the entire target visible range can be viewed.

[0107] [Variations] The following modification of the first embodiment is also possible. When determining a target object based on the line of sight, the HMD1 may determine that there is a target object that the user U1 wants to gaze at if the movement in the line of sight direction is equal to or smaller than a predetermined threshold. This makes it possible to eliminate erroneous processing due to unintended rapid eye movement and to more accurately identify the target object. An example of erroneous processing is when the gaze point is positioned on an object for a short period of time, and the object is mistakenly determined to be the target object.

[0108] Furthermore, when setting the target visual range, the HMD 1 may determine the size or area of ​​the image region of the object and set an upper limit. When the target visual range is larger or wider than a predetermined threshold, the HMD 1 may set an upper limit range corresponding to the predetermined threshold as the target visual range. For example, when the target object is too large in the field of view, it is difficult to display the obstructing object when changing the display mode to move it outside the target visual range. In such a case, setting an upper limit for the target visual range is effective.

[0109] <Embodiment 2> The second embodiment will be described with reference to FIG. 20 and the like. The second embodiment has the following additional functions in comparison with the first embodiment. In the field of view, there may be cases where an object that is a candidate for a target object (sometimes referred to as a target candidate object) is occluded by another object, that is, a virtual object or an individual entity object, and the presence of the object is not known to the user. This function is a function that can confirm the presence of the target candidate object in such cases.

[0110] [Check for the presence of target candidate objects (1)] FIG. 20 is an explanatory diagram of an example of an operation in the second embodiment. FIG. 20 shows an example of a display mode change when an object (target candidate object) is occluded by a real object. (a) shows a state before the change. In the field of view 101, a tower 508 which is an individual real object and an explanation panel 1213 which is a virtual object are displayed. In addition, as objects located in the direction of the gaze point 1201, there are the tower 508 and a guide map 1202 (shown by a dotted line) which is a virtual object that is hidden and cannot be seen because it is occluded by the tower 508. That is, there is the guide map 1202 as an invisible target candidate object. In this case, the HMD1 changes the display mode of the object (508, 1202) in the line of sight direction as shown in (b). In this example, the HMD1 moves the display position of the guide map 1202 so that at least a part of the guide map 1202 which is a target candidate object is visible out of the occlusion by the tower 508. In (b), at least a portion of the changed guide map 1203 is visible to the user U1. The HMD1 may change the display range corresponding to the target candidate object so that the entire display range is visible. The HMD1 may also change the display range so that a predetermined percentage of the display range corresponding to the target candidate object is visible. This allows the user U1 to recognize and confirm the existence of the guide map 1202. This allows the user U1 to select the guide map 1202 as a target object using the gaze point (the gaze point after movement 1201b).

[0111] FIG. 21 shows another display example. In the state before the change in (a), objects located in the direction of the gaze point 1211 include an explanatory panel 1213, which is a virtual object related to the tower 508, and a guide map 1212, which is a virtual object hidden by the explanatory panel 1213 and cannot be seen. That is, the guide map 1212 is an invisible target candidate object. In this case, the HMD1 changes the display mode of the objects (1212, 1213) in the line of sight. In this example, the HMD1 performs, for example, an increase in transparency as a display mode change of the explanatory panel 1213 in front that is shielding the guide map 1212, which is a target candidate object, as shown in (b). As a result, the guide map 1212 behind the explanatory panel 1213 becomes visible through the explanatory panel 1213. As a result, the user U1 can confirm the existence of the guide map 1212, which is a target candidate object, and can select it as a target object using the gaze point 1211.

[0112] FIG. 22 is a supplementary explanatory diagram for a case where a plurality of objects overlap in the line of sight direction, as in the example of FIG. 21, etc., making it difficult to determine the target object where the gaze point is located. FIG. 22 shows a schematic diagram of overlapping objects in the depth direction (Z direction) when the user U1 views the field of view 101. An explanation panel 1233 is arranged in front of the user U1's viewpoint, and a guide map 1232 is arranged in the rear. The guide map 1232 corresponds to the guide map 1212, which is a target candidate object in FIG. 21. The explanation panel 1233 corresponds to an obstructing object in terms of the obstruction obstruction relationship. In FIG. 22, the guide map 1212, which is a target candidate object where the gaze point 1231 is located, is obstructed by the explanation panel 1233, which is an unfocused obstructing virtual object, and is hidden and cannot be seen by the user U1, as in FIG. 21. An unfocused object is an object where the gaze point is not located, or an object that is far away from the gaze point.

[0113] The fixation point 1231 is an example of a fixation point calculated from the line of sight (104, 105) of both eyes. In this example, the fixation point 1231 is located near the guide map 1232 in the depth direction (Z direction). Of the distances from the fixation point 1231 to objects, the distance to the guide map 1232 is the smallest and is within a predetermined distance range. Therefore, the guide map 1232 becomes a target candidate object.

[0114] In this case, the HMD1 detects that the explanation panel 1233 is not in focus based on the direction of the line of sight and the gaze point. In order to confirm the presence of the target candidate object, the HMD1 changes the display mode of the out-of-focus explanation panel 1233, for example by adjusting the transparency up (similar to FIG. 21). As a result, the explanation panel 1233b after the change becomes transparent (high transparency state), and the guide map 1232 behind it becomes visible, allowing the user U1 to confirm the presence of the guide map 1233, which is the target candidate object.

[0115] [Check for the presence of target candidate objects (2)] Fig. 23 shows a process flow relating to the function of confirming the presence of the target candidate object of the HMD 1 in the embodiment 2. The flow in Fig. 23 has steps S1100 to S1104 as a difference from the above-mentioned flow. This part is performed as a pre-processing for step S2 in Fig. 9.

[0116] In step S1100, the HMD 1 checks whether the mode corresponding to this function is on (enabled), and if it is on, performs the following process. The mode can be set or instructed by the user U1 via the operation input unit 435, for example.

[0117] In step S1101, the HMD1 judges whether there is a target candidate object, which is an object whose presence is unknown to the user U1 because it is occluded by an object (virtual object or individual entity object) on the entire display surface 11. This target candidate object is an object that is invisible to the user U1 and therefore cannot be recognized, and cannot be selected by the gaze point. If there is such a target candidate object (Y), the process proceeds to step S1102, and if there is not, the process proceeds to step S2.

[0118] In step S1102, the HMD1 checks and waits for a trigger to perform the presence confirmation process. This trigger is a trigger that allows the user U1 to instruct whether to perform the presence confirmation process. This trigger can be, for example, when an instruction input is received through the operation input unit 435 or the microphone 436, or when the detected line of sight of the user U1 moves to the vicinity of a target candidate object. For example, as shown in FIG. 20, the HMD1 may display a guide or a button in the field of view 101 such as "There is a hidden object. Would you like to display it and check it?", and the pressing of the button may be used as a trigger. Also, it is possible to omit step S1102 for inputting the trigger and automatically perform the presence confirmation process of the object.

[0119] When the above-mentioned trigger occurs, in step S1103, the HMD1 changes the display mode of the target candidate object (moving the display position or displaying a duplicate, etc.) or changes the display mode of the obstructing object (adjusting the transparency up, etc.) by the object display mode control processing unit 416. In step S1104, the HMD1 maintains the state after the display mode change for a certain period of time. This allows the user U1 to confirm the existence of the target candidate object. The user U1 can easily confirm the existence of the target candidate object without the need for any special operation. After step S1104, the process proceeds to the above-mentioned step S2.

[0120] As described above, in the function of the second embodiment, when there is an object whose existence is unknown due to being occluded by an object, the display mode of at least one of the target candidate object and the obstructing object is changed so that at least a part of the target candidate object can be seen. This allows the user U1 to reliably confirm the target candidate object and select it as a target object. Note that when there is an individual entity object that is hidden and cannot be seen, and there is appearance information of the individual entity object, the individual entity object may also be treated as a target candidate object and its existence may be confirmed in the same manner.

[0121] [Effects etc. (2)] As described above, according to the second embodiment, even if there is a hidden target candidate object, the presence can be confirmed by a kind of display mode change, and the user U1 can select it as a target object. If the HMD1 has a function to detect only one line of sight of the user U1, it is difficult to determine the gaze point in the depth direction. In this case, in the second embodiment, the display mode is changed so that the hidden object on the display surface is visible, that is, by making the display surface into a state where there is only one object in the depth direction, the object in the line of sight can be determined as the target object even if there is only one line of sight.

[0122] [Object data and attribute information] In the above, a case has been described in which the HMD 1 determines the control content with reference to parameters such as the category related to the degree of restriction (in other words, tolerance) and the visibility value (in other words, importance) regarding the change in the display mode of each object. The degree of restriction and the tolerance in the above category are one of the attribute information representing the degree of restriction and tolerance regarding the change in the display mode for each object. In the data for each object, such a degree of restriction or tolerance, category, or other information may be set as one of the attribute information. In addition, as an example of other information of the attribute information for each object, the visibility value, importance, or priority of each object may be set. These parameters may be managed and held as data by the HMD 1 or an external device.

[0123] Fig. 24 shows an example of object data managed and held by the HMD 1. This object data is management data including attribute information for each object. This object data may be management information that differs for each application, or management information or user setting information that differs for each user. The HMD 1 may generate and set each piece of information for the object data by itself, or may refer to the information from an external device (for example, the information server 120 in Fig. 1).

[0124] The object data table in FIG. 24 has columns of ID, object, type, category, visibility value, related object, and shared user. "ID" is an identifier for each "object." Examples of "object" are the tower and explanation panel described above. "Type" here has two types: A. individual real object and B. virtual object. As described above, the category (corresponding restriction level) has three levels: 1 (high), 2 (medium), and 3 (low). The visibility value (corresponding importance) has three levels: 1 (high), 2 (medium), and 3 (low). "Related object" indicates the relationship with other objects. For example, an object with ID=1 has an object with ID=2 as a related object, and an object with ID=2 has an object with ID=1 as a related object. "Shared user" indicates the identifier of a shared user when the object is shared by multiple users as shared users, as will be described later.

[0125] In this example, the smaller the number for "visibility value", the higher it is. The "category" and "visibility value" may be set by HMD1 or by the user. For example, HMD1 may set the "visibility value" based on the general notability of the object. Alternatively, HMD1 may set the "visibility value" according to the user U1's level of interest in the object. For example, in FIG. 11 and the like mentioned above, a "visibility value" is set for each individual entity object such as a facility on a map based on general notability. The "category" may be determined by combining the "restriction level" and "visibility value".

[0126] Furthermore, during control processing, the HMD 1 processes and stores object information appropriately at each time point, separate from the object data. This object information includes, for each object, information on the display position on the display surface 11, the orientation of the three-dimensional arrangement of the three-dimensional object, the display range (image area), the target visual range, and the display mode change state. The display mode change state includes the presence or absence of change, the change method, and the like. The HMD 1 controls the object display using the above-mentioned object data, object information, line of sight, gaze point, and other information.

[0127] <Embodiment 3> The third embodiment will be described with reference to Fig. 25 etc. The third embodiment has a function of changing the display mode of a shared object between shared users.

[0128] [Shared Users and Objects] FIG. 25 is an explanatory diagram of an operation example of the HMD1 of the third embodiment. In FIG. 25, there are a plurality of users (for example, two) (U1, U2) each wearing an HMD1. The first user U1 uses an HMD1A, and the second user U2 uses an HMD1B. These users (U1, U2) share the virtual objects "A" object 103 and "B" object 102. The users U1 and U2 are shared users who share these virtual objects. The "A" and "B" objects are shared objects shared by the shared users (U1, U2), respectively. The HMD1 (1A.1B) of the shared users (U1, U2) perform communication 2500 for sharing by the above-mentioned short-range wireless communication.

[0129] FIG. 25 shows a first example of the state of display, visibility, etc. Shared users (U1, U2) are looking at shared objects (A, B). In particular, at a certain point in time, the first user U1 is looking at the object 103 of "A" at the gaze point P1 of the line of sight E1 in the field of view 101A as a target object. The HMD1A sets a target visual range 107A of the object 103 of "A". Also, the second user U2 is looking at the object 102 of "B" at the gaze point P2 of the line of sight E2 in the field of view 101B as a target object. The HMD1B sets a target visual range 107B of the object 102 of "B". In the first example, there is an obstruction relationship as seen from the second user U2, and the object 103 of "B" on the rear side viewed by the second user U2 is the target object, and the object 102 of "A" on the front side viewed by the first user U1 is the obstruction object.

[0130] Note that the lines of sight E1 and E2 indicate the lines of sight (104, 105) of both eyes combined into one line. In Fig. 25, the display contents of the visual field 101A seen by the user U1 and the display contents of the visual field 101B seen by the user U2 are illustrated as being the same, but since the viewpoint positions of the users are different, the actual display contents, i.e., the way the objects are seen, are different.

[0131] The next Fig. 26 shows a second example. Fig. 26 shows a case where the first user U1 is looking at the "B" object 102 on the rear side, and the second user U2 is looking at the "A" object 103 on the front side, which is the opposite of the first example in Fig. 25. In the second example, there is an obstruction relationship as seen from the first user U1, and the "B" object 103 on the rear side viewed by the first user U1 is the target object, and the "A" object 102 on the front side viewed by the second user U2 is the obstruction object.

[0132] [Display example (3-1)] In the above case, the HMD1 changes the display mode of the shared object of the shared user. First, FIG. 27 shows a change example corresponding to the first example of FIG. 25. (a) shows a state before the display mode is changed as the image in the field of view 101A is seen from the first user U1, and (c) shows the state after the change. (b) shows a state before the display mode is changed as the image in the field of view 101B is seen from the second user U2, and (d) shows the state after the change. In (a), the HMD1A of the first user U1 generates and displays mark information m2 in the field of view 101A based on communication 2500 with the HMD1B of the second user U2. This mark information m2 is an image indicating which object the second user U2 is looking at, that is, which is the target object where the gaze point P2 of the second user U2 is located. For example, the HMD1B of the second user U2 transmits information to the HMD1A indicating that the target object is "B," and the HMD1A of the first user U1 transmits information to the HMD1B indicating that the target object is "A." In response to the information from the HMD1B, the HMD1A generates a mark, for example, the number "2" representing the second user U2 as the mark information m2, and displays it near the object "B," which is the target object of the second user U2. This allows the first user U1 to recognize which shared object the second user U2, who is a shared user, is looking at.

[0133] On the other hand, in (b), similarly, the HMD1B of the second user U2 generates and displays mark information m1 representing which object the first user U1 is looking at in the field of view 101B based on communication 2500 with the HMD1A of the first user U1. In response to the information from the HMD1A, the HMD1B generates a mark with the number "1" representing the first user U1 as the mark information m1, for example, and displays it near the object "A" which is the target object of the first user U1. This allows the second user U2 to recognize which shared object the first user U1 is looking at.

[0134] Furthermore, the HMD1 (1A, 1B) of each shared user (U1, U2) may change the display mode of the object according to the visibility relationship of the shared object between the shared users and the occlusion obstruction relationship as described above. Examples are shown in (c) and (d). First, an example of the change from (a) to (c) is as follows. In the state of (a), the object "B" seen by the second user U2 is occluded behind the target visibility range 107A of the target object "A" seen by the first user U1. Since the first user U1 can view the target object "A" without any occlusion obstruction, the HMD1A leaves the display of the object "A" as it is. The object "B" seen by the second user U2 may be displayed as it is, but it is partially occluded from the view of the first user U1. Therefore, in this example, the HMD1A changes the display mode of the target object "B" of the shared user so that the part where the second user U2 is looking at the object "B" is easily visible to the first user U1. (c) shows an example in which the display position of the "B" object is moved so that the entire object is visible. This change in display mode may be made in response to a predetermined input operation by the first user U1, rather than automatically. For example, the HMD1A may display "Do you want to check the object that the shared user (2) is looking at" on the display surface 11, and may make a change to (c) in response to a button press operation by the user U1.

[0135] Next, an example of a change from (b) to (d) is as follows. In the state (b), part of the target object "B" seen by the second user U2 is obstructed by the object "A" seen by the first user U1. Therefore, in this example, the HMD1B changes the display mode of the target object "A" of the first user U1, which is an obstructing object, so that the entirety of the target object "B" becomes visible. (d) shows an example of moving the display position of the object "A". This allows the second user U2 to confirm the target object "B".

[0136] As described above, various methods such as increasing transparency (FIG. 3) and duplicate display (FIG. 8) can be applied to change the display mode of a shared object. (e) shows, as another display example, a state after another display mode change from the state seen by the first user U1 in (a). In the case of (a), the HMD1A changes the state so that the entirety of the "B" object is visible by increasing the transparency of the "A" object, as in (e). This change may be made in response to a predetermined operation, as in the case of (c). This allows the first user U1 to check not only the "A" target object, but also the "B" target object of the second user U2.

[0137] (f) shows another display example after changing the display mode from the state seen by the second user U2 in (b). In the case of (b), the HMD1B changes the display mode so that the whole of the target object "B" can be seen by increasing the transparency of the object "A" as in (f).

[0138] As described above, in the third embodiment, when the visual recognition states of gaze on a shared object are different among the shared users, mark information representing the visual recognition states is displayed. This allows the shared users to confirm the shared object that each shared user is gazing at and viewing. This facilitates communication, work, and the like between the shared users. In the second example of FIG. 26, the same control as in the first example can be applied.

[0139] [Display example (3-2)] FIG. 28 shows another display example. (a) shows a state in which the first user U1 is looking at the object "A" in front, similar to the first example of FIG. 25 and (a) of FIG. 27. (b) shows a state in which the second user U2 is looking at the object "B" in the back, similar to (b) of FIG. 27. In the case of (a), the HMD1A changes the display mode as shown in (c). The HMD1A leaves the display of the target object "A" of the first user U1 and the target object "B" of the second user U2 unchanged, and generates a duplicate object 102r of the object "B" in an unobstructed position so that the entirety of the obscured object "B" can be confirmed. It is more preferable to determine the position at which the duplicate object 102r is displayed so as to correspond to the direction in which the second user U2 is located (the right side in this example). The duplicate object 102r may also be displayed with mark information m2 indicating a shared user gaze object. This allows the first user U1 to check not only the target object "A" but also the entire target object "B" of the second user U2.

[0140] Similarly, in the case of (b), the HMD1B changes the display mode as shown in (d). The HMD1B generates a duplicate object 102r (which looks different from the duplicate object 102r in (c)) for the partially occluded target object "B" and displays it in an empty position. Alternatively, as another method, the HMD1B may leave the target object "B" as it is and change the display position of the object "A", which is an obstructing object, in the same manner as described above.

[0141] (e) and (f) are other display examples. (e) is the state as seen by the first user U1. When viewed from the first user U1, the object "B" viewed by the second user U2 is partially occluded. The HMD1A changes the display mode of the shared user's object "B" in the same manner as above. Meanwhile, (f) is the state as seen by the second user U2. When viewed from the second user U2, the target object "B" is not occluded by the object "A" and can be seen in its entirety. Therefore, the HMD1B does not change the display mode.

[0142] [Display example (3-3)] FIG. 29 shows, as another display example, an example of a change in display mode when shared users (U1, U2) are looking at the same shared object (for example, object 102 of "B"). (a) is a state seen from the first user U1, and (c) is a state seen from the second user U2. In (a), the first user U1 is looking at the object "B" on the rear side from the right side of the object "A". The target object of "B" is partially hidden by the object "A". In (c), the second user U2 is looking at the object "B" on the rear side from the left side of the object "A". The target object of "B" is partially hidden by the object "A". In the state of (a), the HMD1A displays mark information m2 on the object "B" indicating that the second user U2 is also looking at it. In the state of (c), the HMD1B displays mark information m1 on the object "B" indicating that the first user U1 is also looking at it. (b) and (d) are examples after the change in each display mode. 10B is an example of changing the display position of the obstructing object "A" to, for example, a position on the left side, and 10D is an example of changing the display position of the obstructing object "A" to, for example, a position on the right side.

[0143] As described above, in the third embodiment, when each shared user is in a visual state in which the shared users are gazing at the same shared object, mark information representing the visual state is displayed. This allows the shared users to confirm the shared object that each shared user is gazing at and visually recognizing.

[0144] In the above example, the occlusion obstruction relationship is the fourth pattern described above (Fig. 2), but this is not limiting and similar control is possible for other patterns. The above mark information can also be displayed when an individual entity object is a shared object. The display mode can be changed for objects other than optical see-through entity objects. The above-mentioned restriction degree and visibility value can also be applied to shared objects.

[0145] [Effects etc. (3)] As described above, according to the third embodiment, the display mode change is performed on the shared object of the shared users in a manner appropriate for each HMD1 of each user. As a result, each user can reliably view the shared object without any confusion in the view while eliminating or reducing the visibility obstruction caused by the obstruction between the objects. In the third embodiment, at least one HMD1 between the shared users displays a mark representing the shared user's gaze object and changes the display mode according to the relationship between visibility and obstruction. When changing the display mode, the method and details are determined taking into consideration not only the obstruction relationship, the restriction degree, and the visibility value described above, but also the visibility relationship of which shared object the shared user is viewing. For example, in the example of FIG. 27, when considering the HMD1A side of the first user U1, in the relationship between the target object "A" in front of the first user U1 and the object "B" in the back of the second user U2, the display of the entire target object "A" should be given first priority. In the state of (a), the entire object can be seen, so in the case of the above-mentioned embodiment 1, there is no need to change the display mode. The object "B" is not an obstructing object, but is a target object that the shared user is looking at. Therefore, in the case of embodiment 3, the display mode can be changed so that the entire object "B" can also be confirmed. When making the change, for example, a method or details are selected so that the entire objects of both objects "A" and "B" can be seen. For example, in the case of the transparency adjustment method of (e), a part of the target object "A" temporarily becomes transparent and becomes slightly difficult to see, so that a more suitable method such as the display position movement method of (c) or the duplicate display method of (c) in FIG. 28 may be selected.

[0146] [Variation (3-1)] In the third embodiment, the mark information representing the shared user's gaze object is separate from the gaze point. The mark information is displayed in an area of ​​the target visual range of the object gazed upon by the shared user other than an area blocked by other objects. If the mark information is displayed in an blocked area, it becomes unclear which object is being gazed upon, the front or rear object, so this display makes it clear. Furthermore, the HMD1 of the shared users may constantly communicate with each other to update the display state (including the mark information) in approximately real time, or may communicate periodically to update the display state periodically.

[0147] As a modified example, the display position of the mark information representing the shared user gaze object may be set to a position aligned with the gaze point. As another modified example, a mark representing the gaze point may be displayed at a position corresponding to the gaze point in the field of view. A pointer for a selection operation by a remote controller or the like may be displayed in the field of view. FIG. 30 shows an example of a modified example in which a mark 3001 representing a gaze point and a pointer 3003 are displayed in the field of view 101 in addition to the mark representing the shared user gaze object. For example, a diamond-shaped mark 3001 is displayed at the position of the gaze point P1 of the first user U1. For example, a triangular mark 3002 is displayed at the position of the gaze point P2 of the second user U2. Also, for example, a cross-shaped pointer 3003 is displayed as a pointer for operation by the first user U1.

[0148] [Variation (3-2)] FIG. 31 shows a display example in another modified example. (a) is a state seen from the first user U1, and is the same as (a) of FIG. 27 described above. The first user U1 is looking at the target object "A" in the front. (b) is a state seen from the second user U2. The second user U2 is looking at the object "B" in the back from a line of sight different from the line of sight of the first user U1, for example, from a direction different by 90 degrees. Here, the shape and location of the shared object "B" as seen from the first user U1 are different from those as seen from the second user U2. In (b), the sides of the objects "A" and "B" as seen from the second user U2 are illustrated as "A#" and "B#".

[0149] (c) and (d) show the state after the display mode is changed. In (c), the HMD1A displays the target object "A" as it is, and changes the display mode of the partially occluded object "B" seen by the shared user in order to confirm it. At this time, the HMD1A changes the display mode of the object "B" so that it is displayed in a shape and location as seen by the second user U2 as shown in (b). In this example, the HMD1A leaves the object "B" as it is, and generates and displays a duplicate object 3101 of the object "B" with a speech bubble in an empty position. This duplicate object 3101 is generated as a duplicate object that looks the same as the object 3102 in (b). This allows the first user U1 to confirm the entire shared object "B", especially the state as seen by the second user U2. The state as seen by the second user U2 in (b) may be displayed as it is since there is no occlusion obstruction relationship, or may be displayed as follows. As shown in (d), the HMD1B changes the display mode of the object "A" that the first user U1 is looking at, so that the shape and location that the first user U1 is looking at are visible, in the same manner as described above. In this example, the object "A" 3103 as seen in (a) is generated and displayed by superimposing it in front of the object "A."

[0150] <Fourth embodiment> The fourth embodiment will be described with reference to FIG. 32 etc. In the fourth embodiment, a modified example of the target object determination method is shown. In the above-mentioned embodiment, the target object is judged and determined by detecting the gaze point from the user's line of sight. In the modified example, a selection input operation from the user for a tag displayed attached to each object is accepted. In this way, the HMD determines the target object.

[0151] Fig. 32 shows a display example in the fourth embodiment. As described above, a tower 508 which is an individual entity object, and an explanatory panel 503 and a guide map 504 which are virtual objects are displayed in the field of view 101 of the HMD 1 of the user U1. The HMD 1 displays each object in the field of view 101 with a tag. This tag is an image for identifying and selecting an object. In this example, the tags (701, 702, 703) have a rectangle connected by a line leading from the object and a number for identifying the object.

[0152] The user U1 performs an object selection input operation using a predetermined operation means provided in the HMD1. The predetermined operation means can be, for example, voice input, but is not limited to this, and various means can be applied, such as a pointer using a remote controller, detection of a gaze point using line of sight, and recognition of hand gestures. In the case of object selection using voice input, for example, it is as follows. For example, when the user U1 wants to select the tower 508 as an object, he / she inputs the number ("3") of the tag 703 attached to that object by voice. The HMD1 recognizes the number of the input voice and grasps the object associated with the tag with that number.

[0153] For example, in a situation where multiple objects are crowded together, it may be difficult to determine the target object using only the gaze point. In such a case, the accuracy of determining the target object can be increased by also using the tag selection input reception in embodiment 4. In addition, by using the tag selection method in embodiment 4, the above-mentioned display mode change function can be applied even to a device that does not have a gaze point detection function as the HMD 1.

[0154] In the state before the change in FIG. 32(a), the guide map 504 overlaps in front of the tower 508. The gaze point 507 of the user U1 overlaps the tower 508 and the guide map 504, and it may be difficult to determine the target object. Even in this case, the target object can be easily determined by using a tag. For example, the tower 508 is selected as the target object. (b) shows the state after the change. The HMD1 adjusts, for example, the transparency of the guide map 504 that is blocking the selected tower 508 up. This makes the entire tower 508 visible.

[0155] The HMD 1 may display the tag all the time, or may display the tag only when it is determined that it is difficult to determine the target object using only the gaze point, or may display the tag in response to a tag display instruction input by the user U1.

[0156] [Variations] FIG. 33 shows an example in which the display device or information processing device according to the modified example of the fourth embodiment is applied to a smartphone. FIG. 33 shows an example in which a tag is attached to each object as a display example on the display surface of the smartphone 700. Even in the case of the smartphone 700, the occlusion and obstruction relationship between objects is assumed to be a three-dimensional arrangement taking into account the position in the depth direction. Therefore, the method of changing the display mode and the like in each of the above-mentioned embodiments can be similarly applied. Although not shown, the functional block configuration of the smartphone 700 is basically the same as the configuration of FIG. 10. In the smartphone 700, gaze detection and gaze point detection are not performed, and other operation input means are used. In the smartphone 700, gaze detection and gaze point detection may be realized using other means (for example, the camera unit 431). In the smartphone 700, the actual object (corresponding individual actual object) is displayed as an image captured by the mounted camera (camera unit 431). In the case of the smartphone 700, as a tag selection input reception method and other operation input, in addition to voice input and the like, a selection input by tapping on a touch panel on the display surface and the like is also possible.

[0157] <Fifth Preferred Embodiment> A fifth embodiment will be described with reference to Fig. 34. In the above-described embodiment, a case where an object in front occludes and obstructs an object in the rear is shown as an occlusion obstruction relationship between two objects in the depth direction. In the case of such an occlusion obstruction relationship, an example is shown in which the display mode is changed so that at least the target object is easily visible. There are other relationships of objects for which the display mode is changed besides the occlusion obstruction relationship described above.

[0158] In the fifth embodiment, the relationship between objects when the user visually recognizes multiple objects is determined by the difference in luminance. Consider the case where two objects (individual real objects or virtual objects) are placed one behind the other in the field of view, or are placed close to each other, such as to the left and right. If there is a large difference in luminance (in other words, brightness) between the objects, one object may interfere with the visual recognition of the other object. In this case, the HMD of the fifth embodiment changes the display mode.

[0159] FIG. 34 shows a display example. In the field of view 101, an object 102 named "A" in front and an object 103 named "B" in the rear are arranged. The gaze point 106 of the user U1 is located on the object 102 named "A" in front, and the object "A" is the target object. In this case, the user U1 can basically see the entirety of the object 102 named "A", and there is no occlusion obstruction relationship as described above. Here, if there is a large difference in brightness between the object "A" and the object "B", for example, if "B" has a higher brightness, the object "B" may obstruct the visibility of the target object "A". This is not limited to the objects in front and behind, but also in the case of a non-occlusion relationship where there are nearby objects on the left and right, a nearby object may obstruct the visibility.

[0160] The HMD1 determines the difference in luminance between the objects, and from that difference, determines an obstructing object in terms of luminance. The HMD1 changes the display mode of the determined obstructing object, for example, object "B". For example, the HMD1 moves the display position of object "B" away from object "A", as in the case after the change in (b). Alternatively, if the change in the display mode of object "B" is undesirable, the HMD1 may change it by moving the target object "A", for example.

[0161] In particular, in the fifth embodiment, as another method of changing the display mode, the HMD1 may use a method of temporarily changing the luminance of the object. For example, the HMD1 temporarily reduces the luminance of the "B" object. This reduces the luminance difference, making it easier for the user U1 to visually recognize the "A" target object.

[0162] Although the present invention has been specifically described based on the embodiments, the present invention is not limited to the above-mentioned embodiments and can be modified in various ways without departing from the scope of the present invention. Combinations of the embodiments and additions, deletions, or substitutions of components are also possible. [Explanation of symbols]

[0163] 1...HMD (head-mounted information processing device), 11...display surface, U1...user, 101...field of view, 102, 103...object, 104, 105...line of sight, 106...point of gaze, 107...target visual range, 120...information server.

Claims

1. A display device for displaying an image; a processor for controlling display of said image; Equipped with displaying, on the display device, at least one of an individual entity object cut out from an entity in the outside world and a virtual object arranged in three dimensions as an object, the virtual object; Calculating a user's gaze point, and determining the object closest to the gaze point position as a target object; setting the object having a high display relevance to the target object as a related object for the target object; A target visible range is set as an image area obtained by combining the image area of ​​the target object and the image area of ​​the related object; Detecting the object obstructing at least a portion of the target visual range as an obstructing object; When the obstructing object is present, a display mode of at least one of the object in the target visual recognition range and the obstructing object is changed so as to eliminate or reduce obstruction caused by the obstructing object to visibility of the target visual recognition range. Display device.

2. 2. The display device according to claim 1, comparing attributes of the target object and the obstructing object that indicate the degree of restriction or tolerance regarding the change in the display mode of the object, and determining at least an object that is to be subjected to the change in the display mode; Display device.

3. 2. The display device according to claim 1, The change in the display mode is a movement of the display position, an adjustment of the transparency, a reduction or enlargement, or a display of a duplicate object. Display device.

4. 2. The display device according to claim 1, When a plurality of objects overlap as the object in the line of sight of the user, the object having the highest visibility among the objects is determined as the target object. Display device.

5. 2. The display device according to claim 1, When there are a plurality of objects as the object within a field of view associated with the display device, determining an object designated by an input operation by the user as the target object; Display device.

6. 2. The display device according to claim 1, Detecting the object obstructing at least a portion of the target visual range as the obstructing object; Display device.

7. 2. The display device according to claim 1, When a plurality of objects overlap as the object in the line of sight of the user and a rear object is occluded and hidden by a front object, the display mode is changed so that the rear object is a target candidate object and at least a part of the object is not occluded and is visible. Display device.

8. 8. The display device according to claim 7, The change in the display mode is a movement of the display position, an adjustment of the transparency, a reduction or enlargement, or a display of a duplicate object. Display device.

9. 2. The display device according to claim 1, In a case where each user of a plurality of users uses the display device, the plurality of users are shared users and the object is used as a shared object, and the user is a first user and the other users are second users, Based on communication between a first display device of the first user and a second display device of the second user, the first display device of the first user displays mark information indicating that the second user is gazing at the shared object that the second user is gazing at as the target object, with respect to the shared object that the second user is gazing at as the target object; Display device.

10. 10. The display device according to claim 9, When at least a part of the shared object that the second user is gazing at as the target object is hidden by another object, the first display device of the first user changes a display mode of the object so that the entirety of the target object of the second user is visible. Display device.

11. 11. The display device according to claim 10, When changing the display mode of the object so that the entirety of the target object of the second user is visible, a state as viewed by the second user is displayed as the entirety of the target object of the second user. Display device.

12. 2. The display device according to claim 1, In a field of view corresponding to the display device, a first object and a second object as the objects overlap in a line of sight of the user or are disposed in close proximity to each other, Detecting a difference in luminance between the first object and the second object; When the difference in luminance is equal to or greater than a threshold, the object having the greater luminance is detected as the obstructing object. Display device.

13. A display method for a display device including a display device that displays an image and a processor that controls display of the image, comprising: displaying, on the display device, at least one of an individual entity object cut out from an entity in the outside world and a virtual object arranged in three dimensions; calculating a user's gaze point and determining the object closest to the gaze point position as a target object; setting the object having a high display relevance to the target object as a related object for the target object; setting an image area obtained by combining the image area of ​​the target object and the image area of ​​the related object as a target visible range; detecting the object obstructing at least a portion of the target visibility range as an obstructing object; When the obstructing object is present, changing a display mode of at least one of the object of the target visual recognition range and the obstructing object so as to eliminate or reduce obstruction caused by the obstructing object to visibility of the target visual recognition range; A display method comprising:

Citation Information

Patent Citations

  • Game system and information storage medium

    JP2002042156A

  • Modifying Virtual Object Display Characteristics to Increase Power Performance of Augmented Reality Devices

    JP2016504615A

  • See-through computer display systems

    US20160239985A1

  • Information processing device, information processing method, and program

    WO2016203792A1

  • Information processing device, information processing method, and program

    WO2017104198A1