Display apparatus

Abstract

To provide a technique for eliminating or reducing visual obstruction when an entity or an object such as a virtual object that a user wishes to visually recognize is obstructed by another object, in relation to a display apparatus.SOLUTION: A display apparatus of an embodiment includes a display device and a processor, and displays at least a virtual object out of an individual entity object extracted from an entity in an external world and a virtual object arranged three-dimensionally on the display device, determines an object that a user wishes to gaze at as a target object, detects an object that interferes as an obstructing object when the user visually recognizes the target object, and changes, when the obstructing object is present, a display mode of at least one of the target object and the obstructing object so as to eliminate or reduce an obstruction caused by the obstructing object with respect to visual recognition of the target object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to technology for display devices and information processing devices, and to technology for displaying images of virtual objects and the like. [Background technology]

[0002] In recent years, display devices capable of displaying images of virtual objects and the like have become widely used. One example is a head-mounted information processing device (also known as a head-mounted display: HMD), which is worn on the user's head and equipped with a display unit, camera unit, and the like. Head-mounted information processing devices display real-world objects and virtual objects, seamlessly blending the real world and virtual world in real time, allowing users to experience virtual objects as if they were actually present in the real world. Display methods include the so-called video see-through type and optical see-through type. In the video see-through type, images corresponding to real objects and virtual objects are generated and displayed on a display unit in front of the head. In the optical see-through type, images of virtual objects are displayed on a display unit superimposed on the real objects in front of the user's eyes, while the real objects in front of the user are visible.

[0003] An example of prior art related to the above-mentioned display device is Japanese Patent Application Laid-Open No. 2015-090635 (Patent Document 1). Patent Document 1 states that "information is displayed appropriately while ensuring the user's field of view" and the following: In this information display system having a see-through head-mounted display, a control unit detects a user's gaze point based on image data of both eyes of the user, determines based on the gaze point whether the user is gazing at a virtual screen or at a background beyond the virtual screen, determines whether the user's line-of-sight area overlaps with the display position of an object on the virtual screen, and, if the gaze point moves, changes the display position and / or display form of the object based on the determination result. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-090635 Summary of the Invention [Problem to be solved by the invention]

[0005] In display devices such as conventional head-mounted information processing devices, when a real object in real space and a virtual object are displayed overlapping each other, depending on their relative positions and the user's line of sight, the real object or virtual object that the user wants to view may be blocked by other real objects or virtual objects, making it difficult to view or obstructing visibility.

[0006] Patent Document 1 describes an HMD that realistically sees through real objects and displays virtual objects on a virtual screen. The system determines whether the user is gazing at the virtual screen or the background based on the user's gaze point, and determines whether the line of sight overlaps an object on the virtual screen. The system changes the display position and / or display form of the virtual object based on both of these determination results. Patent Document 1 also describes that when the object the user is gazing at is overlapped and covered by a virtual object, the information display system changes the display position or display form of the virtual object based on its transparency. However, Patent Document 1 only considers eliminating visual obstructions in the line of sight, and does not consider obstructions to the range the user wants to view. Patent Document 1 also does not suggest any display that reflects occlusion relationships when real objects and virtual objects are arranged in three dimensions (3D).

[0007] The object of the present invention is to provide a technology for a display device such as a head-mounted information processing device that can display virtual objects arranged in three dimensions, which can eliminate or reduce the obstruction of visibility when the visible range of an object such as a real object or virtual object that the user wants to view is obstructed by another object, allowing the user to view the entire object in an optimal manner, and which can realize such a function with little effort and ease of use for the user. Other problems and advantages will be described in the "Description of the Invention" below. [Means for solving the problem]

[0008] A representative embodiment of the present invention has the following configuration: A display device according to the embodiment includes a display device that displays an image and a processor that controls the display of the image, and displays at least two or more three-dimensionally arranged virtual objects as objects on the display device, calculates a user's point of gaze, and sets an area including the object that is closest to the position of the point of gaze as a target visible range, detects an object that obstructs at least a part of the target visible range as an obstructing object, and, if the obstructing object is present, changes the display mode of at least one of the target visible range and the obstructing object so as to eliminate or reduce obstruction by the obstructing object to visibility in the target visible range. In a case where multiple users, with the user as a first user and other users as second users, use the display device and the multiple users are shared users and use the object as a shared object, the first display device of the first user displays mark information indicating that the second user is gazing at the shared object based on communication between the first display device of the first user and the second display device of the second user. [Effects of the Invention]

[0009] According to a representative embodiment of the present invention, in relation to the technology of a display device such as a head-mounted information processing device capable of displaying virtual objects arranged in three dimensions, when the viewing range of an object such as a real object or virtual object that a user wants to view is obstructed by, for example, another object, the obstruction to viewing can be eliminated or reduced, allowing the user to view the entire object favorably, and such a function can be realized in an easy-to-use manner with little effort on the part of the user. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows an outline of the configuration and a display example of a head-mounted information processing device (HMD) which is a display device according to a first embodiment of the present invention. [Figure 2] In the first embodiment, classification of objects, occlusion and obstruction relationships, and categories are described. [Figure 3] In the first embodiment, a display example in the case of transparency adjustment will be shown. [Figure 4] In the first embodiment, a display example in the case of transparency adjustment will be shown. [Figure 5] In the first embodiment, a display example in the case of reduction / enlargement is shown. [Figure 6] In the first embodiment, a display example in the case of moving the display position will be shown. [Figure 7] In the first embodiment, a display example in the case of moving the display position will be shown. [Figure 8] In the first embodiment, a display example in the case of duplicate display will be shown. [Figure 9] In the first embodiment, the main processing flow is shown. [Figure 10] In the first embodiment, an example of a functional block configuration is shown. [Figure 11] In the first embodiment, a display example is shown. [Figure 12] In the first embodiment, a display example is shown. [Figure 13] In the first embodiment, a processing flow of an operation example will be shown. [Figure 14] In the first embodiment, a display example is shown. [Figure 15]In the first embodiment, a display example is shown. [Figure 16] In the first embodiment, a display example is shown. [Figure 17] In the first embodiment, a display example is shown. [Figure 18] In the first embodiment, a display example is shown. [Figure 19] In the first embodiment, a display example is shown. [Figure 20] 10 shows a display example in a display device according to a second embodiment of the present invention. [Figure 21] A display example will be shown in the second embodiment. [Figure 22] In the second embodiment, supplementary explanatory diagrams are shown. [Figure 23] In the second embodiment, a processing flow of an operation example will be shown. [Figure 24] In each embodiment, an example of object data is shown. [Figure 25] 10 shows a first example of sharing in the display device according to the third embodiment of the present invention. [Figure 26] 13 shows a second example of sharing in the display device according to the third embodiment of the present invention. [Figure 27] A display example will be shown in the third embodiment. [Figure 28] A display example will be shown in the third embodiment. [Figure 29] A display example will be shown in the third embodiment. [Figure 30] A display example will be shown in the third embodiment. [Figure 31] A display example will be shown in the third embodiment. [Figure 32] 10 shows a display example in a display device according to a fourth embodiment of the present invention. [Figure 33] 13 shows a display example in a modification of the fourth embodiment. [Figure 34] 10 shows a display example in a display device according to a fifth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Embodiments of the present invention will be described in detail below with reference to the drawings. In the drawings, identical components are generally designated by the same reference numerals, and repeated description will be omitted. In the drawings, the depiction of each component may not represent its actual position, size, shape, or scope, etc., in order to facilitate understanding of the invention. The present invention is not necessarily limited to the position, size, shape, and scope, etc., disclosed in the drawings. Unless otherwise specified, each component may be singular or plural. For purposes of explanation, when describing program-based processing, the program, function, processing unit, etc. may be described as the main focus. However, the main focus of these hardware components is a processor, or a controller, device, computer, system, etc., configured with the processor. A computer executes processing in accordance with a program loaded into memory using resources such as memory and communication interfaces as appropriate, thereby realizing predetermined functions, processing units, etc. The processor may be configured, for example, with a semiconductor device such as a CPU or GPU. The processor may be configured with a device or circuit capable of performing predetermined calculations. Processing is not limited to software program processing; it can also be implemented using dedicated circuits. Dedicated circuits such as FPGAs and ASICs can be used. The program may be pre-installed as data on the target computer, or may be distributed as data from a program source to the target computer and installed. The program source may be a program distribution server on a communication network or a non-transitory computer-readable storage medium. The program may be composed of multiple program modules. For the purpose of explanation, various data and information may be described using expressions such as tables and lists, but such structures and formats are not limited. Furthermore, data and information for identifying various elements may be described using expressions such as identification information, identifiers, IDs, names, and numbers, but these expressions are interchangeable.

[0012] <First Embodiment> A display device and a display method according to a first embodiment of the present invention will be described with reference to Fig. 1 and other figures. The display device according to the first embodiment is a virtual object display device, and is shown as being applied to a head-mounted information processing device (hereinafter referred to as HMD). The display method according to the first embodiment is a method having steps executed by the display device according to the first embodiment.

[0013] The display device of the first embodiment includes a display device (in other words, a display) capable of displaying virtual objects, and a processor that controls the display of the virtual objects on the display device, and displays, on the display surface of the display device, at least virtual objects, including external real objects that include, as parts thereof, individual real objects cut out from or recognized from external real objects, and virtual objects arranged in three dimensions. In the case of a video see-through type, individual real objects and virtual objects can be displayed as images of the objects. In the case of an optical see-through type, a virtual object can be displayed as an object so as to be aligned with the external real object.

[0014] The display device of the first embodiment determines and determines an individual entity object or a virtual object that is an object that the user wishes to gaze at as a target object, and detects an individual entity object or a virtual object that is an object that interferes with the user's viewing of the target object as an obstructing object. When the display device of the embodiment detects the presence of an obstructing object, it changes the display mode of at least one of the target object and the obstructing object so as to eliminate or reduce the obstruction of the obstructing object from the viewing of the target object.

[0015] [1. Display device (HMD)] FIG. 1 shows an outline of the configuration and a display example of a head-mounted information processing device (HMD) 1, which is a display device according to the first embodiment. FIG. 1 shows a schematic configuration of the appearance when a user U1 wears the HMD 1 on his / her head. FIG. 1 also shows how the user U1 views an image of a three-dimensional object displayed in a field of view 101 by the HMD 1. FIG. 1 also shows an example of a change in the display mode of the object in the field of view 101. (a) is an example of a display before the change, showing a case where there is an occlusion obstruction relationship between objects "A" and "B." (b) is an example of a display after the change, showing a state where the occlusion obstruction relationship between objects "A" and "B" is temporarily resolved.

[0016] The HMD1 is worn on the head of the user U1 and displays an image of an object or the like within a field of view 101 of the user U1. The field of view 101 is associated with a display surface 11 of a display device provided in the HMD1. The object is an individual real object that is part of a real object, or a virtual object arranged in three dimensions. The user U1 can view, for example, objects 102 and 103 within the field of view 101. For example, the object 102 is a virtual object shaped like a rectangular parallelepiped and marked with "B." The object 103 is a virtual object shaped like a rectangular parallelepiped and marked with "A." From the user U1's perspective, the object 102 is located behind the object 103 located in front. The object 103 in front occludes at least a portion of the object 102 in the rear, thereby obstructing the view of the object 102, or in other words, making it difficult to view. For the sake of explanation, such objects 102 and 103 (a pair of two objects) are referred to as objects in an "occlusion obstruction relationship."

[0017] The lines of sight of the user U1's eyes include a line of sight 104 of the left eye and a line of sight 105 of the right eye. The direction of the lines of sight 104, 105 of the user U1's eyes can be used to calculate a gaze point 106, which is the position in three-dimensional space where the user U1 is gazing. An object located near the gaze point 106, for example, the object "B" 102, is associated with a desired object that the user U1 gazes at and recognizes as a target or goal. The HMD1 determines and determines such an object as a target object based on the lines of sight and the gaze points of the eyes. In this example, in (a), the object "B" 102, where the gaze point 106 is located, is determined as the target object. The HMD1 sets a target visible range 107 for the target object. The target visible range 107 is a range that is related to the target object and that the user U1 is estimated to want to recognize.

[0018] On the other hand, the object "A" 103 in front occludes a part (for example, the lower left part) of the target visible range 107 of the object "B" 102, which is the target object that the user U1 intends to view. In this case, the occluding object "A" 103 prevents the user U1 from viewing the entire target visible range 107 of the object "B" 102, which is the target object. The HMD1 determines and detects such an object that obstructs visibility as an obstructing object. The HMD1 understands the relationship between objects such as "A" and "B" as an "occlusion obstruction relationship."

[0019] When such an obstruction relationship exists, the HMD 1 changes the display mode of these objects. For example, the HMD 1 changes the display mode of the object "A" 103, which is an obstructing object obstructing the target visible range 107. Specifically, in this example, the HMD 1 changes the display position of the object "A" 103 within the field of view 101 to a position outside the target visible range 107. In other words, the HMD 1 moves the object 103 to an empty position outside the target visible range 107 and replaces it with the state of the object 103a after the movement. With this change, the HMD 1 makes the entire target visible range 107 unobstructed. This allows the user U1 to view the entire target visible range 107 of the object "B" 102, which is the target object. When moving the object "A" 103, which is an obstructing object, the HMD 1 may determine the display position after the movement so as to be as close as possible to the original display position of the object 103 and the target object. The above example of changing the display mode of the obscuring object is an example of changing the display position on the obstructing object side, but it is not limited to this, and various change methods described later are possible.

[0020] Note that information and data such as virtual objects may be generated within the HMD 1, or may be generated outside the HMD 1, for example, by an information server 120, and supplied to the HMD 1 via an external network. The information server 120 can handle large amounts of information, and can generate and store, for example, high-quality, high-resolution virtual objects. The external device may also be a user's mobile information terminal, a home device, or the like.

[0021] [Target Object] In the first embodiment, a gaze point 106 in a three-dimensional space, which can be calculated from the two gaze directions 104 and 105 in FIG. 1, is used as a means for specifying, determining, and confirming a target object, which is an object that the user U1 wishes to gaze at. The HMD 1 can determine, for example, the object closest to the gaze point 106 as the target object. This means is not limited to this, and various means can be applied. Other means include a pointer using a remote controller or the like, voice input, and hand gesture recognition. When using a pointer, the user U1 operates the pointer displayed on the display screen using a remote controller or the like. The HMD 1 may determine, as the target object, an object where the pointer is located within the field of view 101, or an object designated by an on-operation of the pointer. When using voice input, the user U1 inputs information identifying a displayed object by voice. The HMD 1 recognizes the input voice, and if it recognizes, for example, "B," it may determine the "B" object 102 as the target object.

[0022] Object Further explanation of terminology will be provided using FIG. 2. (A) of FIG. 2 shows classification of "objects." In the first embodiment, there are two main types of objects that the HMD 1 displays on the display surface 11. These two types of objects are referred to as "individual entity objects" and "virtual objects." These objects are elements that can form an occlusion obstruction relationship. In the HMD 1 of the first embodiment, these objects are objects that can be arranged three-dimensionally in the field of view 101 corresponding to the display surface 11. In other words, these objects are objects that can be arranged both in front and behind in the depth direction when the field of view 101 is viewed from the viewpoint of the user U1. Objects arranged in front and behind may overlap each other, resulting in an occlusion obstruction relationship.

[0023] This object is not necessarily an image (referring to an image generated by a display device). An "individual real object" is an object based on a real object (in other words, a real image). In the case of a video see-through type, an "individual real object" is an image of an individual real object cut out from the real object. In the case of an optical see-through type, an "individual real object" is an individual real object cut out from the real object (in other words, recognized) and is not an image. A "virtual object" is an image of any virtual object generated by a display device in relation to or independent of a real object.

[0024] (B) of FIG. 2 shows patterns of occlusion / obstruction relationships between objects in the first embodiment. There are four patterns in which the above two types of objects are arranged in front of and behind each other. In the first pattern, an individual entity object is arranged in front of an individual entity object in the back. In the second pattern, a virtual object is arranged in front of an individual entity object in the back. In the third pattern, an individual entity object is arranged in front of a virtual object in the back. In the fourth pattern, a virtual object is arranged in front of a virtual object in the back. With some exceptions, the HMD 1 of the first embodiment can apply display mode changes to each of these patterns.

[0025] [Display example] 3 to 8 show various examples of changes in display mode as display examples in the field of view 101 corresponding to the display surface 11 of the HMD 1. FIG.

[0026] [Display example (1)] FIG. 3A shows an example of a display mode change in which the transparency (or in other words, the transparency) of the object "A" 103, which is the front obstructing object, is increased when there is an obstructing relationship between the objects "A" and "B" as shown in FIG. 1A. This allows the obstructing object to be transparent, making it easier to view the target visible range 107 of the partially obstructed target object "B" 102. This allows the user U1 to view the entire target visible range 107 of the target object. In this example, the HMD 1 increases the transparency of only a portion 103X of the image area of ​​the object 103, which is the front obstructing object, that obstructs the rear target visible range 107, making it closer to transparency. This transparency increase adjustment reduces the degree of visibility obstruction.

[0027] FIG. 3B shows another example in which the HMD 1 adjusts the transparency of only the portion 103X of the front obstructing object that is blocking the target visible range 107 to the maximum. In other words, this portion 103X with maximum transparency is hidden. In terms of appearance, the target visible range 107 of the rear "B" object 102 appears to be temporarily in front of the "A" object 103. This prevents the target visible range 107 from being blocked at all, eliminating the obstruction to visibility.

[0028] FIG. 4A shows another example in which the same transparency adjustment is made to all of the obstructing objects "A" 103. This makes all of the obstructing objects transparent, making it easier to see the target visible range 107. At the same time, because the obstructing objects are displayed with the same transparency, they are also easy to see.

[0029] FIG. 4B shows another example in which all of the obscuring objects "A" 103 are hidden with maximum transparency. In this case, the obscuring objects are completely invisible, making it easy to confirm the entire target visible range 107. As shown in the examples of FIGS. 1 to 4 above, the visibility obstruction of the target visible range of the target object caused by the obscuring objects can be eliminated or the degree of visibility obstruction can be reduced by changing the display mode.

[0030] [Display example (2)] FIG. 5 shows another example of a change in display mode. The change from (a) to (b) in FIG. 5 shows a case where the obstructing object "A" 103 is reduced and its transparency is increased. After the change, the "A" object 103 is replaced with the object 103b. In this way, the HMD 1 changes the size of the obstructing object so that it is smaller than the target object. This further reduces the degree of visibility obstruction caused by the obstructing object. Alternatively, the obstructing object may be reduced in size, which still has the effect of making it easier to confirm the target visible range. Similarly, as another method, the change from (a) to (c) in FIG. 5 shows a case where the target object "B" is enlarged relative to the obstructing object "A." After the change, the "B" object 102 is replaced with the enlarged object 102c. In this case, too, the effect of making it easier to confirm the target visible range is obtained.

[0031] [Display example (3)] 6 shows an example in which the display mode of the target object is changed rather than the obstructing object. For example, suppose the target object is a virtual object, and the obstructing object is a virtual object or an individual entity object that is less suitable for increasing the transparency or changing the display position than the target object in terms of visual discomfort for user U1. In this case, changing the display mode of the target object as shown in FIG. 6 is useful.

[0032] FIG. 6 shows a case where the display position of a target object is changed. In (a) before the change, a "C" object 109 is placed in the foreground, and a "B" object 102, which is a virtual object, is placed in the background. The "B" object 102 is the target object. A portion of the target visible range 107 of the "B" target object is occluded by the "C" object 109. The "C" object 109 in the foreground is a virtual object or an individual entity object that is not suitable for adjusting the transparency up or changing the display position. (b) shows the state after the change. The HMD 1 moves the display position of the "B" object 102, which is the target object, to outside the occlusion range of the "C" object 109, which is an obstructing object. The "B" object 102 and the target visible range 107 are replaced by the moved object 102b and the target visible range 107b. As a result, the entire target visible range 107b is visible. Accompanying this movement, the gaze point of the user U1 moves, for example, from the gaze point 106 to the gaze point 106b. This allows the user U1 to view the entire target visible range 107b of the moved object 102b, which is located at the gaze point 106b. This is equivalent to viewing the entire target visible range 107 of the original object 102.

[0033] When moving a target object, the HMD 1 moves it to an empty position within the field of view 101, that is, to a position that does not obstruct the visibility of other objects. In this example, since the left side of objects "B" and "C" is empty, the HMD 1 moves the target object to the left.

[0034] FIG. 7 shows another example of a change in display mode. The HMD 1 may move both the target object and the obstructing object for objects "A" and "B" that are in an obstructing relationship. Moving both objects is effective when the angle of view of the display is small. In the change from (a) to (b), the target object "B" object 102 and the obstructing object "A" object 103 are moved in directions away from each other (left and right in this example). This makes the entire target visible range 107 visible.

[0035] [Display example (4)] FIG. 8 shows yet another method of changing the display mode, which is a method of displaying a duplicate object instead of moving an object. In the change from (a) to (b), the HMD 1 displays the target object "B" 102, which is partially obscured by the obstructing object "A" 103, as is. Furthermore, the HMD 1 generates a duplicate object 102r of the "B" object 102 and displays it in an empty position (for example, a position on the left). In addition to displaying the duplicate object 102r, the HMD 1 may also display information informing the user U1 that it is a duplicate. The HMD 1 makes the entire target visual range 107r of the duplicate object 102r visible. This allows the user U1 to view the entire target visual range 107r of the duplicate object 102r from the gaze point 106r after the movement. This is equivalent to viewing the entire target visual range 107r of the original object 102. In this method, the user U1 can visually confirm the entire target object using the duplicate object, and can also grasp the positional relationship between the original "B" object 102 and "C" object 103 while maintaining it as is.

[0036] As described above, the HMD 1 of the first embodiment changes the display mode of at least one of the objects, such as the display position, transparency, size, or duplication, when an obstructing object obstructs at least a portion of the target object's target visible range. The various change methods can also be applied in combination. This can eliminate or reduce the degree of obstruction to the visibility of the target object caused by the obstructing object. The HMD 1 determines the details of the display mode change, taking into account the details of the obstruction relationship. For example, when changing the display mode of the obstructing object is not appropriate, the HMD 1 changes the display mode of the target object.

[0037] [Transmission of display mode change status] When there is an occlusion obstruction relationship between objects, the HMD 1 temporarily changes the display mode of the object as in the above example. At this time, the HMD 1 may output a GUI or the like to clearly inform the user U1 that the display mode is being temporarily changed. For example, the HMD 1 may display an image on the display surface indicating that the display mode is being changed. Image 130 in FIG. 3(B) is an example of such an image. Also, for example, when changing the display position of an object, the HMD 1 may use animation, effects, or the like to express the state of the change, or may display the changed object in a specific color, or the like.

[0038] Furthermore, the HMD 1 may temporarily lock the determination process of the gaze point during the above-mentioned process of changing the display mode, which prevents erroneous determination of the target object when the gaze point 106 moves in conjunction with a change in the object display position, such as in Fig. 6.

[0039] [Processing flow] FIG. 9 shows a main processing flow for explaining the basic operation of the HMD 1 according to the first embodiment. The flow in FIG. 9 includes steps S1 to S8. In step S1, the HMD 1 detects the gaze point 106 at which the user U1 is gazing in space based on the detection of the line of sight (104, 105) of both eyes of the user U1 shown in FIG. 1. Based on the position of the detected gaze point 106, the HMD 1 determines and confirms a target object that is presumed to be the desired object that the user U1 is attempting to view. Since the HMD 1 is aware of the positions of each object in three-dimensional space and the position of the gaze point 106, it can compare these positions and determine and confirm, for example, the object closest to the position of the gaze point 106 as the target object. Note that the target object is determined using the gaze point 106 here, but variations will be described later.

[0040] Next, in step S2, the HMD1 selects and determines a target visual range that is presumed to be what the user U1 intends to view for the determined target object. For example, in the example of FIG. 1 etc., the target visual range is selected as the same image area as the apparent display range of the target object (an image area with pixels along the shape). Alternatively, the target visual range may be selected as an image area that includes the target object (for example, a circumscribing rectangle or a circumscribing ellipse). Alternatively, the target visual range may be a region such as a rectangle or an ellipse of a predetermined size centered on the gaze point.

[0041] In step S3, the HMD 1 determines whether an obstructing object exists that obstructs the target visible range of the determined target object. For example, the HMD 1 may determine that an obstructing object exists when a predetermined percentage or more of the target visible range is obstructed by an object in front. If an obstructing object exists (Y), the process proceeds to step S4; if not (N), the process skips step S4.

[0042] In step S4, the HMD 1 changes the display mode of the object so as not to obscure the target visible range of the target object. As in the example described above, the display mode change method can be selected from a suitable method such as display position, transparency, size, and duplication for at least one of the obstructing object and the target object. In particular, when the obstructing object is less suitable for display mode change than the target object, the HMD 1 selects the method for changing the display mode of the target object.

[0043] In step S5, when the HMD1 changes the display mode, it maintains the state after the change for a certain period of time. This allows the user U1 to view the entire target visible range of the target object in that state. If there are no obstructing objects (S3-N), the user U1 can view the entire target visible range of the target object even without changing the display mode.

[0044] In step S6, the HMD1 determines whether the gaze point of the user U1 has moved outside the target visible range of the target object. If the gaze point remains unchanged and is within the target visible range (S6-N), the process returns to step S5. This maintains the state of the display mode change, and maintains the state in which the target visible range is visible.

[0045] On the other hand, if the gaze point has moved outside the target visual recognition range (S6-Y), the process proceeds to step S7. In step S7, the HMD 1 restores the display mode change state of the target object and the obstructing virtual object in the occlusion obstruction relationship to the original state before the change.

[0046] After this, in step S8, the HMD 1 checks whether to continue or end the control process based on, for example, the state of gaze. If it continues (N), the process returns to step S1, and the detection of a new gaze point and the like are repeated in the same way. If it ends (Y), this flow ends.

[0047] Based on the above processing flow, in the first embodiment, when a target object is occluded by an obstructing object, the visibility obstruction in the target visible range can be eliminated or the degree of visibility obstruction can be reduced by changing the display mode of the object.

[0048] In the above example, the display mode change state is maintained for a certain period of time depending on the state of the gaze point, but this is not limited to this. The display mode change may also be terminated when user U1 inputs a specified operation, or when it is detected that the line of sight or gaze point has reached a specified state.

[0049] [Function Block] 10 shows an example of a functional block configuration of the HMD1, which is a display device according to the first embodiment. Note that the configuration is basically similar for other types of display devices. In this configuration example, the components are implemented in a single device, but this is not limiting, and some components may be implemented separately in separate devices.

[0050] In Figure 10, the HMD1 is configured using a processor 410, a memory unit 420, a camera unit 431, a distance measurement sensor 440, a left eye gaze detection unit 432, a right eye gaze detection unit 433, a display processing unit 434, an operation input unit 435, a microphone 436, headphones 437, a vibration generation unit 438, and a communication unit 439 as appropriate, and each component is connected to each other via a bus 450.

[0051] The processor 410 is composed of a CPU, ROM, RAM, etc., and constitutes the controller of the HMD 1. The processor 410 executes processing in accordance with an operating system (OS) 422 and an application program 423 for operation control, which are stored as a control program 421 in a memory unit 420. In this way, the processor 410 controls each component and realizes the functions of the OS, middleware, application, etc., and other functions.

[0052] The memory unit 420 is configured with a nonvolatile storage device or the like, and stores various programs 421 handled by the processor 410 or the like, and information data 424. The information data 424 stores gaze point information 425 indicating the position of the gaze point gazed at by the user U1, target object information 426 indicating the shape and position of a target object visually recognized by the user U1, virtual object information 427 indicating the shape and position of a virtual object, and the like.

[0053] The camera unit 431 photographs the field of view and visual field conditions in front of the HMD 1, and acquires images by converting light incident from the lens into an electrical signal using an image sensor. In the case of an optical see-through HMD, the user U1 directly sees the field of view and real objects in the field of view in front of the HMD 1. In the case of a video see-through HMD, the camera unit 431 photographs the field of view and real objects in the field of view in front of the HMD 1, and the image of the photographed real objects is displayed on the display device of the display processing unit 434.

[0054] The distance measurement sensor 440 is a sensor that measures the distance between the HMD 1 and an external physical object. The distance measurement sensor 440 may be a TOF (Time Of Flight) sensor, a stereo camera, or another method. The HMD 1 obtains three-dimensional position information of the external physical object using the distance measurement sensor 440 and the position data, and displays an object that reflects the occlusion relationship between the individual physical object and the virtual object. The HMD 1 may refer to the position data of the external physical object, including occluded objects, based on some feature points of the external physical object. This position data may be created or stored by the HMD 1, or may be obtained from an external information server 120 or the like.

[0055] The left eye gaze detection unit 432 and the right eye gaze detection unit 433 detect the gazes (104, 105) by capturing the movement and direction of the left eye and the right eye, respectively. Note that this gaze detection process can utilize well-known techniques commonly used for eye tracking processing. For example, a method using corneal reflex is known in which infrared light is irradiated onto the face from an infrared light-emitting diode (LED) and photographed with an infrared camera. The position of the reflected light on the cornea is used as a reference point, and the gaze is detected based on the position of the pupil relative to the position of the corneal reflex. Another known method is to photograph the eyes with a visible light camera, use the inner corner of the eye as a reference point, and use the iris as a moving point to detect the gaze based on the position of the iris relative to the inner corner of the eye. The intersection of the left eye gaze 104 detected by the left eye gaze detection unit 432 and the right eye gaze 105 detected by the right eye gaze detection unit 433 is detected as the gaze point 106 at which the user U1 is gazing.

[0056] The display processing unit 434 is composed of a display device and a unit that performs display processing. In the case of an optical see-through HMD, the display processing unit 434 has, for example, a projection unit that projects light corresponding to virtual objects and notification information for the user, and a transparent half mirror that forms an image of the projected light in front of the user's eyes. In this case, the display surface 11 in FIG. 1 corresponds to the half mirror. This allows the user U1 to view both real objects in the field of view and field of view in front of the user's eyes and images of the formed virtual objects, etc., as if they were floating in real space. In the case of a video see-through HMD, the display processing unit 434 has a display device such as a liquid crystal display panel that displays images of real objects in front of the user's eyes (including individual extracted real objects) captured by the camera unit 431 and images of generated virtual objects, etc., together. In this case, the display surface 11 corresponds to a screen such as a liquid crystal display panel. As a result, the user U1 can use the HMD 1 to view real objects in the field of view in front of the user's eyes superimposed on virtual objects, etc.

[0057] The operation input unit 435 is an input means such as a keyboard, key buttons, or touch keys, and allows the user U1 to set and input desired information. The operation input unit 435 is provided in a position and form that allows the user U1 to easily perform input operations on the HMD 1. Alternatively, the operation input unit 435 may be provided separately from the HMD 1 body and connected via wire or wirelessly, like a remote controller. The HMD 1 may also display a graphical user interface (GUI) such as an input operation screen on the display surface 11 of the display processing unit 434, and acquire input operation information according to the position on the input operation screen to which the gaze detected by the left eye gaze detection unit 431 and the right eye gaze detection unit 432 is directed. The HMD 1 may also display a pointer on the input operation screen, and the user U1 may operate the pointer using the operation input unit 435 to acquire input operation information. The HMD 1 may also acquire input operation information by collecting voice representing an input operation spoken by the user U1 using a microphone 436.

[0058] The microphone 436 collects external sounds and the user's own voice. The HMD1 can capture instruction information from the user U1's voice and conveniently execute operations in response to the instruction information. The headphones 437 are worn by the user U1's ears and output audio, such as notification information, to the user U1. The vibration generator 438 generates vibrations under the control of the processor 410 and converts notification information, such as notification information for the user U1, sent by the HMD1 into vibrations. The vibration generator 438 can reliably convey notifications to the user U1 by, for example, generating vibrations on the head of the user U1 wearing the HMD1 closely. Examples of notification information for the user U1 include notifications when an obstructing object appears, notifications informing the user of a change in display mode, notifications on how to change the display mode, and notifications of the presence of a shared user (described below). Such notifications can further improve usability.

[0059] The communication unit 439 has a communication interface for wireless communication with nearby information processing terminals such as other HMDs or smartphones, or external devices such as the information server 120 in FIG. 1, via short-range wireless communication, wireless LAN, or base station communication, and includes a communication processing circuit, antenna, and the like corresponding to various predetermined communication interfaces. While short-range wireless communication may be, for example, communication using an electronic tag, any communication that allows the HMD 1 to wirelessly communicate with nearby information processing terminals may be used. Examples of such communication interfaces include Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or wireless LAN such as Wi-Fi (registered trademark). For base station communication, long-range wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used.

[0060] The communication unit 439 may use other means, such as optical communication or sonic communication, as wireless communication means. In this case, instead of a transmitting / receiving antenna, a light emitting / receiving unit and a sonic wave output / input unit are used, respectively. Furthermore, when handling high-definition video, the amount of data is dramatically increased. In this case, using a high-speed, large-capacity communication network, such as 5G (5th Generation: 5th generation mobile communication system) or local 5G, for wireless communication can dramatically improve usability.

[0061] [Placement Data] The HMD 1 of the first embodiment may acquire and use placement data (i.e., spatial data) of external entities via communication from an external device such as the information server 120 of FIG. 1 . This placement data is data that indicates the placement (including position, shape, etc.) of individual entities in a three-dimensional space. For example, this placement data is data that includes various facilities and the like as individual entities within a map space. This placement data may also include attribute information and related information (e.g., facility names and descriptions) for each individual entity object. In another example, this placement data is data that includes individual entities such as walls and objects within a building space. When such placement data is available, it is generally easy to grasp the relationships, such as overlapping, between objects in a three-dimensional space. Therefore, the HMD 1 uses the placement data to more easily determine the boundaries of entities within the field of view, making it easier to extract and recognize individual entities.

[0062] [Component] The components realized based on processing by processor 410 in Figure 10 include a virtual object generation processing unit 411, a gaze point detection processing unit 412, a target object target visible range identification processing unit 413, an obstructing object discrimination processing unit 414, an object category processing unit 415, and an object display mode control processing unit 416.

[0063] The virtual object generation processing unit 411 generates virtual objects, which are objects in a virtual space that differs from the real space. Note that the HMD 1 may use data of virtual objects generated by an external device such as the information server 120 by wireless communication.

[0064] The gaze point detection processing unit 412 three-dimensionally calculates and detects the gaze point 106, which is the intersection of the gaze directions of both eyes in Figure 1 and the gaze point of user U1, from the gaze 104 of the left eye detected by the left eye gaze detection unit 432 and the gaze of the right eye detected by the right eye gaze detection unit 433.

[0065] The target object target visible range identification processing unit 413 determines the object where the gaze point is located, in other words, the target object which is the object closest to the gaze point, and identifies and determines the target visible range 107 (Figure 1), which is the range in which the user U1 is presumed to intend to view the target object.

[0066] The obstructing object discrimination processing unit 414 discriminates obstructing objects that overlap with the target visible range of the target object in the depth direction as seen from the user U1 and obstruct the target visible range by shielding it.

[0067] The object category processing unit 415 classifies objects into predetermined categories (in other words, types) according to the degree of restriction and tolerance for changes in the display mode of the object. The HMD 1 determines the method and details of changing the display mode according to the object category. The number and details of the categories are not limited.

[0068] The object display mode control processing unit 416 performs control processing to change the display mode of an object that is in an obstructing relationship with another object. The display mode change is at least one of moving the display position, adjusting the transparency, changing the size (reducing / enlarging), displaying a duplicate object, etc.

[0069] When an obstructing object determined by the obstructing object determination processing unit 414 obstructs at least a part of the target visible range of the target object identified by the target object target visible range identification processing unit 413, the HMD 1 controls the change of the display mode of the object in an obstructing relationship by the object display mode control processing unit 416. The object display mode control processing unit 416 changes the display mode of at least one of the obstructing object or the target object so as to eliminate or reduce the obstruction of the target object by the obstructing object. The object display mode control processing unit 416 determines the object to be changed, the display mode change method, etc., taking into account the categories of the objects before and after the obstructing object in an obstructing relationship, etc.

[0070] For example, when the obstructing object is a virtual object (pattern 2 / pattern 4 of FIG. 2 ) and the obstructing object has a lower degree of restriction than the target object, the object display mode control processing unit 416 changes the display position of the obstructing object or adjusts its transparency up so as to eliminate or reduce the obstruction of the target object by the obstructing object. Also, when the obstructed target object is a virtual object (pattern 3 / pattern 4) and the target object has a lower degree of restriction than the obstructing object, the object display mode control processing unit 416 changes the display position of the target object so as to eliminate or reduce the obstruction of the target object by the obstructing object. In this way, the obstruction of visibility of the target visible range of the target object by the obstructing object can be eliminated or the degree of obstruction of visibility can be reduced.

[0071] [detail] The details of the processing and display in the first embodiment will be described using Figures 11 and subsequent figures. Figure 11 shows an example of a display in the field of view 101 of the HMD 1, and schematically shows examples of individual entity objects, virtual objects, and target visual ranges. In (A) of Figure 11, an example of an entity object is a landscape viewed by the user U1 from a high place, which includes a tower 508, a building 500, and the like. The HMD 1 recognizes the tower 508, for example, as an individual entity object from this landscape. In the case of a video see-through type, the HMD 1 cuts out the tower 508 from the image of the landscape as an individual entity object. In the case of an optical see-through type, the HMD 1 recognizes the tower 508 from the landscape as an individual entity object. The above-mentioned placement data may be used when recognizing the tower 508, for example.

[0072] For example, when the HMD 1 focuses on a tower 508, which is an individual entity object, it generates an explanatory panel 503 and a guide map 504 as examples of virtual objects related to the tower 508 and displays them as shown in the figure, superimposed on a landscape including the tower 508. The explanatory panel 503 is a virtual object that displays explanatory information about the tower 508 (for example, its height is 634 m) as a speech bubble-like panel, for example. The explanatory panel 503 is positioned on the right side with the origin of the speech bubble in contact with the tower 508. The guide map 504 is a virtual object that provides guidance on the location of the tower 508 on the map. The guide map 504 is positioned in the upper left of the field of view 101.

[0073] Points of gaze 501, 502, and 507 are examples of points of gaze of user U1 with respect to this landscape. Point of gaze 507 is a case where the user is gazing at tower 508, which is an individual entity object. In response to the gaze at tower 508, HMD1 may display an explanatory panel 503, which is a virtual object. When the point of gaze is located on an entity object such as tower 508, as in point of gaze 507, HMD1 cuts out or recognizes the portion of the entity object, tower 508, from the landscape as an individual entity object based on analysis and placement data. Then, HMD1 determines the display range of the individual entity object, tower 508, indicated by a dashed line, as target visible range 509.

[0074] The gaze point 501 is when gazing at an explanation panel 503, and the gaze point 502 is when gazing at a guide map 504. The HMD1 sets a target visible range of the target object, with the object at which the gaze point of the user U1 is located as the target object. When the gaze point is located on a virtual object, the HMD1 determines the display range (corresponding image area) of the virtual object as the target visible range. For example, when the gaze point 501 is on the explanation panel 503, the display range indicated by the dashed line on the explanation panel 503 becomes the target visible range 505. When the gaze point 502 is on the guide map 504, the display range indicated by the dashed line on the guide map 504 becomes the target visible range 506.

[0075] In this example, each target visible range indicated by a dashed line is the same range that matches the shape and area of ​​the object on the display, but this is not limiting. The target visible range may be a range larger or smaller than the object. The target visible range may be a predefined size or shape (e.g., rectangle or ellipse). For example, when the target object is a building 500, the target visible range 511 indicates a case where an ellipse that roughly encompasses the building 500 is set as the target visible range.

[0076] 11B shows another example of setting the target visual range. The HMD 1 may perform control so that objects (virtual objects or individual entity objects) related to an object (virtual object or individual entity object) where the gaze point is located are included together in a single target visual range. In this example, for an individual entity object called tower 508 where gaze point 507 is located, explanation panel 503 is a related virtual object that is suitable for display together with tower 508. In this case, the HMD 1 sets a display range indicated by a dashed line in the figure, which combines target visual range 509 of tower 508 in (A) and target visual range 505 of explanation panel 503, as a single target visual range 510 for the two related objects (508, 503).

[0077] FIG. 12 shows another display example. Among objects (virtual objects or individual entity objects) overlapping in the user U1's line of sight, the relationship between the depth direction position of the gaze point and the object's position may be unclear, making it difficult or impossible to determine the target object (e.g., an object close to the gaze point) where the gaze point is located. In the example of FIG. 12, a tower 508, an individual entity object, and a guide map 504, a virtual object, overlap in the line of sight corresponding to the gaze point 507, and the guide map 504 partially obscures the tower 508. For example, suppose that the gaze point 507 is located approximately midway between the tower 508 and the guide map 504 in the depth direction of the gaze point 507. Therefore, the HMD 1 cannot determine which object is the target object. In this case, the HMD 1 selects and confirms the target object based on the object's visibility (in other words, importance) to the user U1, etc. For example, HMD1 compares multiple candidate objects (508, 504), prioritizes them in terms of visibility and importance, determines the object with the highest priority as the target object, and sets the display range of that target object as the target visibility range.

[0078] In this example, the criterion for determining the prioritization based on visibility value is to prioritize individual entity objects over virtual objects. Furthermore, individual entity objects are also compared by applying a general visibility value (for example, the prominence of the facility on the map). As a result, in this example, the HMD 1 determines that the tower 508 has a higher priority than the guide map 504, and sets the individual entity object, the tower 508, as the target object, and sets the target visible range 509. This allows the user U1 to optimally select and determine the target visible range of the target object that the user U1 wishes to view.

[0079] Note that in FIG. 1 and other figures, the gaze point 106 is information shown for explanation purposes and is not actually displayed on the display surface 11. As a modified example, the HMD 1 may display an image such as a mark representing the gaze point on the display surface 11 in accordance with the position of the gaze point 106. The image such as the gaze point mark may be an image separate from the pointer used for operation, or may be an image having the same function. The pointer is information for position specification by, for example, an OS or an application. The target object may be selected using an image such as a gaze point mark or a pointer.

[0080] [Object Category] In the first embodiment, objects are classified into three categories based on the object attributes used to control changes in display mode. The three categories are shown in (C) of FIG. 2. The first category is an object that has the highest degree of restriction on changes in display mode, and is an object that would cause a sense of incongruity if the display mode were changed, or an individual entity object. Examples of objects that would cause a sense of incongruity if the display mode were changed include an entity object to which a virtual object is fixed, or an entity object to which a virtual object is embedded and transformed into an integrated object. In the case of an optical see-through type, entity objects and individual entity objects are classified as the first category because it is difficult to change the display mode. An example of a virtual object being embedded or processed into an individual entity object is when a hole is represented as a virtual object and fixed or embedded in part of a real wall (corresponding individual entity object) in AR (augmented reality) or a video game. Since the wall and hole should be treated as a single entity and not separated, they are treated as related objects and classified as the first category, which has the highest degree of restriction.

[0081] The second category is an object that is subject to some restrictions regarding changes to its display mode, but has a lower degree of restriction and a higher degree of tolerance than the first category. The second category includes, for example, virtual objects such as the explanation panel 503 (FIG. 11) that is displayed in association with the first category virtual objects or individual entity objects.

[0082] The third category includes objects with even lower restrictions and higher tolerance than the second category, in other words, the least restrictive of the three. The third category includes, for example, virtual objects such as a guide map 504 (FIG. 11) that have no or few restrictions on their display positions relative to real objects or other virtual objects. The third category includes independent virtual objects and objects that can be moved naturally to a display position where the user U1 can view the object he or she wants to view.

[0083] In the example of Figure 11 (A), tower 508, which is an individual entity object, is in the first category. Explanation panel 503, which is a virtual object, explains tower 508 and has some display position restrictions, such as being suitable for display in a position close to tower 508, so it is in the second category. Guide map 504, which is a virtual object, is an object that does not look unnatural even when moved, so it is in the third category.

[0084] The HMD 1 may perform a display mode change process according to the object category classification by the object category processing unit 415. For objects in an occlusion obstruction relationship, the HMD 1 compares the categories of the target object and the obstructing object according to the degree of restriction on the display mode change. Based on the comparison result, the HMD 1 determines the object to be changed and the method and details of the display mode change.

[0085] For example, if the target object is in a category that is not less restrictive than the obstructing object (i.e., the same or higher), the object display mode control processing unit 416 changes the display mode of the obstructing object. On the other hand, if the target object is in a category that is less restrictive than the obstructing object, the object display mode control processing unit 416 changes the display mode of the target object. As a result, the HMD 1 can eliminate or reduce the obstruction of visibility of the target visible range of the target object in an optimal manner according to the restriction level for each object. Furthermore, the HMD 1 can minimize the discomfort in visibility that accompanies the change in display mode for both the target object and the obstructing object.

[0086] In the above method and criteria, if the restriction level of the target object and the restriction level of the obstruction object are the same, the display mode of the obstruction object is changed. This is not limited to this, and other methods and criteria may be used to change the display mode of the target object if the restriction level of the target object and the restriction level of the obstruction object are the same. In this case, the method prioritizes maintaining the display mode of the obstruction object in the front, which is closer to the user U1.

[0087] [First pattern] Here, a processing example for the case where an entity object occludes another entity object (pattern 1 in FIG. 2) will be described. First, if there is no appearance information for the occluded portion of the entity object behind the entity object, the HMD 1 processes it as not being occluded. In this case, since there is no occlusion obstruction relationship, no change in the display mode occurs. In the flow of FIG. 9, as an exception processing, in step S3, it is treated as not being occluded (N). Also, if there is appearance information for the occluded portion of the entity object behind the entity object, for example, if that appearance information is obtained from the above-mentioned placement data, the HMD 1 processes it as being occluded (Y). In other words, in step S3 of the flow of FIG. 9, it is treated as being occluded (Y). That is, the HMD 1 sets the individual entity object corresponding to the portion of the entity object behind the entity object as the target object. In this case, as a display mode change, the HMD 1 generates a duplicate object by duplicating the appearance of the individual entity object that is the target object, and displays the duplicate object in an empty position, for example, as in FIG. 8. This allows the user U1 to see the part of the actual object that is covered by looking at the duplicate object.

[0088] When the HMD1 prioritizes the display of the occluded target object, it may use a method of displaying a duplicate object directly at the occluded position. In this method, the duplicate object is displayed superimposed in front of the actual object that is the occluding obstructing object. This is similar to the method of increasing the transparency of the occluding obstructing object (Figure 3).

[0089] On the other hand, when HMD1 prioritizes maintaining the display of the obstructing object in front closer to user U1, it cannot move the individual entity object, which is the target object in the rear, so it uses a method of displaying a duplicate object in another vacant position (Figure 8).

[0090] In either case, when a physical object is displayed as a video image in a video see-through type, the HMD1 may change the display mode of the individual physical object by processing the individual physical object cut out from the video image and treating it as a virtual object.

[0091] [Example (1)] An example of the operation of the HMD 1 according to the first embodiment will be described with reference to FIG. 13 and so on. FIG. 13 shows a processing flow for the example of the operation of FIG. 11 and so on. FIG. 13 shows a more detailed processing example than FIG. 9, and includes steps S601 to S613. FIG. 13 particularly shows the details of steps S2 and S4 of FIG. 9. In step S601, the HMD 1 detects the gaze point of the user U1 using the attention point detection processing unit 412, and determines whether there is an object located at the gaze point. If there is an object located at the gaze point, in other words, if one object within a predetermined distance range is determined (Y), then in step S602 the HMD 1 determines that object as the target object.

[0092] Next, in step S2, the target object target visible range identification processing unit 413 starts a target visible range selection process for the target object. This process is made up of steps S603 to S609. In step S603, the HMD 1 determines whether there is an object overlapping in the line of sight of the gaze point. If there is not (N), the process proceeds to step S604, and if there is (Y), the process proceeds to step S609. In step S604, the HMD 1 regards the object overlapping in the line of sight of the gaze point as a target object, and determines whether the gazed target object is an actual object (corresponding individual actual object) or a virtual object. If the target object is an actual object (A), the process proceeds to step S605, and if the target object is a virtual object (B), the process proceeds to step S606.

[0093] In step S605, the HMD 1 identifies and selects an individual real object, which is individually extracted or recognized from the real object, as the target visible range of the target object. In step S606, the HMD 1 identifies and selects the display range of the virtual object as the target visible range of the target object.

[0094] Here, in step S607, the HMD 1 determines whether there is an object related to the target object, which is a real object (S605) or a virtual object (S606). As described above, the related object is a virtual object or the like whose display position should be linked. If there is a related object (Y), in step S608, the HMD 1 identifies and selects the target object and its related object as the target visible range of one target object ((B) of FIG. 11).

[0095] Meanwhile, in step S609, the HMD 1 selects one object from among the multiple objects overlapping in the line of sight of the fixation point according to a predetermined criterion, sets it as a target object, and identifies and selects the target visible range of the target object. At this time, in the first embodiment, the above-mentioned visibility value and importance are used. The HMD 1 sets the object with the highest visibility value and importance from among the multiple overlapping objects as the target object, and identifies and selects the display range of the target object as the target visible range. As a result of the above step S2, the target visible range of the target object is determined.

[0096] Thereafter, in step S3, the HMD 1 determines whether or not there is a virtual object (sometimes referred to as an "obstructing virtual object") that acts as an obstructing object blocking the target visible range of the target object, using the obstructing object determination processing unit 414. If there is an obstructing virtual object (Y), the process proceeds to step S4, and if there is not (N), the process skips step S4. In the first embodiment, if there is a virtual object that blocks at least a part of the target visible range, the HMD 1 determines it as an obstructing virtual object and proceeds to step S4.

[0097] Step S4 includes steps S611 to S613. In step S611, the HMD 1 determines, using the object category processing unit 415, whether the target object has a higher restriction level than the obstructing virtual object, i.e., whether the target object is in a higher category than the obstructing virtual object. For example, if the target object is in the first category and the obstructing virtual object is in the second category, the former is in a higher category. If the target object is in a higher category than the obstructing virtual object (Y), the process proceeds to step S612; otherwise (N), the process proceeds to step S613. In step S612, the HMD 1 changes the display mode of the obstructing virtual object by moving the display position or adjusting the transparency, as described above, using the object display mode control processing unit 416. In step S613, the HMD 1 changes the display mode of the target object by moving the display position, etc., using the object display mode control processing unit 416. As a result, the entire target visible range becomes visible. Then, the process proceeds to step S5, as described above.

[0098] [Example (2)] 14 shows an example of operation in the second pattern. In the state before the change (a), the target object is a tower 508, which is an individual entity object of the first category, and the obstructing object is a guide map 504, which is a virtual object of the third category, in terms of the occlusion obstruction relationship. The target visible range 509 of the tower 508, where the gaze point 507 is located, is partially obstructed by the guide map 504. In this case, the HMD 1 selects the guide map 504, which has a lower degree of restriction and is in a lower category, as the target for change, and adjusts, for example, the transparency up. As a result, in the state after the change (b), the guide map 504 becomes transparent, and the entire target visible range 509 of the tower 508, which is the target object, can be seen.

[0099] 15 shows another example of operation in which the display position is moved as a change in the display mode. In the same case as in (a) of FIG. 14, the HMD 1 moves the display position of the guide map 504 of the lower category to a position outside the target visible range 509 of the tower 508. This allows the entire target visible range 509 of the tower 508, which is the target object, to be visible without any obstructions.

[0100] [Example (3)] Fig. 16 shows an example of operation in the case of the third pattern. Fig. 16 shows the opposite case to Fig. 14 etc., in which the target object is a guide map 504, which is a virtual object of the third category, and the obstructing object is a tower 508, which is an individual entity object of the first category. In (a), the target visible range 506 of the guide map 504, where the gaze point 502 is located, is partially obstructed by the tower 508. In this case, as shown in (b), the HMD 1 moves the guide map 504, which is of a lower category, to a position outside the tower 508 so that the tower 508 does not overlap with the target visible range 506. This makes it possible to view the entire target visible range 506 of the guide map 504, which is the target object, without any obstructions.

[0101] [Example (4)] 17 shows an example of operation in the fourth pattern. The target object is an explanatory panel 503, which is a virtual object of the second category, and the obstructing object is a guide map 504, which is a virtual object of the third category. In (a), the target visible range 505 of the explanatory panel 503 is partially obstructed by the guide map 504. In this case, the HMD 1 increases the transparency of the guide map 504, which is of a lower category, as shown in (b). This makes the guide map 504 transparent, allowing the entire target visible range 505 of the explanation panel 503, which is the target object, to be seen.

[0102] 18 shows another example of operation in the case of moving the display position. In the same case as in (a) of FIG. 17, the HMD 1 moves the display position of the guide map 504 of the lower category to a position outside the target visible range 505. This makes it possible to see the entire target visible range 505 of the explanation panel 503, which is the target object, without any obstructions.

[0103] [Example (5)] Fig. 19 shows another example of operation. Fig. 19 shows the opposite case to that of Fig. 17, in which the target object is a guide map 504, which is a virtual object of the third category, and the obstructing object is an explanation panel 503, which is a virtual object of the second category. In (a), the target visible range 506 of the guide map 504 is partially obstructed by the explanation panel 503. In this case, the HMD 1 moves the guide map 504, which is of a lower category, and changes it to a position where the explanation panel 503 and other objects do not overlap within the target visible range 506. This makes it possible to view the entire target visible range 506 of the guide map 504, which is the target object, without any obstructions.

[0104] When the target object is in the first category and the obstruction object is in the second category, the same control as when the target object is in the first category and the obstruction object is in the third category can be applied.

[0105] [Effects etc. (1)] As described above, according to the first embodiment, in the HMD1 capable of displaying virtual objects arranged in three dimensions, when the viewable range of an object, such as a real object or a virtual object, that the user U1 wants to view is obstructed by, for example, another object, the viewable range can be resolved or reduced by changing the display mode, allowing the user U1 to view the entire object favorably. Furthermore, such a function can be realized with little effort on the part of the user and in a convenient manner. According to the first embodiment, even when there is an obstruction between objects, the user can favorably view the entire target viewable range of the target object that the user wants to gaze upon. According to the first embodiment, the display mode can be automatically changed in accordance with the obstruction to support the user's view, allowing such a function to be realized with little effort on the part of the user and in a convenient manner.

[0106] In Patent Document 1, when there is an object that obstructs the visibility of the background in the line of sight, the display mode of the object is changed. In contrast, in Embodiment 1, when there is an obstructing object that obstructs the visibility of the target visible range of a target object due to an obstructing relationship between objects arranged three-dimensionally, the display mode of the obstructing object or the target object is changed so that the entire target visible range can be viewed.

[0107] [Variations] The following modification of the first embodiment is also possible. When determining a target object based on the line of sight, the HMD 1 may determine that there is a target object that the user U1 wants to gaze at if the movement in the line of sight direction is equal to or less than a predetermined threshold. This eliminates erroneous processing due to unintentional rapid eye movement and makes it possible to more accurately identify the target object. An example of erroneous processing is when the gaze point is positioned on an object in a short period of time, and the object is mistakenly identified as the target object.

[0108] Furthermore, when setting the target visible range, the HMD 1 may determine the size or area of ​​the image region of the object and set an upper limit. If the target visible range is larger or wider than a predetermined threshold, the HMD 1 may set an upper limit range corresponding to the predetermined threshold as the target visible range. For example, if the target object is too large in the field of view, it may be difficult to display it when changing the display mode to move an obstructing object outside the target visible range. In such cases, setting an upper limit for the target visible range is effective.

[0109] <Embodiment 2> The second embodiment will be described with reference to FIG. 20 etc. The second embodiment has the following additional functions compared to the first embodiment. In the field of view, there may be cases where an object that is a candidate for a target object (sometimes referred to as a target candidate object) is occluded by another object, that is, a virtual object or an individual entity object, and the user is unaware of its existence. This function is a function that can confirm the existence of the target candidate object in such cases.

[0110] [Check for the existence of target candidate objects (1)] FIG. 20 is an explanatory diagram of an example of operation in the second embodiment. FIG. 20 shows an example of a change in display mode when a certain object (target candidate object) is occluded by a real object. (a) shows the state before the change. In the field of view 101, a tower 508, which is an individual real object, and an explanation panel 1213, which is a virtual object, are displayed. In addition, objects located in the direction of the gaze point 1201 include the tower 508 and a guide map 1202 (shown by a dotted line), which is a virtual object that is hidden by the tower 508 and cannot be seen. In other words, the guide map 1202 is an invisible target candidate object. In this case, the HMD 1 changes the display mode of the objects (508, 1202) in the line of sight direction, as shown in (b). In this example, the HMD 1 moves the display position of the guide map 1202, which is a target candidate object, so that at least a portion of the guide map 1202 is visible and not occluded by the tower 508. In (b), at least a portion of the changed guide map 1203 is visible to the user U1. The HMD1 may change the display range corresponding to the target candidate object so that the entire display range is visible. Alternatively, the HMD1 may change the display range so that a predetermined percentage of the display range corresponding to the target candidate object is visible. This allows the user U1 to recognize and confirm the existence of the guide map 1202. This allows the user U1 to select the guide map 1202 as a target object using the fixation point (fixation point 1201b after movement).

[0111] FIG. 21 shows another display example. In the state before the change shown in (a), objects located in the direction of the gaze point 1211 include an explanatory panel 1213, which is a virtual object related to the tower 508, and a guide map 1212, which is a virtual object hidden by the explanatory panel 1213 and is therefore invisible. That is, the guide map 1212 is an invisible target candidate object. In this case, the HMD 1 changes the display mode of the objects (1212, 1213) located in the line of sight. In this example, as shown in (b), the HMD 1 changes the display mode of the explanatory panel 1213 in front of the guide map 1212, which is a target candidate object, by, for example, increasing its transparency. As a result, the guide map 1212 behind the explanatory panel 1213 becomes visible through the explanatory panel 1213. This allows the user U1 to confirm the existence of the guide map 1212, which is a target candidate object, and enables the user U1 to select the guide map 1212 as a target object using the gaze point 1211.

[0112] FIG. 22 is a supplementary explanatory diagram for a case where multiple objects overlap in the line of sight, making it difficult to determine the target object where the point of gaze is located, as in the example of FIG. 21, etc. FIG. 22 schematically shows the overlap of objects in the depth direction (Z direction) when user U1 views the field of view 101. An explanatory panel 1233 is arranged in front of user U1's viewpoint, closer to the viewpoint, and a guide map 1232 is arranged behind it. The guide map 1232 corresponds to the guide map 1212, which is a target candidate object in FIG. 21. The explanatory panel 1233 corresponds to an obstructing object in terms of occlusion obstruction. In FIG. 22, the guide map 1212, which is a target candidate object where the point of gaze 1231 is located, is obstructed by the explanatory panel 1233, which is an out-of-focus obstructing virtual object, and is hidden and invisible to user U1, as in FIG. 21. An out-of-focus object is an object where the point of gaze is not located or an object that is far away from the point of gaze.

[0113] The fixation point 1231 is an example of a fixation point calculated from the line of sight (104, 105) of both eyes. In this example, the fixation point 1231 is located near the guide map 1232 in the depth direction (Z direction). Of the distances from the fixation point 1231 to objects, the distance to the guide map 1232 is the shortest and is within a predetermined distance range. Therefore, the guide map 1232 becomes a target candidate object.

[0114] In this case, the HMD1 detects from the direction of the line of sight and the point of gaze that the explanation panel 1233 is out of focus. To confirm the presence of the target candidate object, the HMD1 changes the display mode of the out-of-focus explanation panel 1233, for example, by adjusting the transparency up (similar to FIG. 21). As a result, the changed explanation panel 1233b becomes transparent (high transparency), allowing the guide map 1232 behind it to be seen, and the user U1 can confirm the presence of the guide map 1233, which is the target candidate object.

[0115] [Check for the existence of target candidate objects (2)] Fig. 23 shows a processing flow relating to the function of confirming the presence of the target candidate object of the HMD 1 in embodiment 2. The flow in Fig. 23 differs from the above-described flow in that it includes steps S1100 to S1104, which are performed as preprocessing for step S2 in Fig. 9.

[0116] In step S1100, the HMD 1 checks whether the mode corresponding to this function is on (enabled), and if it is on, performs the following process. The mode can be set or instructed by the user U1 via the operation input unit 435, for example.

[0117] In step S1101, the HMD1 determines whether there are any target candidate objects, which are objects that are occluded by other objects (virtual objects or individual entity objects) across the entire display surface 11 and therefore whose existence is unknown to the user U1. These target candidate objects are objects that are invisible to the user U1 and therefore cannot be recognized, nor can they be selected using the gaze point. If there are any such target candidate objects (Y), the process proceeds to step S1102; if there are no such target candidate objects, the process proceeds to step S2.

[0118] In step S1102, the HMD 1 checks and waits for a trigger to perform the presence confirmation process. This trigger allows the user U1 to instruct whether or not to perform the presence confirmation process. This trigger can be, for example, when an instruction is input via the operation input unit 435 or the microphone 436, or when the detected line of sight of the user U1 moves near a target candidate object. For example, as shown in FIG. 20, the HMD 1 may display a guide or button in the field of view 101, such as "There is a hidden object. Would you like to display it and check it?", and the pressing of the button may be used as a trigger. It is also possible to omit step S1102 for inputting the trigger and automatically perform the object presence confirmation process.

[0119] When the above trigger occurs, in step S1103, the HMD1 changes the display mode of the target candidate object (such as moving the display position or displaying a duplicate), or changes the display mode of the obstructing object (such as adjusting the transparency up), using the object display mode control processing unit 416. In step S1104, the HMD1 maintains the state after the change in display mode for a certain period of time. This allows the user U1 to confirm the existence of the target candidate object. The user U1 can easily confirm the existence of the target candidate object without any special operation. After step S1104, the process returns to step S2 described above.

[0120] As described above, in the function of the second embodiment, when there is an object whose existence is unknown because it is obscured by another object, the display mode of at least one of the target candidate object or the obstructing object is changed so that at least a part of the target candidate object is visible. This allows the user U1 to reliably confirm the target candidate object and select it as the target object. Note that when there is an individual entity object that is hidden and cannot be seen, and appearance information of that individual entity object is available, that individual entity object may also be treated as a target candidate object and its existence may be confirmed in the same way.

[0121] [Effects etc. (2)] As described above, according to the second embodiment, even when there is a hidden target candidate object that cannot be seen, the presence of the hidden target object can be confirmed by a kind of change in the display mode, and the user U1 can select the target object. Note that if the HMD1 is capable of detecting only one gaze direction of the user U1, it is difficult to determine the gaze point in the depth direction. In this case, in the second embodiment, the display mode is changed so that the hidden object on the display surface is visible, that is, by making it so that there is only one object in the depth direction on the display surface, it is possible to determine the object in that gaze direction as the target object even if there is only one gaze direction.

[0122] [Object data and attribute information] The above describes a case in which the HMD 1 determines the control content for changing the display mode of each object by referring to parameters such as categories related to the degree of restriction (in other words, tolerance) and visibility (in other words, importance). The degree of restriction and tolerance in the above categories are pieces of attribute information that represent the degree of restriction or tolerance for changing the display mode for each object. In the data for each object, such degree of restriction or tolerance, category, or other information may be set as one piece of attribute information. Furthermore, as examples of other information in the attribute information for each object, visibility, importance, priority, etc. may be set for each object. These parameters may be managed and stored as data by the HMD 1 or an external device.

[0123] FIG. 24 shows an example of object data managed and held by the HMD 1. This object data is management data including attribute information for each object. This object data may be management information that differs for each application, or management information or user setting information that differs for each user. The HMD 1 may generate and set each piece of information for the object data by itself, or may refer to that information from an external device (for example, the information server 120 in FIG. 1).

[0124] The object data table in Figure 24 has columns for ID, object, type, category, visibility value, related object, and shared user. "ID" is an identifier for each "object." Examples of "object" are the tower and explanatory panel mentioned above. "Type" here has two types: A. individual entity object and B. virtual object. As mentioned above, there are three categories (corresponding restriction levels): 1 (high), 2 (medium), and 3 (low). There are three visibility values ​​(corresponding importance levels): 1 (high), 2 (medium), and 3 (low). "Related object" indicates the relationship with other objects. For example, an object with ID=1 has an object with ID=2 as a related object, and an object with ID=2 has an object with ID=1 as a related object. "Shared user," as will be described later, indicates the identifier of a shared user when multiple users share the object.

[0125] In this example, the smaller the number for "viewability," the higher the value. The "category" and "viewability" may be set by the HMD1 or by the user. For example, the HMD1 may set the "viewability" based on the general prominence of the object. Alternatively, the HMD1 may set the "viewability" according to the user U1's level of interest in the object. For example, in the aforementioned FIG. 11, a "viewability" is set for each individual entity object, such as a facility on the map, based on general prominence. The "category" may be determined by combining the "restriction level" and "viewability."

[0126] Furthermore, during control processing, the HMD 1 processes and stores object information separately from the object data as appropriate for each point in time. This object information includes, for each object, information such as the display position on the display surface 11, the orientation of the three-dimensional object's three-dimensional arrangement, the display range (image area), the target visual range, and the display mode change status. The display mode change status includes whether or not a change has been made, the change method, etc. The HMD 1 controls the object display using the above-mentioned object data, object information, line of sight, gaze point, etc.

[0127] <Third Embodiment> The third embodiment will be described with reference to Fig. 25 etc. The third embodiment has a function of changing the display mode of a shared object between shared users.

[0128] Shared Users and Objects FIG. 25 is an explanatory diagram illustrating an example of the operation of the HMD1 according to the third embodiment. In FIG. 25, there are multiple (e.g., two) users (U1, U2) each wearing an HMD1. The first user U1 uses an HMD1A, and the second user U2 uses an HMD1B. These users (U1, U2) share virtual objects, an "A" object 103 and a "B" object 102. The users U1 and U2 are shared users who share these virtual objects. The "A" and "B" objects are shared objects shared by the shared users (U1, U2), respectively. Communication 2500 for sharing is performed between the HMD1s (1A and 1B) of the shared users (U1, U2) by the short-range wireless communication described above.

[0129] FIG. 25 shows a first example of the display, visibility, and other states. Shared users (U1, U2) are looking at shared objects (A, B). In particular, at a certain point in time, the first user U1 is looking at the object “A” 103, where the gaze point P1 of the line of sight E1 is located, as a target object in the field of view 101A. The HMD 1A sets a target visible range 107A for the object “A” 103. The second user U2 is looking at the object “B” 102, where the gaze point P2 of the line of sight E2 is located, as a target object in the field of view 101B. The HMD 1B sets a target visible range 107B for the object “B” 102. In the first example, there is an occlusion obstruction relationship as seen from the second user U2, where the object “B” 103 on the rear side viewed by the second user U2 is the target object, and the object “A” 102 on the front side viewed by the first user U1 is the obstructing object.

[0130] Note that lines of sight E1 and E2 represent the lines of sight (104, 105) of both eyes combined into one line. In Fig. 25, the display content in field of view 101A seen by user U1 and the display content in field of view 101B seen by user U2 are illustrated as being the same, but since the viewpoint positions of each user are different, the actual display content, i.e., the way the objects are seen, are different.

[0131] The next Fig. 26 shows a second example. Fig. 26 shows a case where, contrary to the first example in Fig. 25, the first user U1 is looking at the "B" object 102 on the rear side, and the second user U2 is looking at the "A" object 103 on the front side. In the second example, there is an occlusion obstruction relationship as seen from the first user U1, and the "B" object 103 on the rear side viewed by the first user U1 is the target object, and the "A" object 102 on the front side viewed by the second user U2 is the obstructing object.

[0132] [Display example (3-1)] In the above case, the HMD1 changes the display mode of the shared object of the shared user. First, FIG. 27 shows a change example corresponding to the first example in FIG. 25. (a) shows the state before the change of the display mode, as viewed from the first user U1, of the image in the field of view 101A, and (c) shows the state after the change. (b) shows the state before the change of the display mode, as viewed from the second user U2, of the image in the field of view 101B, and (d) shows the state after the change. In (a), the HMD1A of the first user U1 generates and displays mark information m2 in the field of view 101A based on communication 2500 with the HMD1B of the second user U2. This mark information m2 is an image indicating which object the second user U2 is looking at, i.e., which target object the second user U2's gaze point P2 is located at. For example, the HMD1B of the second user U2 transmits information indicating that the target object is "B" to the HMD1A, and the HMD1A of the first user U1 transmits information indicating that the target object is "A" to the HMD1B. In response to the information from the HMD1B, the HMD1A generates a mark with the number "2" representing the second user U2 as mark information m2, for example, and displays it near the object "B", which is the target object of the second user U2. This allows the first user U1 to recognize which shared object the second user U2, who is a shared user, is looking at.

[0133] Similarly, in (b), the HMD 1B of the second user U2 generates and displays mark information m1 representing which object the first user U1 is looking at in the field of view 101B based on communication 2500 with the HMD 1A of the first user U1. In response to the information from the HMD 1A, the HMD 1B generates, as mark information m1, a mark with the number "1" representing the first user U1, and displays it near the object "A" that is the target object of the first user U1. This allows the second user U2 to recognize which shared object the first user U1 is looking at.

[0134] Furthermore, the HMD1 (1A, 1B) of each shared user (U1, U2) may change the display mode of the object depending on the visibility relationship of the shared object between the shared users and the occlusion obstruction relationship as described above. Examples are shown in (c) and (d). First, an example of a change from (a) to (c) is as follows. In the state (a), the object "B" viewed by the second user U2 is occluded behind the target visibility range 107A of the target object "A" viewed by the first user U1. Since the first user U1 can view the target object "A" without any occlusion obstruction, the HMD1A leaves the display of the object "A" unchanged. The object "B" viewed by the second user U2 may be displayed as is, but is partially occluded from the view of the first user U1. Therefore, in this example, the HMD1A changes the display mode of the target object "B" of the shared user so that the part where the second user U2 is viewing the object "B" is easily visible to the first user U1. (c) shows an example in which the display position of object "B" is moved so that the entire object is visible. This change in display mode may not be automatic, but may be made in response to a predetermined input operation by the first user U1. For example, the HMD 1A may display a message such as "Do you want to check the object that the shared user (2) is looking at?" on the display surface 11, and may make a change as shown in (c) in response to a button press operation by the user U1.

[0135] Next, an example of a change from (b) to (d) is as follows. In the state (b), part of the target object "B" that the second user U2 is looking at is obscured by the object "A" that the first user U1 is looking at. Therefore, in this example, the HMD1B changes the display mode of the target object "A" of the first user U1, which is an obscuring object, so that the entire target object "B" becomes visible. (d) shows an example of moving the display position of the object "A." This allows the second user U2 to confirm the target object "B."

[0136] As described above, various methods such as increasing transparency (FIG. 3) and duplicate display (FIG. 8) can be applied to change the display mode of a shared object. (e) shows another display example, in which the state seen by the first user U1 in (a) is changed to another state after the display mode is changed. In the case of (a), the HMD1A changes the state so that the entirety of the object "B" is visible by increasing the transparency of the object "A," as in (e). This change may be made in response to a predetermined operation, as in the case of (c). This allows the first user U1 to check not only the target object "A" but also the target object "B" of the second user U2.

[0137] (f) shows another display example after changing the display mode from the state seen by the second user U2 in (b). In the case of (b), the HMD 1B changes the state so that the entire target object "B" can be seen by increasing the transparency of the object "A," as in (f).

[0138] As described above, in the third embodiment, when the visual recognition states of gazes on a shared object differ among the shared users, mark information indicating the visual recognition states is displayed. This allows the shared users to check the shared object that each shared user is gazing at and recognizing. This facilitates communication and work between the shared users. In the second example of FIG. 26, the same control as in the first example can be applied.

[0139] [Display example (3-2)] FIG. 28 shows another display example. (a) shows a state in which the first user U1 is looking at the object "A" in front, similar to the first example in FIG. 25 and (a) in FIG. 27. (b) shows a state in which the second user U2 is looking at the object "B" in the back, similar to (b) in FIG. 27. In the case of (a), the HMD 1A changes the display mode as shown in (c). The HMD 1A leaves the display of the target object "A" of the first user U1 and the target object "B" of the second user U2 unchanged, and generates a duplicate object 102r of the object "B" in an open position that is not obscured so that the entirety of the obscured object "B" can be confirmed. It is more preferable to determine the position in which the duplicate object 102r is displayed so that it corresponds to the direction in which the second user U2 is located (to the right in this example). The duplicate object 102r may also be displayed with mark information m2 indicating a shared user gaze object. This allows the first user U1 to check not only the target object "A" but also the entire target object "B" of the second user U2.

[0140] Similarly, in the case of (b), the HMD 1B changes the display mode as shown in (d). The HMD 1B generates a duplicate object 102r (which looks different from the duplicate object 102r in (c)) for the partially occluded target object "B" and displays it in an empty position. Alternatively, as another method, the HMD 1B may leave the target object "B" as is and change the display position of the object "A", which is an obstructing object, in the same manner as described above.

[0141] (e) and (f) are other display examples. (e) is the state as seen from the first user U1. When viewed from the first user U1, the object "B" viewed by the second user U2 is partially occluded. The HMD1A changes the display mode of the shared user's object "B" in the same manner as above. On the other hand, (f) is the state as seen from the second user U2. When viewed from the second user U2, the target object "B" is not occluded by the object "A" and can be seen in its entirety. Therefore, the HMD1B does not change the display mode.

[0142] [Display example (3-3)] FIG. 29 shows another display example in which the display mode is changed when shared users (U1, U2) are looking at the same shared object (e.g., object 102 of "B"). (a) shows the state as seen from the first user U1, and (c) shows the state as seen from the second user U2. In (a), the first user U1 is looking at the object "B" behind it from the right side of the object "A". The target object "B" is partially hidden by the object "A". In (c), the second user U2 is looking at the object "B" behind it from the left side of the object "A". The target object "B" is partially hidden by the object "A". In the state of (a), the HMD 1A displays mark information m2 on the object "B" indicating that the second user U2 is also looking at it. In the state of (c), the HMD 1B displays mark information m1 on the object "B" indicating that the first user U1 is also looking at it. (b) and (d) are examples after the display mode has been changed. (b) is an example of changing the display position of the obstructing object "A" to, for example, a position on the left side. (d) is an example of changing the display position of the obstructing object "A" to, for example, a position on the right side.

[0143] As described above, in the third embodiment, when the shared users are in a visual recognition state in which they are gazing at the same shared object, mark information indicating the visual recognition state is displayed, thereby enabling the shared users to confirm the shared object that they are gazing at and visualizing.

[0144] In the above example, the occlusion obstruction relationship is the fourth pattern described above (Figure 2), but this is not limiting and similar control is possible for other patterns. The above mark information can also be displayed when an individual entity object is a shared object. The display mode can be changed for objects other than optical see-through entity objects. The above-mentioned restriction level and visibility value can also be applied to shared objects.

[0145] [Effects etc. (3)] As described above, according to the third embodiment, a display mode change appropriate for each user's HMD 1 is performed on the shared object of the shared users. This allows each user to reliably view the shared object without any visual confusion while eliminating or reducing visual obstruction caused by occlusion between objects. In the third embodiment, at least one HMD 1 between the shared users displays a mark representing the shared user's gaze object and changes the display mode according to the relationship between visibility and occlusion obstruction. When changing the display mode, the method and details are determined taking into consideration not only the occlusion obstruction relationship, restriction level, and visibility value described above, but also the visibility relationship, i.e., which shared object the shared user is viewing. For example, in the example of FIG. 27 , considering the HMD 1A side of the first user U1, between the target object “A” in front of the first user U1 and the object “B” in the back of the second user U2, the first priority should be given to displaying the entire target object “A.” In the state (a), the entire object can be seen, so in the case of the first embodiment described above, there is no need to change the display mode. The object "B" is not an obstructing object, but is a target object that the shared user is looking at. Therefore, in the case of the third embodiment, the display mode can be changed so that the entire object "B" can also be seen. When making this change, for example, a method or details are selected so that the entire objects of both objects "A" and "B" can be seen. For example, in the case of the transparency adjustment method (e), part of the target object "A" temporarily becomes transparent and becomes somewhat difficult to see, so that a more suitable method may be selected, such as the display position movement method (c) or the duplicate display method (c) in Figure 28.

[0146] [Variation (3-1)] In the third embodiment, the mark information representing the object gazed upon by the shared user is separate from the gaze point. The mark information is displayed in an area of ​​the target visible range of the object gazed upon by the shared user other than an area blocked by other objects. If the mark information is displayed in an blocked area, it becomes unclear whether the object being gazed upon is the front or rear object, so this display method clarifies this. Furthermore, the HMDs 1 of the shared users may constantly communicate with each other to update the display state (including the mark information) in approximately real time, or may communicate periodically to update the display state periodically.

[0147] As a modified example, the display position of the mark information representing the shared user's gaze object may be aligned with the gaze point. As another modified example, a mark representing the gaze point may be displayed in the field of view at a position corresponding to the gaze point in the field of view. A pointer for selection operation using a remote controller or the like may be displayed in the field of view. FIG. 30 shows an example of a modified example in which, in addition to the mark representing the shared user's gaze object, a mark 3001 representing the gaze point and a pointer 3003 are displayed in the field of view 101. For example, a diamond-shaped mark 3001 is displayed at the position of the gaze point P1 of the first user U1. For example, a triangular mark 3002 is displayed at the position of the gaze point P2 of the second user U2. Furthermore, for example, a cross-shaped pointer 3003 is displayed as a pointer for operation by the first user U1.

[0148] [Variation (3-2)] FIG. 31 shows a display example in another modified example. (a) shows the state as seen from the first user U1, similar to (a) in FIG. 27 described above. The first user U1 is looking at the target object "A" in the foreground. (b) shows the state as seen from the second user U2. The second user U2 is looking at the object "B" in the background from a different line of sight from the line of sight of the first user U1, for example, from a direction that is 90 degrees different. Here, the shape and location of the shared object "B" as seen from the first user U1 are different from those as seen from the second user U2. In (b), the sides of the objects "A" and "B" as seen from the second user U2 are illustrated as "A#" and "B#."

[0149] (c) and (d) show the state after the display mode has been changed. In (c), the HMD 1A displays the target object "A" as is, but changes the display mode of the partially occluded object "B" that the shared user is looking at, so that the user can confirm it. At this time, the HMD 1A changes the display mode of the object "B" so that it appears in the shape and location as seen by the second user U2, as shown in (b). In this example, the HMD 1A leaves the object "B" as is, and generates and displays a duplicate object 3101 of the object "B" along with a speech bubble in an empty position. This duplicate object 3101 is generated as a duplicate object that looks the same as the object 3102 in (b). This allows the first user U1 to confirm the entire shared object "B," particularly as seen by the second user U2. The state as seen by the second user U2 in (b) may be displayed as is, since there is no occlusion obstruction, or it may be displayed as shown below. As shown in (d), the HMD1B changes the display mode of the object "A" that the first user U1 is looking at, in the same manner as above, so that the shape and location that the first user U1 is looking at are visible. In this example, the object "A" 3103, which appears as in (a), is generated and displayed by superimposing it in front of the object "A."

[0150] <Fourth Embodiment> A fourth embodiment will be described with reference to FIG. 32 and the like. In the fourth embodiment, a modified example of the target object determination method is shown. In the above-described embodiment, the target object is determined and determined by detecting the gaze point from the user's line of sight. In this modified example, a selection input operation from the user is accepted for tags displayed attached to each object. In this way, the HMD determines the target object.

[0151] FIG. 32 shows a display example in the fourth embodiment. As described above, a tower 508, which is an individual entity object, and an explanatory panel 503 and a guide map 504, which are virtual objects, are displayed in the field of view 101 of the HMD 1 of the user U1. The HMD 1 displays each object in the field of view 101 with a tag attached. This tag is an image for identifying and selecting the object. In this example, the tags (701, 702, 703) have a rectangle connected by a line leading from the object and a number for identifying the object.

[0152] The user U1 performs an object selection input operation using a predetermined operation means provided in the HMD1. The predetermined operation means can be, for example, voice input, but is not limited to this, and various means can be applied, such as a pointer on a remote controller, gaze point detection by line of sight, and hand gesture recognition. An example of object selection by voice input is as follows: If the user U1 wants to select, for example, the tower 508 as an object, he or she inputs the number ("3") of the tag 703 attached to that object by voice. The HMD1 recognizes the number in the input voice and identifies the object associated with the tag with that number.

[0153] For example, in a situation where multiple objects are crowded together, it may be difficult to determine the target object using only the gaze point. In such a case, the accuracy of determining the target object can be increased by also using the tag selection input reception according to embodiment 4. Furthermore, by using the tag selection method according to embodiment 4, the above-mentioned display mode change function and other functions can be applied even to a device that does not have a gaze point detection function as the HMD 1.

[0154] In the state before the change in FIG. 32(a), the guide map 504 overlaps in front of the tower 508. The user U1's gaze point 507 overlaps the tower 508 and the guide map 504, which can make it difficult to determine the target object. Even in this case, the target object can be easily determined by using a tag. For example, the tower 508 is selected as the target object. (b) shows the state after the change. The HMD1 adjusts, for example, the transparency of the guide map 504 that is obscuring the selected tower 508. This makes the entire tower 508 visible.

[0155] The HMD 1 may display tags all the time, or may display tags only when it is determined that it is difficult to determine the target object using only the gaze point, or may display tags in response to a tag display instruction input by the user U1.

[0156] [Variations] FIG. 33 shows an example in which a display device or information processing device according to a modification of the fourth embodiment is applied to a smartphone. FIG. 33 shows an example in which tags are attached to each object as a display example on the display surface of the smartphone 700. Even in the case of the smartphone 700, the occlusion / obstruction relationship between objects is considered to be a three-dimensional arrangement that takes into account the position in the depth direction. Therefore, the display mode change and other methods of the above-described embodiments can be similarly applied. The functional block configuration of the smartphone 700 is not shown, but is basically the same as the configuration of FIG. 10. The smartphone 700 does not perform gaze detection or gaze point detection, but uses other operation input means. The smartphone 700 may achieve gaze detection or gaze point detection using other means (e.g., the camera unit 431). The smartphone 700 displays actual objects (corresponding individual actual objects) as images captured by the built-in camera (the camera unit 431). In the case of the smartphone 700, in addition to voice input and other operation inputs, selection input by tapping on the touch panel on the display surface is also possible.

[0157] <Fifth Embodiment> A fifth embodiment will be described with reference to Fig. 34. In the above-described embodiment, a case where an object in front occludes and obstructs an object in the rear has been described as an occlusion obstruction relationship between two objects in the depth direction. In the case of such an occlusion obstruction relationship, an example has been described in which the display mode is changed so that at least the target object is easier to view. Relationships of objects for which the display mode is changed exist other than the occlusion obstruction relationship described above.

[0158] In the fifth embodiment, when a user visually recognizes multiple objects, the relationship between the objects is determined by the difference in luminance. Consider a case where two objects (individual real objects or virtual objects) are placed one behind the other in the field of view, or even if not located one behind the other, they are placed close to each other, such as on the left and right. If there is a large difference in luminance (in other words, brightness) between the objects, one object may interfere with the recognition of the other object. In this case, the HMD of the fifth embodiment changes the display mode.

[0159] FIG. 34 shows an example display. An object "A" 102 in front and an object "B" 103 in the rear are arranged in a field of view 101. The gaze point 106 of user U1 is located on the object "A" 102 in front, and the object "A" is the target object. In this case, user U1 can basically see the entirety of the object "A" 102, and there is no occlusion obstruction relationship as described above. Here, if there is a large difference in luminance between the object "A" and the object "B," for example, if "B" has a higher luminance, the object "B" may obstruct the visibility of the target object "A." This is not limited to objects in front and behind, but similarly, nearby objects may obstruct the visibility of the target object "A" even in cases where there are nearby objects on the left and right, etc., in a non-occlusion relationship.

[0160] The HMD1 determines the difference in brightness between the objects and, based on that difference, determines an obstructing object from the viewpoint of brightness. The HMD1 changes the display mode of the determined obstructing object, for example, object "B." For example, the HMD1 moves the display position of object "B" so as to move it away from object "A," as in the case after the change in (b). Alternatively, if changing the display mode of object "B" is undesirable, the HMD1 may change it by, for example, moving the target object "A."

[0161] In particular, in the fifth embodiment, as another method for changing the display mode, the HMD 1 may use a method for temporarily changing the brightness of the object. For example, the HMD 1 temporarily reduces the brightness of the object "B." This reduces the brightness difference, making it easier for the user U1 to view the target object "A."

[0162] Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-described embodiments and various modifications can be made without departing from the spirit of the present invention. Combinations of the embodiments and modifications in which components are added, deleted, or replaced are also possible. [Explanation of symbols]

[0163] 1...HMD (head-mounted information processing device), 11...display surface, U1...user, 101...field of view, 102, 103...object, 104, 105...line of sight, 106...point of gaze, 107...target visual range, 120...information server.

Claims

1. a display device for displaying an image; a processor for controlling the display of said image; Equipped with displaying at least two or more three-dimensionally arranged virtual objects as objects on the display device; Calculating the user's gaze point and setting an area including the object closest to the gaze point as a target visible range; Detecting the object that blocks at least a portion of the target visual range as an obstructing object; A display device that changes a display mode of at least one of the target visible range and the obstructing object when the obstructing object is present so as to eliminate or reduce obstruction caused by the obstructing object to visibility of the target visible range, In a case where a plurality of users use the display device, with the user being a first user and another user being a second user, and the plurality of users are shared users and use the object as a shared object, Based on communication between the first display device of the first user and the second display device of the second user, the first display device of the first user displays mark information indicating that the second user is gazing at the shared object that the second user is gazing at; Display device.

2. 2. The display device according to claim 1, The object closest to the position of the user's gaze point is set as a target object; setting the object that is highly related to the target object in terms of display as a related object for the target object; an image area obtained by combining the image area of ​​the target object and the image area of ​​the related object is set as the target visible range; Display device.

3. 2. The display device according to claim 1, When at least a part of the shared object gazed at by the second user is hidden by another object, the first display device of the first user changes a display mode of the object so that the entirety of the shared object gazed at by the second user is visible. Display device.

4. 4. The display device according to claim 3, When changing the display mode of the object so that the entirety of the shared object gazed upon by the second user is visible, a state as viewed by the second user is displayed as the entirety of the shared object gazed upon by the second user. Representation device.

Images