Display unit, and method for controlling display unit

What is AI technical title?
AI technical title is built by Patsnap AI team. It summarizes the technical point description of the patent document.
The display device adjusts the display state of other users' avatars based on the user's gaze to prevent obstruction, ensuring objects are visible and maintaining realism in virtual events.

JP2025167161APending Publication Date: 2025-11-07CANON KK

View PDF 2 Cites 0 Cited by

Patent Information

Application Number: JP2024071521
Authority / Receiving Office: JP · JP
Patent Type: Applications
Current Assignee / Owner
Filing Date: 2024-04-25
Publication Date: 2025-11-07

AI Technical Summary

Technical Problem

When participating in a virtual event, avatars of other users may obstruct the objects a user wants to see, leading to a loss of realism and difficulty in ensuring the user can view desired objects.

Method used

A display device with gaze detection capabilities adjusts the display state of other users' avatars, including position, orientation, and mode, based on the user's gaze position to prevent obstruction.

Benefits of technology

Enables users to view desired objects without obstruction by other users' avatars, maintaining a realistic virtual environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 2025167161000001_ABST

Patent Text Reader

Abstract

To provide a display unit that, even when an avatar of another user is displayed on a space that a user sees, allows the user to visually recognize an object that the user wants to see without being blocked by the avatar of the other user.SOLUTION: A display unit has display control means that displays an avatar of another user in a space that is a user's field of vision, and acquisition means that acquires the position of the user's line of sight. When the position of the user's line of sight and the display position of the avatar of the other user satisfy predetermined conditions, the display control means performs control to change the display state of the avatar including at least one of the display position, direction, and display mode of the avatar.SELECTED DRAWING: Figure 7

Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to a display device and a method for controlling a display device. [Background technology]

[0002] In recent years, eyeglass-type head-mounted displays (HMDs) with gaze detection functions and head-up displays (HUDs) for vehicles have become increasingly automated and intelligent. Furthermore, users can participate in virtual events using display devices such as HMDs and HUDs, along with devices such as smartphones and personal computers (PCs). In virtual events where users can participate as avatars, not only the user's avatar but also the avatars of other users are displayed.

[0003] Patent Document 1 discloses a technology that detects a user's eye movements based on electrooculography signals and synchronizes them with the user's blinks and gaze movements to control the blinks and gaze movements of the user's avatar. Patent Document 2 discloses a technology that moves a user's avatar object to an empty seat selected by the user with their gaze in a map object presented in a virtual space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-177277 [Patent Document 2] Japanese Patent Application Publication No. 2018-142319 Summary of the Invention [Problem to be solved by the invention]

[0005] When participating in a virtual event as an avatar and other users' avatars are displayed in the space the user is viewing, the avatars of the other users may overlap the object the user wants to see. If the avatars of the other users are hidden so that the user can see the object they want to see, the sense of realism of the virtual event will be lost. Furthermore, even if the user's gaze information can be used to control or move their own avatar, it is difficult to ensure that the object the user wants to see is not obstructed by the avatars of other users.

[0006] To provide a display device that enables a user to view an object that the user wants to see without being obstructed by the avatar of another user, even when the avatar of another user is displayed in a space the user is viewing. [Means for solving the problem]

[0007] The display device of the present invention has a display control means for displaying an avatar of another user in a space within the user's field of vision, and an acquisition means for acquiring the user's gaze position, and is characterized in that the display control means controls the display state of the avatar, including at least one of the display position, orientation, and display mode of the avatar, to be changed when the user's gaze position and the display position of the avatar of the other user satisfy predetermined conditions. [Effects of the Invention]

[0008] According to the present invention, a display device can be provided that allows a user to view an object that the user wants to see without it being obstructed by the avatar of another user, even if the avatar of another user is displayed in the space the user is viewing. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram of the configuration of an HMD. [Figure 2]FIG. 1 is a schematic diagram of a configuration for detecting the line of sight of an HMD. [Figure 3] FIG. 1 is a diagram for explaining the principle of a gaze detection method. [Figure 4] FIG. 10 shows an eye image. [Figure 5] 10 is a flowchart illustrating an example of a gaze detection process. [Figure 6] 10A to 10C are diagrams illustrating the display control of other users' avatars in the first embodiment. [Figure 7] 4 is a flowchart illustrating a display control process according to the first embodiment. [Figure 8] 10 is a flowchart illustrating a display control process according to the second embodiment. [Figure 9] 10A and 10B are diagrams illustrating the display control of other users' avatars in the second embodiment. [Figure 10] 10 is a flowchart illustrating a display control process according to a third embodiment. [Figure 11] 10 is a flowchart illustrating a process of determining a moving direction of an avatar. [Figure 12] 13A to 13C are diagrams illustrating the display control of other users' avatars in the third embodiment. [Figure 13] 10 is a flowchart illustrating a display control process according to a fourth embodiment. [Figure 14] 10 is a flowchart illustrating a process for determining a moving speed of an avatar. [Figure 15] 10 is a flowchart illustrating a process for determining whether an avatar can move. [Figure 16] 10 is a flowchart illustrating a gaze determination process for a gaze position. [Figure 17] FIG. 10 is a diagram illustrating calculation of a convergence angle. [Figure 18] FIG. 13 is a diagram illustrating the display of information about other users in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment 1> Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] (composition) The configuration of a head-mounted display (HMD), which is an example of a display device according to the present invention, will be described with reference to Fig. 1. The display device according to the present invention includes an HMD capable of implementing technologies such as virtual reality (VR), mixed reality (MR), and augmented reality (AR). The display device according to the present invention also includes a head-up display (HUD) and the like, as long as it can display the avatars of other users in the space within the user's field of vision.

[0012] Fig. 1 is a schematic diagram of the configuration of an HMD 100. The HMD 100 is a display device that is worn on the head of a user. Fig. 1 shows the configuration of the HMD 100 as viewed from the top of the head of a user wearing the HMD 100, and a block diagram showing the functional configuration of the HMD 100.

[0013] A user wears a housing 103 of the HMD 100 on their head and can observe real space with their left eye 101 and right eye 102 through a transmissive left display 104 and right display 105, respectively. In the example of FIG. 1, the HMD 100 is an optical see-through HMD. The optical see-through HMD 100 can superimpose virtual objects, information about virtual objects, information about real objects, and the like onto the real world that the user sees through the left display 104 and right display 105.

[0014] The HMD 100 may be a video see-through HMD having a non-transmissive display. The video see-through HMD 100 has eyepieces placed between each of the left and right eyes and the non-transmissive display. The video see-through HMD 100 has a left camera 106 and a right camera 107.

[0015] In the non-transparent mode, the video see-through type HMD 100 displays the internally stored video (photographed video, game video, etc.) on the non-transparent display. In the transparent mode, the non-transparent display may display real-time images of a virtual event or the like obtained from an external device such as a server connected to the HMD 100 via a network such as the Internet. In the transparent mode, the non-transparent display displays images captured by the left camera 106 and the right camera 107 so that the real space appears transparent. The non-transparent display may display an image that combines internal video with the captured images.

[0016] The HMD 100 has a left eye imaging unit 108 and a right eye imaging unit 109. The left eye imaging unit 108 and the right eye imaging unit 109 are used to detect the position of the user's gaze. The HMD 100 also has an operation member 110. The operation member 110 includes a power button for turning the power of the HMD 100 on and off, buttons for executing various functions of the HMD 100, and the like.

[0017] The image processing unit 111 acquires images captured by the left camera 106 and the right camera 107 and performs various image processing. The image processing unit 111 can detect objects that appear in the captured images. The left camera 106 and the right camera 107 can capture images of the real space in the front direction of the user.

[0018] The display control unit 112 displays images of virtual objects, information about virtual objects, information about real objects, and the like on the left display 104 and the right display 105. In this way, the display control unit 112 can superimpose images of virtual objects and various pieces of information onto the real world seen by the user through the left display 104 and the right display 105. The virtual objects include avatars of other users.

[0019] The gaze detection unit 113 detects the gaze position of the user on the display surfaces of the left display 104 and the right display 105 using the eye images captured by the left eye imaging unit 108 and the right eye imaging unit 109 .

[0020] Fig. 2 is a cross-sectional view of the HMD 100 taken along the YZ plane defined by the Y-axis and Z-axis shown in Fig. 1, showing a schematic diagram of the configuration for gaze detection. Fig. 2 is a cross-sectional view seen from the left eye side of a user wearing the HMD 100, showing a detailed configuration for gaze detection. The following mainly describes the mechanism for detecting the gaze of the left eye 101, but the HMD 100 also has a similar mechanism on the right eye side, and can detect the gaze of the right eye 102.

[0021] 2, the HMD 100 has a CPU 128 and a memory unit 129 inside the housing 103. The CPU 128 controls the entire HMD 100 (each functional unit).

[0022] The memory unit 129 has a function of storing image pickup signals from the image pickup element 125 and the eye image pickup element 123, and records images (image information) to be displayed on the displays (left display 104 and right display 105). The memory unit 129 also has a function of storing gaze correction data that corrects individual differences in gaze and eye characteristic information. The memory unit 129 also stores programs to be executed by the CPU 128. The CPU 128 can implement the functions of the image processing unit 111, display control unit 112, and gaze detection unit 113 shown in FIG. 1 by executing the programs stored in the memory unit 129.

[0023] The HMD 100 has a left display 104 and a drive circuit 124. The left display 104 is a display such as a liquid crystal display that displays captured images. The drive circuit 124 drives the left display 104.

[0024] The HMD 100 includes an image sensor 125, an aperture mechanism 1, and the like as components included in the left camera 106. 26 and a focus mechanism 127. With these configurations, the left camera 106 can capture an image of the object scene through the light splitter 121.

[0025] The HMD 100 has a light source 120, a light receiving lens 122, and an eye image sensor 123 as components included in the left eye imaging unit 108. The light source 120 is a light source that illuminates the left eye 101 for gaze detection. The light source 120 is, for example, an infrared light emitting diode that emits infrared light that is insensitive to the user. The light source 120 may include multiple infrared light emitting diodes.

[0026] An optical image of the illuminated left eye (eye image; an image formed by light emitted from the light source 120 and reflected by the left eye 101) is formed on an eye image sensor 123, which has a two-dimensional array of photoelectric elements such as CMOS, by a light receiving lens 122. The eye image includes a reflection image (corneal reflection image; Purkinje image) formed by corneal reflection of light from the light source 120.

[0027] The light receiving lens 122 positions the pupil of the left eye 101 and the ocular imaging element 123 in a complementary imaging relationship. The line of sight of the left eye 101 is detected from the positional relationship between the pupil (pupil image) and the corneal reflection image in the eyeball image formed on the ocular imaging element 123, using a predetermined algorithm described in Figures 3, 4(A), 4(B), and 5. The memory unit 129 has a function of storing imaging signals from the ocular imaging element 123.

[0028] (Gaze detection method) The gaze detection method (gaze detection algorithm) will be explained using Figures 3, 4(A), 4(B), and 5. Figure 3 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection.

[0029] 3, light sources 120a and 120b are arranged approximately symmetrically with respect to the optical axis of light receiving lens 122, and illuminate user's eyeball 140. A portion of the illumination light emitted from light sources 120a and 120b and reflected by eyeball 140 is collected onto eye imaging element 123 by light receiving lens 122.

[0030] FIG. 4(A) is a schematic diagram of an eye image captured by the eye image sensor 123 (eyeball image projected onto the eye image sensor 123), and FIG. 4(B) is a diagram showing the output intensity of the CCD in the eye image sensor 123.

[0031] 5 is a flowchart illustrating the gaze detection process. When the gaze detection process starts, in step S501, light sources 120a and 120b emit infrared light toward user's eyeball 140. An image of the user's eyeball illuminated by the infrared light is formed on ocular imaging element 123 through light receiving lens 122 and photoelectrically converted by ocular imaging element 123. This results in a processable electrical signal of the eye image. In step S502, CPU 128 acquires the eye image (image data, image signal) obtained from ocular imaging element 123.

[0032] In step S503, the CPU 128 obtains the corneal reflection images Pd and Pe of the light sources 120a and 120b and the coordinates of the point corresponding to the pupil center c from the eye image obtained in step S502. Infrared light emitted from the light sources 120a and 120b illuminates the cornea 142 of the user's eyeball 140. At this time, the corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are condensed by the light receiving lens 122 and formed on the ocular imaging element 123 as corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from the edges a and b of the pupil 141 are also formed on the ocular imaging element 123 as pupil edge images a' and b' in the eye image.

[0033] FIG. 4B shows the luminance information (luminance distribution) of the region α in the eye image of FIG. 4A. In Figure 4(B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction. Figure 4(B) shows the luminance distribution in the X-axis direction. The coordinates of the corneal reflection images Pd' and Pe' in the X-axis direction (horizontal direction) are Xd and Xe, and the coordinates of the pupil edge images a' and b' in the X-axis direction are Xa and Xb.

[0034] As shown in FIG. 4B, an extremely high level of luminance is obtained at the coordinates Xd and Xe of the corneal reflection images Pd' and Pe'. In the region from coordinate Xb to coordinate Xa, which corresponds to the region of the pupil 141 (the region of the pupil image obtained when the light beam from the pupil 141 is focused on the ocular imaging element 123), an extremely low level of luminance is obtained except for coordinates Xd and Xe. In the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained when the light beam from the iris 143 is focused), a luminance intermediate between the above two types of luminance is obtained. Specifically, a luminance intermediate between the above two types of luminance is obtained in the region where the X coordinate (coordinate in the X-axis direction) is greater than coordinate Xa and the region where the X coordinate is less than coordinate Xb.

[0035] 4(B), the CPU 128 can obtain the X-coordinates Xd and Xe of the corneal reflection images Pd' and Pe' and the X-coordinates Xa and Xb of the pupil edge images a' and b'. Specifically, the CPU 128 can obtain the coordinates where the luminance is extremely high as the coordinates of the corneal reflection images Pd' and Pe', and the coordinates where the luminance is extremely low as the coordinates of the pupil edge images a' and b'.

[0036] Furthermore, when the rotation angle θx of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 122 is sufficiently small, the coordinate Xc of pupil-centered image c' (center of the pupil image) obtained when a light beam from pupil center c is focused on ocular imaging element 123 can be expressed as Xc ≒ (Xa + Xb) / 2. That is, CPU 128 can calculate the coordinate Xc of pupil-centered image c' from the X-coordinates Xa and Xb of pupil edge images a' and b'. In this way, CPU 128 can estimate the coordinates of corneal reflection images Pd' and Pe' and the coordinate of pupil-centered image c'.

[0037] In step S504, CPU 128 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of eyeball 140 relative to light receiving lens 122, and can be calculated using a function of the distance (Xd-Xe) between corneal reflection images Pd' and Pe'.

[0038] In step S505, CPU 128 calculates the rotation angle of the optical axis of eyeball 140 relative to the optical axis of light receiving lens 122. The X coordinate of the midpoint between corneal reflection images Pd and Pe approximately coincides with the X coordinate of the center of curvature O of cornea 142. Therefore, if the standard distance from the center of curvature O of cornea 142 to the center c of pupil 141 is Oc, then the rotation angle θx of the optical axis of eyeball 140 in the ZX plane (plane perpendicular to the Y axis) can be calculated using the following equation 1. The rotation angle θy of eyeball 714 in the ZY plane (plane perpendicular to the X axis) can also be calculated using a method similar to that for calculating rotation angle θx. β×Oc×SINθx≒{(Xd+Xe) / 2}-Xc (Formula 1)

[0039] In step S506, CPU 128 uses the rotation angles θx, θy calculated in step S505 to obtain the user's viewpoint (gaze position; the position where the user is looking) on the screen of left display 104. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated using the following equations 2 and 3. Hx=m×(Ax×θx+Bx) (Formula 2) Hy=m×(Ay×θy+By) (Formula 3)

[0040] The parameter m in Equations 2 and 3 is the relationship between the user's eyeball angle and the position of the viewpoint on the left display 104. θx, θy are constants that represent the relationship between the rotation angles θx, θy and the position of the pupil, and are conversion coefficients that convert the rotation angles θx, θy into coordinates corresponding to the pupil center c on the left display 104. The parameter m is determined in advance and stored in the memory unit 129. The parameters Ax, Bx, Ay, and By are gaze correction parameters that correct for individual differences in gaze, and can be obtained by a calibration process. The parameters Ax, Bx, Ay, and By are stored in the memory unit 129 before the gaze detection process starts.

[0041] In step S507, CPU 128 stores the coordinates (Hx, Hy) of the viewpoint (pupil center c) calculated in step S506 in memory unit 129, and ends the gaze detection process. Note that, although Fig. 5 shows an example in which the corneal reflection images of light sources 120a and 120b are used to obtain the rotation angle of the eyeball and to obtain the coordinates of the viewpoint on left display 104, this is not limiting. A method for obtaining the rotation angle of the eyeball from the eyeball image may also be a method of measuring the gaze from the pupil center position, for example.

[0042] (Display control of other users' avatars) Display control of other users' avatars will be described with reference to FIGS. 6(A) to 6(D) and 7. The HMD 100 changes the display state of other users' avatars using gaze information of the user wearing the HMD 100. The display state of the avatar includes at least one of the display position, orientation, and display mode of the avatar. The HMD 100 changes the display state of the other users' avatars when the gaze position of the user and the display position of the other users' avatars satisfy a predetermined condition. In the first embodiment, the predetermined condition is that the avatar of the other users overlaps a predetermined area including the gaze position of the user.

[0043] 6A is a diagram illustrating an example of a video 601 of a virtual event (for example, a live music concert) that the user is viewing through the HMD 100. Two people 602 and 603 present in the video 601 are artists performing on stage.

[0044] 6(B) is a diagram illustrating an example of a state in which avatars of other users are superimposed on video 601. A display such as that shown in FIG. 6(B) allows a user wearing HMD 100 to recognize that multiple other users are participating in the virtual event as avatars. Avatar 604 is displayed superimposed on person 603, with part of person 603 hidden.

[0045] The operation of the HMD 100 that controls the display of other users' avatars will be described with reference to Fig. 7. Fig. 7 is a flowchart illustrating a display control process of the HMD 100 according to the first embodiment.

[0046] In step S701, the CPU 128 starts up the HMD 100. In step S702, the CPU 128 selects content such as a virtual event in which the user will participate, based on the user's operation.

[0047] In step S703, CPU 128 connects to a network such as the Internet, communicates with an external device such as a server, and acquires video of the content of the virtual event selected in step S702. The video of the content is a video of the space within the user's field of vision, and CPU 128 displays the acquired video on a display. The displays are left display 104 and right display 105. The content acquired from the external device includes an image for the right eye and an image for the left eye, and left display 104 and right display 105 display the image for the right eye and the image for the left eye, respectively.

[0048] In step S704, the CPU 128 detects other users participating in the virtual event. The CPU 128 can acquire information about other users from a server that provides the content of the virtual event.

[0049] In step S705, the CPU 128 acquires the gaze position of the user wearing the HMD 100 by executing the gaze detection process described in FIG. 5. In step S706, the CPU 128 determines whether or not another user's avatar is present in a predetermined area including the user's gaze position. The predetermined area may be an area within a predetermined distance from the user's gaze position. The predetermined distance can be set based on, for example, the size of an object (e.g., person 603) where the user's gaze position is located.

[0050] When the avatar of another user overlaps with the predetermined area, CPU 128 can determine that the avatar of another user exists in the predetermined area. If the avatar of another user exists in the predetermined area, the process proceeds to step S707. If the avatar of another user does not exist in the predetermined area, the process proceeds to step S709.

[0051] In step S707, CPU 128 determines whether the user's gaze position is located at (the user's gaze is directed at) the avatar of another user determined to be present in the predetermined area in step S706. CPU 128 can determine that the user's gaze position is located at the avatar of another user when the user's gaze position is included in the display area of the avatar of the other user. If the user's gaze position is not located at the avatar of the other user, the process proceeds to step S708. If the user's gaze position is located at the avatar of the other user, the process proceeds to step S709.

[0052] In step S708, CPU 128 moves avatars of other users that are present in the predetermined area but that are not in the user's line of sight out of the predetermined area.

[0053] FIG. 6(C) shows a state in which the user is looking at person 603, and the user's gaze position 605 is within the area of person 603. Another user's avatar 604 is present in a predetermined area 606 that includes gaze position 605. Furthermore, gaze position 605 does not overlap with another user's avatar 604. In this case, as shown in FIG. 6(D), another user's avatar 604 is moved outside the predetermined area 606 in step S708. Note that avatar 604 may also be moved to a position outside the predetermined area 606 that does not overlap with person 603.

[0054] In step S709, the CPU 128 determines whether or not the user has performed an operation to stop the HMD 100. If an operation to stop the HMD 100 has been performed, the process proceeds to step S710. If an operation to stop the HMD 100 has not been performed, the process returns to step S703. In step S710, the CPU 128 stops the HMD 100 and ends the display control process shown in FIG.

[0055] In step S708, the CPU 128 moves the avatar of the other user so that the user can see the person 603, but the CPU 128 may increase the transparency of the avatar of the other user without moving it. Increasing the transparency of the avatar of the other user also allows the user to see the person 603. In this way, the HMD 100 changes the display state of the avatar by moving the avatar of the other user out of the predetermined area or by increasing the transparency of the avatar, allowing the user to see the object obstructed by the avatar.

[0056] In the first embodiment, when the avatar of another user overlaps with a predetermined area including the gaze position of the user in a virtual event, the HMD 100 Users can change the display state by moving the avatar or making it transparent. By changing the display state of other users' avatars, users can view the objects they want to see.

[0057] <Embodiment 2> In the first embodiment, the HMD 100 controls the display of another user's avatar when the condition (predetermined condition) that the other user's avatar overlaps a predetermined area including the user's gaze position is satisfied. In contrast, in the second embodiment, the HMD 100 controls the display of another user's avatar when the condition (predetermined condition) that the other user's avatar overlaps an area of an object present at the user's gaze position is satisfied. The configuration of the HMD 100 according to the second embodiment is the same as the configuration of the HMD 100 according to the first embodiment described with reference to FIGS. 1 and 2, and therefore a description thereof will be omitted.

[0058] Display control of other users' avatars in embodiment 2 will be described with reference to Fig. 8 and Figs. 9(A) and 9(B). Fig. 8 is a flowchart illustrating display control processing of the HMD 100 according to embodiment 2. Fig. 9 is a diagram illustrating display control of other users' avatars in embodiment 2.

[0059] The flowchart in Fig. 8 explains the differences from the display control process of the HMD 100 according to the first embodiment explained in Fig. 7. In the second embodiment, step S806 is added, and the processes of steps S807, S808, and S810 differ from the processes in the first embodiment. The processes of steps S801 to S805 are the same as the processes of steps S701 to S705 in Fig. 7, respectively.

[0060] In step S806, the CPU 128 detects a main object in the image of the content displayed in step S803 (image 901 in FIG. 9A). A main object is an object other than another user's avatar in the space within the user's field of view. For example, the image processing unit 111 has a function such as face detection or object detection, and can detect the main object based on information about the main object stored in advance in the memory unit 129 or the like. In the example of FIG. 9A, the CPU 128 detects main objects 902 and 903.

[0061] The CPU 128 may detect the main object based on content information acquired from an external device such as a server connected to the HMD 100 via a network such as the Internet. The CPU 128 may detect an object included in the content (an object other than another avatar) as the main object, or may detect an object set as the main object in the content as the main object.

[0062] In step S807, CPU 128 determines whether or not a main object (object) is present at the user's line of sight. If a main object is present at the user's line of sight, the process proceeds to step S808. If a main object is not present at the user's line of sight, the process proceeds to step S811.

[0063] In step S808, CPU 128 determines whether or not another user's avatar is present in the area of the main object (object) that is present at the user's line of sight. When the other user's avatar overlaps the area of the main object, CPU 128 can determine that the other user's avatar is present in the area of the main object. If the other user's avatar is present in the area of the main object, the process proceeds to step S809. If the other user's avatar is not present in the area of the main object, the process Then, the process proceeds to step S811.

[0064] In step S809, CPU 128 determines whether the user's gaze position is located (the user's gaze is directed) on the avatar of another user determined to be present in the area of the main object in step S808. If the user's gaze position is not located on the avatar of another user, the process proceeds to step S810. If the user's gaze position is located on the avatar of another user, the process proceeds to step S811.

[0065] In step S810, CPU 128 moves avatars of other users that are present in the area of the main object and that are not in the user's line of sight (are not directed at by the user) out of the area of the main object.

[0066] 9(A), a main object 903 is present at a gaze position 905, and another user's avatar 904 is present in the area of the main object 903. Furthermore, the gaze position 905 does not overlap with another user's avatar 904. Therefore, as shown in FIG. 9(B), the other user's avatar 904 is moved outside the area of the main object 903.

[0067] In step S810, CPU 128 moves the avatar of the other user to make the main object 903 visible, but it is also possible to increase the transparency of the avatar of the other user without moving it. Increasing the transparency of the avatar of the other user also allows the user to make the main object 903 visible. In this way, HMD 100 can change the display state of the avatar by moving the avatar of the other user out of a predetermined area or by increasing the transparency of the avatar, thereby making it possible to make visible an object obstructed by the avatar.

[0068] The processing in steps S811 and S812 is the same as the processing in steps S709 and S710 in FIG. 7, and the CPU 128 stops the HMD 100 in response to a user operation and ends the display control processing shown in FIG.

[0069] In the second embodiment, when another user's avatar overlaps the area of a main object including the user's line of sight in a virtual event, the HMD 100 changes the display state of the other user's avatar by moving or making the other user's avatar transparent. Changing the display state of the other user's avatar allows the user to view the object they want to see.

[0070] <Embodiment 3> The third embodiment is an embodiment in which the movement direction of another user's avatar is determined based on the user's gaze position. When the third embodiment is applied to the second embodiment, the HMD 100 determines the movement direction of the avatar so that the user's gaze position is not blocked by the movement of the avatar when the avatar is moved outside the area of the main object. Note that the third embodiment is also applicable when the avatar is moved outside the predetermined area described in the first embodiment. The configuration of the HMD 100 according to the third embodiment is the same as the configuration of the HMD 100 according to the first embodiment described in FIGS. 1 and 2, and therefore a description thereof will be omitted.

[0071] Display control of other users' avatars in embodiment 3 will be described with reference to Fig. 10, Fig. 11, and Fig. 12(A) and (B). Fig. 10 is a flowchart illustrating a display control process of the HMD 100 according to embodiment 3. Fig. 10 shows an example in which embodiment 3 is applied to embodiment 2.

[0072] The flowchart in Fig. 10 explains the differences from the display control process of the HMD 100 according to the second embodiment explained in Fig. 8. In the third embodiment, step S1010 is added. The processes of steps S1001 to S1009 and S1011 to S1013 are the same as the processes of steps S801 to S812 in Fig. 8, respectively. In step S1010, the HMD 100 determines the movement direction of the avatar of another user to be moved.

[0073] The process of determining the movement direction of another user's avatar in step S1010 of FIG. 10 will be described with reference to FIG. 11 and FIGS. 12(A) and 12(B). FIG. 11 is a flowchart illustrating the process of determining the movement direction of an avatar. FIGS. 12(A) and 12(B) are diagrams illustrating the display control of another user's avatar in the third embodiment. FIG. 12(A) shows an image 1201 before another user's avatar 1204 is moved. FIG. 12(B) shows an image 1201 after another user's avatar 1204 is moved.

[0074] In step S1101, the CPU 128 acquires the position of the avatar 1204 of another user determined to be present in the area of the main object in step S1008 of FIG.

[0075] In step S1102, the CPU 128 acquires the gaze directions of other users participating as avatars 1204. The CPU 128 acquires the gaze directions of other users from an external device, such as a server connected to the HMD 100 via a network such as the Internet. The server can acquire information including the gaze directions of other users from the HMDs 100 worn by the other users.

[0076] In step S1103, the CPU 128 estimates the gaze position of the other user based on the gaze direction of the other user and the area of the main object 1203 detected in step S1006.

[0077] In step S1104, CPU 128 sets the direction from user's gaze position 1205 on the plane of video image 1201 to another user's avatar 1204 as movement direction 1206. Note that when user's gaze position 1205 and another user's avatar 1204 are close to each other, CPU 128 may set a predetermined direction as the movement direction.

[0078] In step S1105, the CPU 128 moves the avatar 1204 of the other user in the movement direction 1206 to the outside of the area of the main object 1203 that the user is viewing.

[0079] In step S1106, CPU 128 changes the face direction of avatar 1204 of the other user so that it faces in the direction of the gaze position of the other user estimated in step S1103, and displays it. As shown in Fig. 12(B) , after being moved outside the area of main object 1203, avatar 1204 of the other user is displayed with its face direction changed so that it faces in the direction of main object 1203 that the other user was viewing.

[0080] In the above-described third embodiment, the HMD 100 determines the movement direction of the avatar based on the user's gaze position and the display position of the avatar of another user. As a result, when moving the avatar of another user, the HMD 100 can prevent the position where the user is viewing the main object from being obstructed by the avatar of another user.

[0081] Furthermore, the HMD 100 controls the avatar 1204 to change the direction of its face so that the face of the avatar 1204 after moving in the movement direction 1206 faces the object (main object 1203) that the other user was looking at before the avatar 1204 moved. By changing the direction of the face of the avatar 1204 to match the gaze direction of other users, the system can control the direction of the face of the avatar 1204 so that it does not look unnatural, thereby preventing the sense of realism of the virtual event from being lost.

[0082] <Embodiment 4> In the fourth embodiment, the HMD 100 determines whether or not the user is gazing at the gaze position, and if the user is gazing at the avatar of another user, displays information about the other user on that avatar.

[0083] When a user is gazing at another user's avatar, the HMD 100 does not move the other user's avatar even if it overlaps with a predetermined area or a main object.

[0084] When the user gazes at an area other than the avatar of another user within the predetermined area or the area of the main object, the HMD 100 moves the avatar of the other user.

[0085] Furthermore, if the original position of the moved avatar of another user is no longer included in the predetermined area or the area of the main object, the HMD 100 returns the display position of the moved avatar of another user to the original position. The configuration of the HMD 100 according to the fourth embodiment is the same as the configuration of the HMD 100 according to the first embodiment described with reference to Figures 1 and 2, and therefore a description thereof will be omitted.

[0086] 13 to 18, the display control of other users' avatars in the fourth embodiment will be described. Fig. 13 is a flowchart illustrating the display control process of the HMD 100 according to the fourth embodiment. The flowchart in Fig. 13 explains the differences from the display control process of the HMD 100 according to the first embodiment described in Fig. 7.

[0087] In the fourth embodiment, steps S1303, S1307, and S1309 to S1315 are added. The processes of steps S1301, S1302, and S1304 to S1306 are the same as the processes of steps S701 to S705, respectively, in Fig. 7. In step S1303, the CPU 128 initializes the movement flag to FALSE.

[0088] In step S1307, CPU 128 determines the movement speed when moving the avatar of another user. The determination of the movement speed of the avatar of another user in step 1307 will be described with reference to Fig. 14. Fig. 14 is a flowchart illustrating an example of the process of determining the movement speed of the avatar of another user.

[0089] In step S1401, CPU 128 obtains the difference between the gaze position obtained in step S1306 and the gaze position obtained previously, and stores the difference in the gaze position in memory unit 129. In step S1402, CPU 128 calculates the movement speed of the user's gaze position using the gaze position difference information stored up to now in memory unit 129. The movement speed of the user's gaze position may be the average value (average movement speed) of the movement speeds calculated for each gaze position difference stored up to now in memory unit 129.

[0090] In step S1403, CPU 128 determines the movement speed of the avatar of the other user based on the movement speed of the gaze position. CPU 128 determines the movement speed of the avatar of the other user so that it is slower than the movement speed of the gaze position of the user.

[0091] The process of step S1308 in Fig. 13 is the same as the process of step S706. In step S1309, the CPU 128 determines that the object exists in the predetermined area in step S1308. The process then determines whether the avatar of the other user can be moved. The determination of whether the avatar of the other user can be moved in step 1309 will be described with reference to Fig. 15. Fig. 15 is a flowchart illustrating the process of determining whether the avatar of the other user can be moved.

[0092] In step S1501, CPU 128 determines whether or not the user is gazing at the gaze position based on the convergence angle formed by the user's left and right gazes. Determining whether or not the user is gazing at the gaze position will be described with reference to Fig. 16 and Fig. 17. Fig. 16 is a flowchart illustrating an example of gaze determination processing for a gaze position. Fig. 17 is a diagram illustrating calculation of the convergence angle.

[0093] In step S1601, CPU 128 acquires the gaze positions of both eyes detected in step S1306 of Fig. 13. In step S1602, CPU 128 calculates the angle of convergence from the gaze positions of both eyes acquired in step S1601, and calculates distance D1 to the gaze positions based on the angle of convergence.

[0094] The calculation of the convergence angle will be described with reference to Fig. 17. As shown in Fig. 17, the convergence angle θ1 is the angle formed by the lines of sight of both eyes looking at point P0 on the subject. The longer the distance from both eyes to point P0, the smaller the convergence angle θ1 becomes, and the shorter the distance from both eyes to point P0, the larger the convergence angle θ1 becomes.

[0095] The convergence angle can be calculated by determining the intersection of the lines of the gaze directions of both eyes. The distance from both eyes to the gaze position can be calculated using trigonometric functions from the calculated convergence angle and the distance between the eyes (the distance between point OL and point OR). In addition, by measuring and pre-storing the correlation between a plurality of objects at different distances and the convergence angle when viewing each object, the distance from both eyes to the gaze position can be estimated from the calculated convergence angle.

[0096] In step S1603, CPU 128 acquires subject distance D2 to the subject at the gaze position. Subject distance D2 may be acquired using depth information obtained from left camera 106 and right camera 107, for example. Furthermore, HMD 100 only needs to determine the distance from HMD 100 to the subject, and may determine the distance to the subject using a distance calculation method using LiDAR (Light Detection and Ranging).

[0097] In step S1604, CPU 128 determines whether the difference between distance D1 calculated in step S1602 and subject distance D2 acquired in step S1603 is smaller than a predetermined value. If the difference between distance D1 and subject distance D2 is smaller than the predetermined value, the process proceeds to step S1605. If the difference between distance D1 and subject distance D2 is equal to or greater than the predetermined value, the process proceeds to step S1606.

[0098] In step S1605, the CPU 128 determines that the user is gazing at the gaze position. In step S1606, the CPU 128 determines that the user is not gazing at the gaze position (gazing absentmindedly).

[0099] 15, CPU 128 determines whether the user is gazing at the gaze position based on the determination result of step S1501. If the user is gazing at the gaze position, the process proceeds to step S1503. If the user is not gazing at the gaze position, the process proceeds to step S1506.

[0100] In step S1503, CPU 128 determines whether the user's gaze position is at (the user's gaze is directed at) the avatar of another user determined to be present in the predetermined area in step S1308 of Fig. 13. is included in the display area of the other user's avatar, it can be determined that the user's gaze position is in the other user's avatar. If the user's gaze position is in the other user's avatar, the process proceeds to step S1504. If the user's gaze position is not in the other user's avatar, the process proceeds to step S1505.

[0101] In step S1505, CPU 128 superimposes information about another user on the avatar of that other user whose gaze position is determined to be at the user's in step S1503. That is, when CPU 128 determines through the processing of steps S1501 to S1503 that the user is gazing at the avatar of that other user, it performs control so that information about that other user is displayed.

[0102] Fig. 18 is a diagram illustrating the display of information about other users in the fourth embodiment. Image 1801 shown in Fig. 18 shows a state in which a user is looking at person 1803, and the user's gaze position 1805 is within the area of person 1803. An avatar 1804 of the other user is present in a predetermined area 1806 that includes gaze position 1805. Furthermore, gaze position 1805 overlaps with avatar 1804 of the other user. In this case, CPU 128 displays information 1807 about the other user participating in the virtual event as avatar 1804, superimposed on image 1801.

[0103] The information about the other user displayed in step S1504 may be hidden after a predetermined time has elapsed, or may be hidden when the user's gaze position moves outside the area of the other user's avatar.

[0104] In step S1505, CPU 128 determines that the avatar of the other user is to be "moved" and ends the processing shown in Fig. 15. In step S1506, CPU 128 determines that the avatar of the other user is to be "not moved" and ends the processing shown in Fig. 15.

[0105] 13, CPU 128 determines, based on the result of the determination made in step S1309, whether or not to move the avatar of another user determined to be present in the predetermined area in step S1308. If it is determined that the avatar of another user should be moved, the process proceeds to step S1311. If it is determined that the avatar of another user should not be moved, the process proceeds to step S1313.

[0106] In step S1311, CPU 128 moves the avatar of another user determined to be moved in step S1310 out of the predetermined area at the movement speed determined in step S1307. In step S1312, CPU 128 sets the movement flag to TRUE.

[0107] In step S1313, CPU 128 determines whether the movement flag is TRUE and the original position of the other user's avatar is outside the predetermined area. If the movement flag is TRUE and the original position is outside the predetermined area, processing proceeds to step S1314. If the movement flag is FALSE or the original position is within the predetermined area, processing proceeds to step S1316.

[0108] In step S1314, CPU 128 returns the avatars of the other users to their original positions. In step S1315, CPU 128 sets the movement flag to FALSE. The processes in steps S1316 and S1317 are the same as the processes in steps S709 and S710 in Fig. 7, and CPU 128 stops HMD 100 in response to a user operation and ends the display control process shown in Fig. 13.

[0109] In steps S1311 and S1314, the avatars of other users are moved. In this case, CPU 128 moves the avatar of the other user at the movement speed determined in step S1307. The avatar movement speed determined in step S1307 is slower than the movement speed of the user's gaze position. By moving the avatar of the other user at a speed slower than the movement speed of the user's gaze position, the user can easily recognize the movement of the avatar. Furthermore, when information about the other user is displayed in step S1504, the user can easily recognize the displayed information about the other user even while the avatar is moving.

[0110] 13 shows an example in which the fourth embodiment is applied to the first embodiment, but the fourth embodiment can also be applied to the second embodiment. In step S1308, the HMD 100 determines whether or not another user's avatar is present in the area of the main object, and may display information about the other user if it is determined in step S1309 that the user is gazing at the other user's avatar. In step S1313, the HMD 100 may determine whether or not the original position of the other user's avatar is outside the area of the main object.

[0111] 13 shows an example in which the display state is changed by moving the avatar of another user in steps S1311 and S1314, but the HMD 100 may change the display state by changing the transparency of the avatar of another user. In this case, the process of moving the avatar of another user in FIG. 13 is replaced with a process of increasing the transparency of the avatar. For example, in step S1311, the HMD 100 changes the display state by increasing the transparency of the avatar of another user. In step S1314, the HMD 100 returns the display state to the state before the change by decreasing the transparency of the avatar.

[0112] In the above-described fourth embodiment, when the user gazes at another user's avatar instead of the main object, the HMD 100 displays information about the other user's avatar. Furthermore, when the user is not gazing at the line of sight, the HMD 100 controls the other user's avatar so that it does not move. By controlling the display of the other user's avatar, the HMD 100 can display video of a virtual event or the like without impairing the sense of realism.

[0113] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0114] The above processor is a processor in a broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0115] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is possible to combine the embodiments appropriately. It is also possible to combine them.

[0116] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more of the functions.

[0117] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a display control means for displaying avatars of other users in a space visible to the user; acquisition means for acquiring the gaze position of the user; and The display control means controls to change a display state of the avatar, including at least one of a display position, a direction, and a display mode of the avatar, when the gaze position of the user and the display position of the avatar of the other user satisfy a predetermined condition. A display device characterized by: (Configuration 2) the predetermined condition is that the avatar of the other user overlaps with a predetermined area including the gaze position of the user; The display device described in configuration 1, characterized in that the display control means controls the avatar to change its display state by moving it outside the specified area or by increasing the transparency of the avatar. (Configuration 3) the predetermined condition is that the avatar of the other user overlaps with an area of an object present at the gaze position of the user; The display control means controls the avatar to change the display state of the avatar by moving the avatar out of the area of the object present at the line of sight position or by increasing the transparency of the avatar. 2. The display device according to configuration 1, (Configuration 4) When changing the display state of the avatar of the other user by moving the avatar, the display control means determines the movement direction of the avatar based on the gaze position of the user and the display position of the avatar. 4. The display device according to any one of configurations 1 to 3. (Configuration 5) The display control means controls the avatar to change its face direction so that the face of the avatar after the avatar has moved in the movement direction is directed toward an object that the other user was looking at before the avatar moved. 5. The display device according to configuration 4. (Configuration 6) The display control means does not change the display state of the avatar when the gaze position of the user is included in a display area of the avatar of the other user. 6. The display device according to any one of configurations 1 to 5, wherein: (Configuration 7) The display control means displays information about the other user when the user is gazing at the avatar of the other user. 7. The display device according to any one of configurations 1 to 6. (Configuration 8) When the avatar is moved, the display control means Move the avatar at a speed slower than the speed of 8. The display device according to any one of configurations 1 to 7, wherein: (Configuration 9) The display control means controls the display state of the avatar to return to the state before the change when the gaze position of the user and the original display position of the avatar of the other user no longer satisfy the predetermined condition. 9. The display device according to any one of configurations 1 to 8. (Configuration 10) When the display control means determines that the user is not gazing at the gaze position based on a convergence angle between the left and right gazes of the user, the display control means does not change the display state of the avatar. 10. The display device according to any one of configurations 1 to 9, wherein: (method) a display control step of displaying avatars of other users in a space visible to the user; an acquisition step of acquiring a gaze position of the user; and In the display control step, when the gaze position of the user and the display position of the avatar of the other user satisfy a predetermined condition, the display state of the avatar including at least one of the display position, orientation, and display mode of the avatar is controlled to be changed. A method for controlling a display device. (program) A program for causing a computer to function as each means of the display device according to any one of configurations 1 to 10. [Explanation of symbols]

[0118] 100: HMD, 112: display control unit, 113: gaze detection unit, 128: CPU

Claims

1. a display control means for displaying avatars of other users in a space visible to the user; acquisition means for acquiring the user's gaze position; and The display control means controls to change a display state of the avatar, including at least one of a display position, a direction, and a display mode of the avatar, when the gaze position of the user and the display position of the avatar of the other user satisfy a predetermined condition. A display device characterized by:

2. the predetermined condition is that the avatar of the other user overlaps with a predetermined area including the gaze position of the user; 2. The display device according to claim 1, wherein the display control means controls the avatar to change its display state by moving the avatar outside the predetermined area or by increasing the transparency of the avatar.

3. the predetermined condition is that the avatar of the other user overlaps with an area of an object present at the gaze position of the user; The display control means controls the avatar to change the display state of the avatar by moving the avatar out of the area of the object present at the line of sight position or by increasing the transparency of the avatar.