Information processing device, information processing system, information processing method, and program

What is AI technical title?
AI technical title is built by Patsnap AI team. It summarizes the technical point description of the patent document.
The information processing device adjusts virtual object transparency based on user gaze to address unintended transparency issues in VR and MR systems, improving user interaction and experience.

JP2025116378APending Publication Date: 2025-08-08CANON KK

View PDF 1 Cites 0 Cited by

Patent Information

Application Number: JP2024010768
Authority / Receiving Office: JP · JP
Patent Type: Applications
Current Assignee / Owner
Filing Date: 2024-01-29
Publication Date: 2025-08-08

AI Technical Summary

Technical Problem

Existing VR and MR systems often display virtual objects in a transparent state contrary to the user's intention, especially when interacting with nearby individuals, leading to user discomfort.

Method used

An information processing device that includes an image acquisition means, gaze acquisition means, and display control means to dynamically adjust the transparency of virtual objects based on user gaze direction, ensuring that virtual objects become transparent when overlapping with people or objects in the real environment.

Benefits of technology

Enhances user experience by reflecting the user's intentions, allowing seamless interaction with both virtual and real-world elements without visual obstruction.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure 2025116378000001_ABST

Patent Text Reader

Abstract

To make it possible to provide an information processing device being more convenient while reflecting intention of a user.SOLUTION: An information processing device according to the present invention includes: image acquisition means for acquiring a photographed image; line of sight acquisition means for acquiring a line of sight of a user; and display control means for displaying a composite image obtained by combining the photographed image and a virtual object on display means. The display control means performs control to determine a transparency of the virtual object overlapping with a person included in the photographed image according to the line of sight of the user.SELECTED DRAWING: Figure 1

Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] Users can wear a head-mounted device (HMD: Head Mounted Display) and experience virtual reality (VR) and mixed reality (MR) spaces.

[0003] In VR and MR, the real space is often hidden by virtual objects, and there is known a technique for making the virtual objects transparent when another person approaches. Patent Document 1 is an example of prior art relating to such a technique for making virtual objects transparent. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-301924 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even when another person is nearby, there are cases where the virtual object is in a transparent state contrary to the user's intention, such as when the user wants to view the virtual object.

[0006] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an information processing device that reflects the user's intentions and is more user-friendly. [Means for solving the problem]

[0007] One aspect of the present invention is an information processing device comprising an image acquisition means for acquiring a captured image, a gaze acquisition means for acquiring a user's gaze, and a display control means for displaying a composite image on a display means that combines the captured image with a virtual object, wherein the display control means controls to determine the transparency of a virtual object included in the captured image that overlaps with a person other than the user, based on the user's gaze. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an information processing device that reflects the user's intentions and is more user-friendly. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating the internal configuration of an information processing device according to the first embodiment. [Figure 2] 4 is a flowchart illustrating processing by a control unit 101 in the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of an image visually recognized by a user in the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of an image visually recognized by a user in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of an image visually recognized by a user in the first embodiment. [Figure 6] FIG. 10 is a diagram illustrating an example of an image visually recognized by a user when the line of sight moves from a location other than the virtual object or the person onto the person in a situation where the virtual object is in a transparent state in the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of an image visually recognized by a user in the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of an image visually recognized by a user in the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of an image visually recognized by a user in the second embodiment. [Figure 10]4 is a flowchart illustrating processing by a control unit 101 in the first embodiment. [Figure 11] 10 is a flowchart illustrating processing by a control unit 101 in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (First embodiment) The internal configuration of an information processing device according to a first embodiment of the present invention will be described below with reference to Fig. 1. The HMD 100 is a head-mounted information processing device that can be worn on the head of a user. The HMD 100 includes a control unit 101, an imaging unit 102, an orientation sensor unit 103, a position and orientation estimation unit 104, a gaze detection unit 105, a subject detection unit 106, an image generation unit 107, an image synthesis unit 108, a display unit 109, a working memory 110, and a non-volatile memory 111.

[0012] The control unit 101 is a CPU that controls each component of the HMD 100. Instead of the control unit 101 controlling the entire device, the entire device may be controlled by having a plurality of hardware devices share the processing.

[0013] The imaging unit 102 is an image acquisition unit including two cameras (imaging devices). The two cameras are used to capture images used for compositing with virtual space images and generating position and orientation information, and include an imaging unit for the left eye and an imaging unit for the right eye. The two cameras are arranged symmetrically with respect to the center of the HMD 100 so as to capture images in front of the user when the HMD 100 is worn. The imaging unit for the left eye captures moving images of the real space corresponding to the left eye of the person wearing the HMD 101, and the left eye imaging unit outputs images of each frame of the moving images (captured images). The imaging unit for the right eye captures moving images of the real space corresponding to the right eye of the person wearing the HMD 101, and the right eye imaging unit outputs images of each frame of the moving images (captured images). In other words, the imaging unit 102 acquires captured images as stereo images with parallax that approximately matches the positions of the left and right eyes of the person wearing the HMD 101. Furthermore, distance information can be acquired by measuring distances using the stereo cameras, indicating the distance from the two cameras to the subject. In an HMD for an MR system, it is preferable that the central optical axis of the imaging range of the imaging unit be arranged so as to substantially coincide with the line of sight of the person wearing the HMD.

[0014] Each of the left-eye and right-eye imaging units has an optical system and an imaging device. Light entering from the outside enters the imaging device via the optical system, and the imaging device outputs an image corresponding to the incoming light as a captured image. The images captured by the two cameras are stored in the working memory 110 after undergoing necessary correction (not shown). Note that the imaging unit 102 may capture and output video instead of captured images.

[0015] In the first embodiment, a two-camera configuration is described, but only one camera or more cameras may be installed. The placement of the cameras is not limited to the above and may be placed in any location.

[0016] The attitude sensor unit 103 is an inertial measurement unit (IMU) and is equipped with a gyro sensor and an acceleration sensor. These equipped sensors can acquire the angular velocity and acceleration of the HMD 100 as sensor information. The attitude sensor unit 103 acquires attitude (and position) information of the HMD 100. Note that the attitude sensor unit 204 may acquire attitude information of the user (the user wearing the HMD 100) that corresponds to the attitude (and position) of the HMD 100.

[0017] The attitude sensor unit 103 is not limited to the above, and the attitude information may be acquired from one or more of a magnetic sensor (including a geomagnetic sensor), an ultrasonic sensor, an acceleration sensor, and an angular velocity sensor. Note that the attitude sensor unit 103 may include sensors other than those described above.

[0018] The position and orientation estimation unit 104 estimates the position and orientation of the HMD 100 in space (hereinafter referred to as the position and orientation) from the image captured by the imaging unit 102 and the sensor information acquired by the orientation sensor unit 103. The position and orientation estimation unit 104 receives the image captured by the imaging unit 102 and estimates the position and orientation using a Visual SLAM (Simultaneous Localization and Mapping) technique. Visual SLAM is a technique for estimating the HMD's own position and orientation by extracting feature points from the image, calculating the distance to the feature points by triangulation, and mapping the feature points in space. Note that the position and orientation estimation unit 104 may estimate its own position using LiDAR SLAM (Light Detection And Ranging) using a laser, in addition to Visual SLAM, which estimates its own position from a camera image. Note that it may also estimate its own position using Depth SLAM, which uses a ToF sensor (ToF; Time of Flight).

[0019] Furthermore, the estimation of position and orientation is not limited to using Visual SLAM, and may be performed using other methods, such as Visual Inertial SLAM, which uses both an IMU and images.

[0020] The gaze detection unit 105 is a gaze acquisition unit that detects the gaze direction of a user wearing the HMD 100 and identifies the point of gaze in the image the user is looking at. One method for detecting the gaze is to emit infrared light from an infrared light emitting unit mounted inside the HMD toward the user's eyeball, and acquire the direction of the user's gaze from the positional relationship between the light reflected by the cornea and the pupil from an image acquired by an infrared camera mounted inside the HMD.

[0021] It may also be a camera that captures an image for detecting the user's line of sight, and is attached inside the HMD to capture an image of the user's eyes when the user wears the HMD 100. The line of sight detection unit 105 detects the line of sight of the user wearing the HMD 100 and identifies the part of the display unit 109 that the user is gazing at.

[0022] The subject detection unit 106 acquires the image captured by the imaging unit 102 and detects a subject from within the image. If it is determined that a specific subject is present in the image, it identifies the position and area of the subject and stores this as subject information in a working memory, which will be described later. The position of the subject may also be identified in space using a method such as triangulation.

[0023] The subject detection unit 106 detects a person or a moving object as a target subject. The subject detection can be performed using various methods, such as object detection using deep learning.

[0024] The image processing unit 107 generates CG objects and performs rendering processing. CG objects include 3D data prepared in advance, and windows for viewing images and videos or editing documents. In order to display these objects superimposed on the image captured by the imaging unit 102, rendering is performed using the position and orientation information estimated by the position and orientation estimation unit 104 to generate a CG image. The image processing unit 107 also outputs CG information including position and area information of the CG object, and stores it in the working memory 110 together with the CG image.

[0025] The image synthesis unit 108 acquires the captured image acquired by the imaging unit 102 and the CG and CG information generated by the image processing unit 107 from the working memory 110, and synthesizes them to generate an output image for display. In this way, it is possible to express a mixed reality space in which CG is superimposed on a real space.

[0026] When combining images, a combining method is determined using gaze information detected by gaze detection unit 105, subject information detected by subject detection unit 106, and CG objects generated by image processing unit 107. The operation of image combining unit 108 will be described in detail later.

[0027] The display unit 109 displays the image generated by the image synthesis unit 108. The display unit 109 has a display panel, such as a liquid crystal panel or an organic EL panel, and an eyepiece optical system for displaying the image displayed on the panel at a size optimal for the user's eyes. When the user wears the HMD 100, an image display unit is placed in front of each of the user's eyes.

[0028] The working memory 110 stores various data required for processing by each component of the HMD 100. For example, the working memory 110 stores sensor information acquired by the orientation sensor unit 103, position and orientation information and tracking status estimated by the position and orientation estimation unit 104, and the like.

[0029] The nonvolatile memory 111 is an electrically erasable and recordable nonvolatile memory, and stores applications executed by the control unit 101, various setting values, and the like.

[0030] The communication unit 112 is configured with, for example, an antenna for wireless communication, a modulation / demodulation circuit for processing wireless signals, and a communication controller in order to transmit and receive data to and from other devices. The communication unit 112 outputs a weighted wireless signal from the antenna and demodulates the wireless signal received by the antenna to achieve short-range wireless communication in accordance with the IEEE802.15 standard (so-called Bluetooth (registered trademark)). Note that the communication unit 112 may be wired, such as a USB cable (registered trademark), or wireless, such as Wi-Fi (Wireless Fidelity) (registered trademark). Note that data may be transmitted and received to and from multiple devices.

[0031] The operation unit 113 includes buttons. The operation unit 113 detects whether or not a button has been operated, and transmits detection information to the control unit 101. Note that the operation unit 113 may have a plurality of types of input formats.

[0032] The output unit 114 is configured from an LED light source, a speaker, a vibration element, and the like.

[0033] <Description of the processing of the control unit 101> The processing of the control unit 101 will be described with reference to the flowchart of FIG.

[0034] 2 is a diagram illustrating the processing of the control unit 101 from when the HMD 100 is started up, to when an application is launched, to when the application is terminated. The control unit 101 performs processing according to the flowchart in FIG. 2, allowing the user to experience VR or MR.

[0035] In step S201, the control unit 101 reads out the execution data of the application stored in the nonvolatile memory 110 and starts it up, and the process proceeds to step S202.

[0036] In step S202, the control unit 101 acquires an image captured by the imaging unit 102, stores it in the working memory 110, and then the process proceeds to step S203.

[0037] In step S203, the control unit 101 acquires sensor data from the attitude sensor unit 103 and stores it in the work memory 110, and then the process proceeds to step S204.

[0038] In step S204, the control unit 101 starts position and orientation estimation processing by the position and orientation estimation unit 104. When the processing is completed, the position and orientation estimation unit 104 stores the position and orientation information of the HMD 100, the generated space map information, and other additional information in the working memory 110. The other additional information includes the tracking state, images used for estimating the position and orientation, time information of the orientation sensor unit 103, and the like. The control unit 101 stores the acquired various data in the working memory 110 and proceeds to step S205.

[0039] In step S205, the control unit 101 starts detecting the user's gaze using the gaze detection unit 105. The gaze detection identifies the gaze point in the image acquired in step S202. The control unit 101 stores the detection result by the gaze detection unit 105 in the working memory 110, and proceeds to step S206. Note that if the information processing device does not have a gaze detection unit, the control unit 101 omits step S205, and proceeds to step S206 after processing step S204.

[0040] In step S206, the control unit 101 starts detecting the position of the subject in the image captured by the imaging unit 102 using the subject detection unit 106. The control unit 101 stores the detected subject information in the working memory 110, and proceeds to step S207.

[0041] In step S207, the control unit 101 starts generating a CG image, that is, an image of a virtual object, by the image generation unit 107, and the process proceeds to step S208.

[0042] In step S208, the control unit 101 starts image synthesis by the image synthesis unit. The control unit 101 uses the latest data on the position and orientation information, line of sight information, and subject information stored in the working memory 110 to set the transparency of the CG image, and performs synthesis so that the CG image is superimposed on the captured image to generate a display image. The control unit 101 stores the generated display image in the working memory 110, and proceeds to step S209. Note that, taking into consideration the depth of the subject in 3D space, occlusion processing may be performed so that the subject hides the CG when the subject is in front of the CG during synthesis.

[0043] In step S209, the control unit 101 displays the image generated in step S208 on the display unit 209, and the process proceeds to step S210.

[0044] In step S210, the control unit 101 checks whether an instruction to end the application has been issued by a user operation. If an instruction to end the application has been issued, the control unit 101 ends the processing of the application. If an instruction to end the application has not been issued, the control unit 101 proceeds to step S202.

[0045] In this way, the control unit 101 controls the display of the image displayed on the display unit 209.

[0046] <Transparency processing explanation> The processing of the image synthesis unit 108 in step S208 will be described in detail with reference to FIGS. 3(a), 3(b), and 3(c).

[0047] In FIG. 3( a ), an image 301 indicates an image generated by the image synthesis unit 108 .

[0048] Image 302 is an image captured by imaging unit 102. Virtual object 303 is a CG object generated by the image processing unit, and an application for viewing videos, displaying documents, etc. is displayed. In other words, the user sees a composite image in which the virtual object is superimposed on an image in real space.

[0049] In this embodiment, the shape of the virtual object 303 is the shape of a window for displaying the execution contents of an application, but the shape is not limited to this and may be any shape.

[0050] The image composition unit 108 composes the captured image 302 so that the virtual object 303 is displayed on the captured image 302, with the captured image 302 being the background of the image.

[0051] By performing the above processing in real time, users can experience a visual experience that feels like CG is being displayed in real space.

[0052] On the other hand, when there are people nearby as shown by 304 in FIG. 3(b), it may be better to make the people visible even while the user is using the device.

[0053] To address such cases, the image synthesis unit 108 acquires subject information detected by the subject detection unit 106, and superimposes the virtual object superimposed on the person on the captured image by making the virtual object transparent so that the user can see the person. Here, the transparency process refers to reducing the visibility of the object, and includes, for example, reducing the display brightness or projection light intensity of the object to increase transparency, and adjusting the color of the object. It also includes thinning the lines of a text object and partially displaying the object (including displaying it in a mesh or dot pattern). In addition to performing the transparency process in the image synthesis unit 108, the image processing unit 107 may generate a virtual object that has been subjected to transparency processing, and the image synthesis unit 108 may synthesize the virtual object that has already been subjected to transparency processing.

[0054] If the virtual object is a transparent object, the image synthesis unit 108 sets the transparency of the CG object around the user, indicated by 305, so that the subject 306 is visible to the user, as shown in FIG. 3(c), and synthesizes it with the background image 302.

[0055] The determination of whether or not to allow the subject to be seen through may be made so that the subject is seen through when the subject is within a predetermined distance from the user. Also, instead of a person as described above, the subject may be a pet or other moving object.

[0056] 3(c) shows an example in which the periphery of the subject (person) is transparent, but the entire virtual object 303 may be transparent. The transparency may be partially changed while the entire object is transparent, or the entire virtual object may have a constant transparency. The transparency process may be started when the subject approaches.

[0057] In this way, when another person approaches the user or speaks to the user, communication with the other person is possible without being visually obstructed by a virtual object.

[0058] <Example of a screen after virtual objects are made transparent> Furthermore, when the virtual object is made transparent so that the subject can be seen by the user through the above-described processing, depending on the situation in which it is being used, the user may wish to cancel the transparent state and continue working. The processing of the image synthesis unit 108 to deal with such situations will be described in detail with reference to Figures 4(a), 4(b), and 4(c).

[0059] Figure 4(a) shows a situation in which a virtual object 401 is transparent and a surrounding person 402 is visible to the user. The user's gaze point 403 is also displayed as a black circle. For example, when the user is gazing at a subject, the virtual object may be transparent like the virtual object 401 in Figure 4(a), but if the user wants to continue working on manipulating the virtual object, a non-transparent virtual object may suit the user's intention.

[0060] FIG. 4(b) is a diagram showing an example of an image visually recognized by the user when the virtual object is made opaque in the situation shown in FIG. 4(a).

[0061] The image synthesis unit 108 determines whether to cancel the transparent state based on the user's gaze point detected by the gaze detection unit 105, the subject information detected by the subject detection unit 106, and the CG image and CG information generated by the image processing unit 107. For example, if the user's gaze moves from gaze point 403 in FIG. 4(a) to gaze point 413 in FIG. 4(b), and the user's gaze point is on the virtual object but not in the area where person 412 is located, it can be determined that the user's attention is on the virtual object. In such a case, the transparency state of the virtual object is set to a non-transparent state as shown by virtual object 411. By making the CG area that was previously invisible through the transparency visible again to the user in this way, the user can continue working without being disturbed by the transparent state.

[0062] On the other hand, depending on the situation, maintaining the transparent state may be more in line with the user's intention.

[0063] FIG. 4(c) is a diagram showing an example of an image visually recognized by the user when the transparent state of the virtual object is maintained in the situation shown in FIG. 4(a).

[0064] If the user's gaze point moves from the state shown in FIG. 4( a) to a point on the transparent virtual object as shown by gaze point 423 and also on the person 422 behind the virtual object, it is considered that the user is gazing at the person, and therefore it can be determined that the user's attention is directed toward the person. Therefore, the image synthesis unit 108 maintains the transparency of the virtual object as shown by virtual object 421. By maintaining the transparency in this manner, the user can continue communicating with the other person without being interrupted by the virtual object. In addition to maintaining the transparency of the virtual object, the virtual object may be made more transparent. For example, the transparency may be further increased or the area of the transparent virtual object may be enlarged. In addition, if it is possible to determine whether the user is looking at a virtual object or a person based on the depth, the convergence angle or the like may be used to determine this. If it is possible to determine whether the user is looking at a virtual object or a person, a process of changing the transparency or maintaining the transparency may be performed according to that information.

[0065] 4(a) to the position indicated by point of gaze 424, it is possible that the user's attention is directed to something other than the virtual object or person. In this case, since the user cannot determine whether their attention is directed to the person or the virtual object, the image synthesis unit 108 maintains the transparent state of the virtual object 421 to prevent frequent changes in the transparency state of the virtual object, which can make it difficult to see.

[0066] <Example of a screen where a person is hidden by a non-transparent virtual object> FIG. 5(a) is a diagram showing a situation in which a person 502 is hidden by a virtual object 501 because the virtual object 501 is in a non-transparent state.

[0067] When the user's point of gaze is at the position of the point of gaze 503, the viewpoint is in the area of the virtual object and not in the area of the person, so the virtual object 501 is in a non-transparent state.

[0068] When the user's point of gaze moves from the position of the point of gaze 503 to the position of the point of gaze 504, the point of gaze 504 is on the virtual object and also in the area where the person 502 is located.

[0069] In such a situation, since the virtual object is visible to the user when it is at the position of the gaze point 503, when the user's gaze point moves to the position of the gaze point 504, it can be determined that the user's attention is directed to the virtual object 501.

[0070] For this reason, the image synthesis unit 108 keeps the virtual object in a non-transparent state as shown by virtual object 501. In other words, the transparency is not increased.

[0071] Furthermore, when the user's viewpoint moves to the position of the gaze point 505, it is not possible to determine whether the user's gaze point will be directed to the person or the virtual object, and therefore the image synthesis unit 108 maintains the transparency of the virtual object in a non-transparent state.

[0072] However, in Figure 5(a), if the user's gaze point moves to an area where the person is not hidden by the CG, it can be determined that the user's attention is directed toward the person, so it may be better to make the virtual object transparent.

[0073] FIG. 5(b) is a diagram showing an example of an image visually recognized by the user when the virtual object is made transparent in the situation shown in FIG. 5(a).

[0074] When the user's line of sight moves and the point of gaze moves to the position indicated by the point of gaze 513, the point of gaze 513 is in the area of the person 512, so the image synthesis unit 108 makes the virtual object 511 transparent so that the person 512 can be seen by the user.

[0075] FIG. 5(c) is a diagram showing an example of an image visually recognized by the user when the virtual object is made transparent in the situation shown in FIG. 5(a).

[0076] As another method for making the virtual object transparent so that the person can be seen from the state shown in FIG. 5(a), a UI (user interface) for explicitly changing the transparent state of the virtual object may be displayed.

[0077] When a virtual object is opaque, as in Fig. 5(a), the person may be completely hidden by the virtual object. In such a case, the method described in Fig. 5(b) cannot make the virtual object transparent so that the person can be seen, so an operation UI is displayed as shown by virtual object 523.

[0078] The virtual object 523 is a UI that is displayed on the screen when a person is displayed so as to be hidden by a virtual object, and serves as a discriminator for the user to explicitly specify the transparency state.

[0079] In this case, when the user's gaze point moves onto 523 (the gaze point is not shown), the virtual object 521 is changed to a transparent state so that the person 522 is visible to the user.

[0080] Note that virtual object 523 may be displayed only in a situation where, for example, virtual object 501 completely hides a person and it is not possible to move the line of sight onto the person. Note that virtual object 523 may also be displayed when a person is recognized in a captured image. Note that virtual object 523 may be displayed at all times, and when a person is hidden by a virtual object, the display may be changed to notify the user of the presence of a person. Note that virtual object 523 may be displayed at all times, and when a person is hidden by a virtual object, the user may be notified by sound, vibration, or the like.

[0081] <Example of a screen when the gaze point moves from a location other than the virtual object or person when the virtual object is transparent> Using Figures 6(a), 6(b), and 6(c), we will explain an example of an image that a user sees when their gaze moves from a location other than the virtual object or the person onto the person in a situation where the virtual object is in a transparent state.

[0082] 6(a) is a diagram showing an example of an image viewed by a user in a situation where a virtual object 601 is set to a transparent state and a person 602 is visible to the user. The gaze point 405 indicates that the user is viewing an area different from the area of the virtual object 601 and the area of the person 602.

[0083] 6(b) is a diagram showing an example of an image visually recognized by the user when the user's gaze point moves from the state shown in FIG. 6(a) onto a person 612. In FIG.

[0084] As shown by gaze point 603, when the user's gaze is located in a location other than on a virtual object or person, if the user's gaze moves onto person 612 as shown by gaze point 613 in FIG. 6(b), it can be determined that the user's attention is on the person. For this reason, the image synthesis unit 108 performs synthesis processing so that the virtual object maintains a transparent state as shown by virtual object 611. Note that the virtual object may be displayed with a transparency greater than that of virtual object 601. For example, the transparency may simply be increased, or the transparent area may be expanded.

[0085] 6(c) is a diagram showing an example of an image visually recognized by the user when the user's line of sight moves from the state shown in FIG. 6(a) to a location on the virtual object as indicated by the gaze point 623, but not on the person 612. In such a case, it can be determined that the user's attention is on the virtual object 621. For this reason, the image synthesis unit 108 sets the virtual object to a non-transparent state as indicated by the virtual object 621, and performs synthesis processing.

[0086] In the above description, all of the processes described with reference to FIGS. 4 to 6 may be implemented, or only the necessary processes from FIGS. 4 to 6 may be implemented.

[0087] In this embodiment, a person is used as an example of the subject, but other moving objects such as pets may also be used.

[0088] Although the example of detecting a subject from within an image has been described, after detecting a subject from within an image, the gaze and voice of the detected person may be further discriminated to determine whether or not the person is directed at the user, and the subject detection result may be obtained. The device may be provided with a means for registering predetermined people in advance, so that when the detected subject is a person, the subject is included in the subject detection result only if the person is a person who has been registered in advance.

[0089] Furthermore, since the movement of the line of sight generally involves blurring, in line of sight detection by the line of sight detection unit 105, a space or time leeway may be provided for determining the position of the user's line of sight.

[0090] For example, when determining whether or not the object is on a person, if the point of gaze is in a predetermined area around the person, it may be determined that the object is on the person. Also, when the point of gaze moves, the determination may not be made immediately, but may be made after a predetermined time has passed.

[0091] It is also possible to detect the depth of gaze from the angle of convergence of the user's eyes and determine the spatial gaze point.

[0092] In this embodiment, an example is described in which the virtual object is made opaque, but the transparency may be set to a lower level without being made completely opaque.

[0093] Note that a means for notifying people in the vicinity depending on the transparency state may be provided outside the HMD so that they can know whether the HMD user is looking at them. For example, a display provided outside the HMD, for example, on the opposite side of the display placed in front of the user's eyes, may be used to notify people in the vicinity so that they can know whether the HMD user is looking at them. Note that even if the HMD user is not looking at them, the surrounding people may be notified so that they recognize that the user is looking at them. Also, for example, an LED may be provided so that it lights up when the person is being viewed.

[0094] <Flowchart for processing virtual objects for transparency> The procedure of the process performed in the process of S208 in Fig. 2 will be described using the flowchart in Fig. 10. Note that instead of being performed in S208, the process of Fig. 10 may be performed in the process of S207 in Fig. 2. Here, it is assumed that a subject has been detected and the transparency of the virtual object has been increased. The following flowchart will be described assuming a scene in which a subject has been detected and the transparency of the virtual object has already been increased before this process is started.

[0095] In step S1001, the control unit 101 determines whether the line of sight detected in step S205 is on a virtual object. If the control unit 101 determines that the line of sight is on a virtual object, the process proceeds to step S1002, and if the control unit 101 determines that the line of sight is not on a virtual object, the process proceeds to step S1003. Note that the line of sight may be determined based on the position of the line of sight when the image is converted into a two-dimensional image without taking depth into consideration, or the line of sight may be determined three-dimensionally.

[0096] In step S1002, the control unit 101 reduces the transparency of the virtual object that the line of sight is in, that is, approaches a non-transparent state, and then ends the process. Note that the virtual object may be made completely non-transparent, or the transparency may be reduced to remain transparent. Note that a process of reducing the transparent area may also be performed. Gradually reducing the transparent area has the effect of preventing the user from suddenly switching their field of view.

[0097] In step S1003, control unit 101 determines whether the gaze position is on a person in the background video. If control unit 101 determines that the gaze is on a person, the process proceeds to step S1004. If control unit 101 determines that the gaze is not on a person, that is, if control unit 101 determines that the gaze is on a background video that does not include a person, the process proceeds to step S1005. Here, the person in the background video is the subject detected in step S206.

[0098] In step S1004, the control unit 101 increases the transparency of the virtual object that the line of sight is in, making the person more visible, and then ends the process. Note that if the virtual object is already completely transparent, or if the transparency of the virtual object has been increased to a transparency previously set by the user, control is performed to maintain the transparency. Note that the transparency of the area overlapping with the person may be increased, or the transparent area may be widened.

[0099] In step S1005, the control unit 101 maintains the transparency of the virtual object and ends the process. Note that, since there is no line of sight on the person, the transparency of the virtual object may be reduced. In FIG. 10, the transparency is maintained because it is impossible to determine whether the line of sight is on either the virtual object or the person. However, since there is no line of sight on the person, it may be determined that there is no need to make the person visible, and the transparency of the virtual object may be reduced. Conversely, the transparency of the virtual object may be increased or the area through which the virtual object is transparent may be expanded so that the virtual object does not get in the way of the viewer viewing the real image of the background.

[0100] In step S1001, it may be determined how the gaze has moved since a time prior to the predetermined time point, rather than the gaze position in the captured image at the predetermined time point. The degree of change in transparency may be changed depending on how the gaze has moved. For example, if the user has been looking at a virtual object since a past time point, the degree of change may be increased so as to further reduce the transparency of the virtual object. For example, if the gaze was on a person at a past time point and the gaze has moved to a virtual object at a predetermined time point, the transparency of the virtual object may be slightly reduced to reduce the degree of change.

[0101] Note that Figure 10 assumes a scene in which a subject has been detected and the transparency of the virtual object has already been increased, but the processing in Figure 10 may be performed so that the transparency does not increase just when a subject is detected, but rather when the user's gaze moves onto a person who is partially visible.

[0102] (Second embodiment) An example of an image viewed by a user on an information processing device according to a second embodiment of the present invention will be described below with reference to Figures 7(a) and 7(b). The second embodiment is a system of this information processing device that operates in VR mode.

[0103] The basic configuration of this information processing system is the same as that of the first embodiment, but the processing of the image synthesis unit 108 is different.

[0104] Fig. 7(a) shows an example of a display image in VR mode. In Fig. 7(a), image 701 is generated by the image synthesis unit 108, and shows a situation in which the user cannot visually recognize an image of real space. Image 701 is generated by superimposing a virtual object 703 on a CG background image 702.

[0105] Reference numeral 702 denotes a background image, which shows an example of displaying an image that has been prepared in advance and stored in the non-volatile memory 111. By using an image that is different from the space in which the user is, the user can have the experience of being transported to a different space.

[0106] The background image is not limited to a recorded image, but may be a CG image that changes in real time depending on the situation. Note that the background image may be an image recorded in an external device (such as a server) that is acquired via the communication unit 112.

[0107] The virtual object 703 is a virtual object generated by the image processing unit 108, and displays an application for viewing videos, displaying documents, etc. In this embodiment, as in the first embodiment, the shape of the virtual object 703 can be any shape.

[0108] By performing the above processing in real time, users can experience a visual experience that feels like CG is being displayed in a virtual space.

[0109] In the VR mode of this embodiment, images captured by the imaging unit 102 are not used in normal use, and therefore the user cannot see what is happening in the surrounding real space.

[0110] FIG. 7(b) is a diagram showing an example of an image visually recognized by the user when a person is detected in the surroundings by the subject detection unit 106 while the user cannot see the surrounding real space.

[0111] When a person is detected by the subject detection unit 106, the image synthesis unit 108 acquires the area of the person detected by the subject detection unit 106. The image synthesis unit 108 further sets the transparency of a background image and a virtual object for the acquired area around the person, and synthesizes it with the image captured by the imaging unit 102.

[0112] As a result, the person 714 appears to be transparent to the background image 712 and the virtual object 713, allowing the user to visually confirm the people around them.

[0113] Note that the determination of whether to transmit the subject (person 714) may be such that it transmits only when the distance of the subject is within a predetermined distance from the HMD 100. Note that the determination of whether to transmit the subject may, in addition to the detection of the subject, determine whether the subject is speaking, and may also detect the direction in which the body or face of the subject is facing. Note that it may also detect whether the subject is moving their mouth or opening their mouth, or may detect the line of sight of the subject. Note that based on the actions of the subject, such as whether the subject is walking, standing still, or performing any other operations, a determination of whether to transmit the subject may be made. For example, when the line of sight, body, or face of the subject is facing the user, and the subject is moving their mouth while making gestures, the image may be processed so as to transmit the subject. Note that the determination of whether to transmit the subject may appropriately combine any of the above, or may be based on any one item.

[0114] <Example of a screen based on a fixation point when a virtual object is in a transparent state in VR> Using FIGS. 8(a), 8(b), and 8(c), an example of a screen based on a fixation point after a virtual object has entered a transparent state in VR will be described.

[0115] FIG. 8(a) shows an image in a state where the people around are seen as being transmitted.

[0116] At this time, when the user's line of sight is on the person 714 detected, the virtual object and the background image maintain the transparent state as shown by the virtual object 801 and the background image 802. Even when the user's line of sight is at the position of the virtual object like the fixation point 803, if the line of sight is on the transmitted person 714, the transparent state is maintained as in FIG. 8(a).

[0117] FIG. 8(b) is a diagram for explaining the process for setting to a non-transparent state in the situation shown in FIG. 8(a).

[0118] As shown by the gaze point 813, when the user's line of sight is on the virtual object 811 and at a position not on the person seen through it, the virtual object and the background image are set to a non-transparent state as shown by the virtual object 811 and the background video 812. As described in FIG. 7(b), when subject detection is performed again, the person may also be made transparent. In addition, once the state changes from the transparent state to the non-transparent state, the same person may not be made transparent for a certain period of time. As described later, once the state changes from the transparent state to the non-transparent state, as shown in FIG. 8(c), the transparent state may be different from the transparent state when it was in the transparent state in the past. For example, the transparency may be made lower than in the past, or the area to be made transparent may be made smaller. Also, when there are a plurality of virtual objects, only some of the virtual objects may be made transparent and the others may not be made transparent, or only the CG serving as the background video may be made transparent.

[0119] In the VR mode of the present embodiment, the background image that was originally visible is hidden as in the example of FIG. 8(a). Therefore, when the user's line of sight moves to a position on the background image 811, that is, when it moves to the position of the gaze point 814, it can be determined that the user intends to check the background image. For this reason, even when the user's line of sight is not on the virtual object 811 as shown by the gaze point 814, the image composition unit 108 may make the background image non-transparent and not display the person.

[0120] Also, as shown in FIG. 8(c), when the user's line of sight is on the virtual object 811 and at a position not on the person 714, the image composition unit 108 may make the virtual object 821 non-transparent and keep the background image 822 in the transparent state. That is, as shown by the person 714 in FIG. 8(c), the user may be made to recognize that the person 714 is present.

[0121] <Screen example in the case of setting the transparency according to the distance to the virtual object, person, and line of sight in VR> In the processing of the image synthesis unit 108 of this embodiment, an example has been described in which the transparency state is set only to transparent or non-transparent, but the transparency may also be set according to the distance from the virtual object and person to the line of sight.

[0122] FIG. 9 is a diagram for explaining the above-described processing of the image synthesis unit 108.

[0123] 9(a), 901 represents the synthesized image, 902 represents a person in the vicinity, 903 represents a virtual object, and 904 represents a background image. Here, the person 902 is visible through the background image 904. In this way, when the user's line of sight is on the person 902 as shown by the gaze point 905, the image synthesis unit 108 sets the virtual object and background image to a transparent state so that the user can see the person 902, as explained above.

[0124] Next, when the user's line of sight moves, the transparency of the virtual object and the background image is set so that the degree of transparency of the person is determined according to the distance between the line of sight and the person and the distance between the line of sight and the virtual object.

[0125] FIG. 9B shows an example in which a gaze point 915 is located at the midpoint between a virtual object 903 and a person 912.

[0126] In contrast to the state in FIG. 9(a) in which the person is completely visible, in the example in FIG. 9(b), the transparency of the virtual object 903 and background image 914 is set so that about half of the person is visible, as shown by person 912.

[0127] In the example of FIG. 9(b), the transparency increases from the periphery of the person, but the transparency of the entire area of the person may be made uniform depending on the distance.

[0128] FIG. 9(c) shows an example where the user's line of sight further moves and the position of the user's line of sight is on the virtual object. Compared with the background image 914 in FIG. 9(b), in the background image 924 of FIG. 9(c), the non-transparent area in the background image has become wider. At this time, the image compositing unit 108 may set the transparency of the virtual object and the background image to be completely non-transparent, or as shown in 922, may reduce it to a transparency level that allows it to be understood that there is a person without being completely non-transparent.

[0129] Note that the distance between the line of sight and the person, and the distance between the line of sight and the virtual object may be determined by the shortest distance, or may be determined by the distance from any arbitrary point.

[0130] Although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0131] <Flowchart related to the process of making a virtual object transparent in VR> The procedure of the process performed in the process of S208 in FIG. 2 will be described using the flowchart of FIG. 11. Note that instead of being performed in S208, the process of FIG. 11 may be performed in the process of S207 in FIG. The flowchart of FIG. 11 assumes VR, so it assumes a scene where the background video and virtual objects are not usually transparent.

[0132] In step S1101, the control unit 101 determines whether a virtual object was made transparent in the image generated immediately before. Here, the information processing device of this embodiment estimates the position and orientation of the information processing device, for example, each time the imaging unit 102 acquires a captured image, generates an image of a virtual space, and displays it on the display unit 109. When generating an image of a virtual space at a predetermined time, the subsequent determination differs depending on whether the CG was made transparent in the image generated immediately before. Therefore, if the control unit 101 determines that a virtual object was made transparent in the image generated immediately before, the process proceeds to step S1102. If the control unit 101 determines that a virtual object was not made transparent in the image generated immediately before, the process proceeds to step S1107. Here, the image generated immediately before may be stored in the working memory 110. In this case, the control unit 101 reads the immediately before image stored in the working memory 110 and performs the determination in step S1101.

[0133] In step S1102, the control unit 101 determines whether the gaze position is on a virtual object. If the control unit 101 determines that the gaze is on a virtual object, the process proceeds to step S1103, and if the control unit 101 determines that the gaze is not on a virtual object, the process proceeds to step S1104. Note that the gaze position may be determined based on the gaze position when the image is converted into a two-dimensional image without taking depth into consideration, or the gaze position may be determined three-dimensionally. Also, in the process of step S1102, if the gaze position is on the image of the virtual space, which is the background image, the control unit 101 determines that the gaze is not on a virtual object.

[0134] In step S1103, the control unit 101 lowers the transparency of the virtual object that the line of sight is in, that is, brings it closer to a non-transparent state, and ends the process. Note that the virtual object may be made completely non-transparent, or the transparency may be reduced to make it transparent. Note that processing to reduce the transparent area may be performed. Gradually reducing the transparent area has the effect of preventing the user from suddenly switching their field of view. Note that the transparency of only the virtual object may be changed, or the transparency of the background image may be reduced while the transparency of the virtual object is reduced. Also, the transparency of the virtual object and the background image may be the same or different.

[0135] In step S1104, the control unit 101 determines whether the gaze position is on a person in the background image. If the control unit 101 determines that the gaze is on a person, the process proceeds to step S1105. If the control unit 101 determines that the gaze is not on a person, that is, if the control unit 101 determines that the gaze is on an image of a virtual space that is the background and not a person, the process proceeds to step S1106. Here, the person in the background image is the subject detected in step S206.

[0136] In step S1105, the control unit 101 increases the transparency of the virtual object that the line of sight is in, making the person more visible, and then ends the process. Note that if the virtual object is already completely transparent or if the transparency of the virtual object has been increased to a transparency previously set by the user, control is performed to maintain the transparency. Note that the transparency of the area overlapping with the person may be increased, or the transparent area may be widened. Note that not only the transparency of the virtual object but also the transparency of the image of the background virtual space may be increased. Note that when making the virtual object transparent, the area closer to the person's face may be more transparent, and the background part of the real space around the person may be made less transparent, i.e., relatively opaque.

[0137] In step S1106, the control unit 101 maintains the transparency of the virtual object and ends the process. Note that in the process of step S1106, the image that serves as the background video corresponds to real space in FIG. 10, but corresponds to virtual space in FIG. 11. Therefore, in step S1106, the control unit 101 may maintain the transparency of the virtual object and the background video, since it is impossible to determine whether the line of sight is on either the virtual object or the person, as described above. Note that the transparency of the virtual object and the background video may be reduced, assuming that the user wants to see the background of the virtual space.

[0138] In step S1107, the control unit 101 determines whether or not a subject was detected in step S206. If the control unit 101 determines that a subject was detected in step S206, the process proceeds to step S1108, and if the control unit 101 determines that a subject was not detected in step S206, the process proceeds to step S1109.

[0139] In step S1107, the control unit 101 may determine whether or not a subject can be detected from a captured image of the real space stored in the working memory 110. In such a case, step S206 may be omitted.

[0140] It should be noted that the determination may be made not based on whether a subject has been detected, but based on whether the subject has approached, or based on whether the subject is speaking to the user. For example, if the subject is approaching, the transparency of the virtual object is increased, and if the subject is speaking to the user, the transparency of the virtual object is increased.

[0141] In step S1108, the control unit 101 increases the transparency of the virtual object and ends the process. Note that not only the virtual object but also the image of the background virtual space is made transparent. Here, the transparency of the virtual object and the image of the background virtual space may be the same or different. For example, the image of the background virtual space may be made transparent so that the background in the area overlapping with the subject cannot be seen by the user, and the virtual object may be faintly visible.

[0142] In step S1109, the control unit 101 ends the process without making the virtual object and background image transparent.

[0143] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0144] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0145] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0146] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0147] [Configuration 1] image acquisition means for acquiring a captured image; A gaze acquisition means for acquiring a user's gaze; a display control means for displaying a composite image obtained by combining the captured image and a virtual object on a display means, the display control means controls to determine a transparency of a virtual object overlapping with a person included in the captured image based on a line of sight of the user. 1. An information processing device comprising:

[0148] [Configuration 2] When a moving object included in the captured image acquired by the image acquisition means and the virtual object overlap as viewed from the user, the display control means increases the transparency of at least a part of an area of the virtual object that overlaps with the moving object, and thereafter, further determines the transparency of the virtual object whose transparency has been increased based on the line of sight of the user acquired by the line of sight acquisition means. 2. The information processing device according to configuration 1,

[0149] [Configuration 3] the display control means further increases or maintains the transparency of the virtual object whose transparency has been increased when the line of sight is at the position of the person for a predetermined period of time; 3. The information processing device according to configuration 2.

[0150] [Configuration 4] the display control means reduces the transparency of the virtual object whose transparency has been increased when the line of sight is in the area of the virtual object for a predetermined period of time; 4. The information processing device according to configuration 2 or 3.

[0151] [Configuration 5] the display control means reduces the transparency of the virtual object whose transparency has been increased when the line of sight is in a region of the virtual object other than a region overlapping with the person for a predetermined period of time; 5. The information processing device according to configuration 4.

[0152] [Configuration 6] the display control means, when the line of sight moves from the area of the person to the area of the virtual object, decreases the transparency of the virtual object whose transparency has been increased; 6. The information processing device according to any one of configurations 2 to 5.

[0153] [Configuration 7] the display control means maintains the transparency of the virtual object when the line of sight is in a region of real space different from a region of the person for a predetermined period of time. 7. The information processing device according to any one of configurations 2 to 6.

[0154] [Configuration 8] the display control means maintains the transparency of the virtual object when the line of sight is on a background image of the virtual space for a predetermined period of time. 8. The information processing device according to any one of configurations 2 to 7.

[0155] [Configuration 9] the display control means controls the virtual object to be transparent when the person is speaking, even if the line of sight is not in an area of the person; 9. The information processing device according to any one of configurations 1 to 8.

[0156] [Configuration 10] the display control means controls the virtual object to be transparent when the line of sight of the person is directed toward the user, even if the line of sight is not in an area of the person; 10. The information processing device according to any one of configurations 1 to 9.

[0157] [Configuration 11] the display control means determines the transparency of the virtual object in accordance with the position of the line of sight and the distance from the person. 11. The information processing device according to any one of configurations 1 to 10.

[0158] [Configuration 12] the display control means determines the transparency of the virtual object according to the position of the line of sight and the distance to the virtual object. 12. The information processing device according to any one of configurations 1 to 11.

[0159] [Configuration 13] The method further includes acquiring a convergence angle of the user, the display control means controls to determine a transparency of a virtual object overlapping with the person included in the captured image based on the convergence angle and the line of sight. 13. The information processing device according to any one of configurations 1 to 12.

[0160] [Configuration 14] When the person is a person registered in advance, the display control means controls the virtual object to be transparent even when the line of sight is not in an area of the person. 14. The information processing device according to any one of configurations 1 to 13.

[0161] [Configuration 15] Further having a second display means different from the display means, the display control means controls the second display means to change the display when the transparency of the virtual object is changed. 15. The information processing device according to any one of configurations 1 to 14.

[0162] [Configuration 16] When the virtual object is made transparent, the display control means controls the second display means to display the eyes of the user. 16. The information processing device according to configuration 15.

[0163] [Configuration 17] When the transparency of the virtual object is reduced, the display control means displays an image on the second display means that is different from what the user sees. 17. The information processing device according to configuration 16.

[0164] [Control method] an image acquisition step of acquiring a captured image; a gaze acquisition step of acquiring a gaze of a user; a display control step of displaying a composite image obtained by combining the captured image and a virtual object on a display means, the display control step performs control to determine a transparency of a virtual object overlapping with a person included in the captured image based on a line of sight of the user. 2. A method for controlling an information processing apparatus comprising:

[0165] [program] 19. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 18.

[0166] [system] A display device; an image acquisition device that acquires a captured image; a gaze acquisition device that acquires a user's gaze; a display control device that displays a composite image obtained by combining the captured image and a virtual object on the display device, the display control device controls to determine a transparency of a virtual object overlapping with a person included in the captured image based on a line of sight of the user; An information processing system comprising:

Claims

1. image acquisition means for acquiring a captured image; A gaze acquisition means for acquiring a user's gaze; a display control means for displaying a composite image obtained by combining the captured image and a virtual object on a display means, the display control means controls to determine a transparency of a virtual object overlapping with a person included in the captured image based on a line of sight of the user.

1. An information processing device comprising:

2. When a moving object included in the captured image acquired by the image acquisition means and the virtual object overlap as viewed from the user, the display control means increases the transparency of at least a part of an area of the virtual object that overlaps with the moving object, and thereafter, further determines the transparency of the virtual object whose transparency has been increased based on the line of sight of the user acquired by the line of sight acquisition means.

2. The information processing apparatus according to claim 1, wherein:

3. the display control means further increases or maintains the transparency of the virtual object whose transparency has been increased when the line of sight is at the position of the person for a predetermined period of time; 3. The information processing apparatus according to claim 2, wherein:

4. the display control means reduces the transparency of the virtual object whose transparency has been increased when the line of sight is in the area of the virtual object for a predetermined period of time; 3. The information processing apparatus according to claim 2, wherein:

5. the display control means reduces the transparency of the virtual object whose transparency has been increased when the line of sight is in a region of the virtual object other than a region overlapping with the person for a predetermined period of time; 5. The information processing apparatus according to claim 4,

6. the display control means, when the line of sight moves from the area of the person to the area of the virtual object, decreases the transparency of the virtual object whose transparency has been increased; 3. The information processing apparatus according to claim 2, wherein:

7. the display control means maintains the transparency of the virtual object when the line of sight is in a region of real space different from a region of the person for a predetermined period of time.

3. The information processing apparatus according to claim 2, wherein:

8. the display control means maintains the transparency of the virtual object when the line of sight is on a background image of the virtual space for a predetermined period of time.

3. The information processing apparatus according to claim 2, wherein:

9. the display control means controls the virtual object to be transparent when the person is speaking, even if the line of sight is not in an area of the person; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

10. the display control means controls the virtual object to be transparent when the line of sight of the person is directed toward the user, even if the line of sight is not in an area of the person; 3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

11. the display control means determines the transparency of the virtual object in accordance with the position of the line of sight and the distance from the person.