Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus that allows suitable communication in two or more MR experiences.SOLUTION: An information processing apparatus controls display of a display unit worn by a user. The information processing apparatus comprises: an acquisition unit that acquires a line-of-sight image of another user wearing another display unit, and information on a virtual object corresponding to a real object arranged in a space where the another user is present; and a creation unit that, when a line of sight of the user to the virtual object displayed on the display unit and a line of sight of the another user to the real object satisfy predetermined criteria, creates a composite image including at least the line-of-sight image of the another user.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed reality (MR) technology is known as a technology that blends the real world and the virtual world in real time. This technology seamlessly blends real space with a virtual space created by a computer. MR uses a head-mounted device (HMD) to allow users to experience the virtual space from a perspective closer to reality.

[0003] There is also known a technology that allows for the sharing of 3D models of people and objects in real time, which, when combined with an XR device, allows remote users to communicate in a space sharing the same objects.

[0004] To collect information to generate a 3D virtual object, methods using multi-view images acquired from multiple cameras, RGBD sensors such as TOF (Time Of Flight) sensors, or depth sensors can be used. Currently, in order to reproduce the target object from a large amount of 3D information in real time, it is necessary to accept a degradation in the image quality of the generated 3D model. In this case, when communicating using the generated 3D model, the degradation in image quality can lead to situations where the user's intended content is not conveyed.

[0005] For example, Patent Document 1 uses a technique for switching moving images displayed on an HMD using a VR video display system that can switch between viewpoints of multiple cameras. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2019-86578 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in Patent Document 1, the screen displayed on the HMD display is switched, which may reduce the user's sense of immersion. In a simultaneous experience shared by multiple people in an MR space, a technology for enabling suitable communication is desired.

[0008] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide a technology for generating real objects as virtual objects and enabling suitable communication in a simultaneous experience shared by multiple people in an MR space. [Means for solving the problem]

[0009] In order to achieve the above object, according to one aspect of the present invention, there is provided an information processing device that controls the display of a display device worn by a user, the information processing device comprising: an acquisition unit that acquires a gaze image of another user wearing another display device and information on a virtual object corresponding to a real object placed in a space where the other user is present; and a generation unit that generates a composite image including at least the gaze image of the other user when the user's gaze with respect to the virtual object displayed on the display device and the other user's gaze with respect to the real object satisfy a predetermined criterion. [Effects of the Invention]

[0010] According to the present invention, a suitable MR experience can be provided for the first user and the second user by using a two-dimensional viewpoint image of the first user holding the actual object in the MR experience. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 2 is a diagram showing the positional relationship between a first user and a second user according to the first embodiment. [Figure 2] FIG. 2 is a diagram showing the positional relationship when multiple users share an MR space according to the first embodiment. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a system according to a first embodiment. [Figure 4] FIG. 1 illustrates an example of a hardware configuration of an information processing device. [Figure 5] FIG. 3 is a diagram showing the positional relationship of a first user according to the first embodiment. [Figure 6] FIG. 4 is a diagram showing the positional relationship of a second user according to the first embodiment. [Figure 7] FIG. 2 is a diagram illustrating the positional relationship between a first user and a second user during implementation of the first embodiment. [Figure 8] 10A and 10B are diagrams showing line-of-sight images of a first user and a second user who are implementing the first embodiment. [Figure 9] 10 is a flowchart of a gaze image display determination process according to the first embodiment. [Figure 10] 10 is a flowchart of a gaze image display position and orientation calculation process according to the first embodiment. [Figure 11] 10 is a flowchart of a line-of-sight image display determination process according to a modified example of the first embodiment. [Figure 12] 10 is a diagram showing an overlapping area (common area) of two spheres, a fixation point area of a first user and a fixation point area of a second user, on an xy plane. [Figure 13] 10 is an example of a flowchart illustrating a process performed until a gaze image of a first user acquired by an information processing device is displayed to a second user through the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Figure 1 is a schematic diagram showing the positional relationship between two users. Figure 1(A) shows a first user 101 wearing an HMD 102 in a space where an arbitrary real object 103 and a three-dimensional device 104 are placed. Note that the real object 103 is an example of a real object.

[0013] 1(B) shows a second user 105 wearing an HMD 106 in a space located away from the first user 101, where the real object 103 shown in FIG. 1(A) is not located. Here, the second user 105 is an example of a user, and the first user 101 is an example of another user. Furthermore, the HMD 106 worn by the second user 105 is an example of a display device, and the HMD 102 worn by the first user 101 is an example of another display device.

[0014] In this example, three-dimensional information of a real object 103 is acquired by a three-dimensional device 104, and a virtual object 107 is generated based on that information and shared with an information device of a second user 105.

[0015] FIG. 2 is a schematic diagram of a case where a virtual object 107 is shared with a second user 105. The first user and the second user 105 communicate while observing the real object 103 and the virtual object 107, respectively, through HMDs 102 and 106 worn by each user. The shared virtual object 107 may not accurately reproduce the shape of the real object 103, and in order to represent this in the schematic diagram, parts of the real object 103 may be missing. In such cases, discrepancies may arise in communication between the first user 101 and the second user 105 in the MR space.

[0016] This causes a discrepancy between the appearance of the real object 103 seen by the first user 101 and the appearance of the virtual object 107 seen by the second user 105, resulting in confusion in communication.

[0017] In the present invention, the above problem is alleviated by presenting a gaze image of first user 101 as two-dimensional information in the virtual space of second user 105. A specific embodiment of this will be described below.

[0018] The embodiment described below shows an example of a specific implementation of the present invention, and is one of the specific examples of the configuration described in the claims.

[0019] (First embodiment) In the first embodiment, an example is shown in which a gaze point area is calculated from the gaze points of the first user 101 and the second user 105, and an operation is performed to determine whether or not to present the gaze image of the first user 101 to the second user 105 based on the overlap of the gaze point areas of both users.

[0020] 3 is a block diagram showing an example of the configuration of a system according to the first embodiment of the present invention. In the system according to this embodiment, an information processing device 3100 is connected to an HMD 3000, a three-dimensional capture device 3400, and an information processing device 3300. In addition, in the system according to this embodiment, the information processing device 3300 is connected to an HMD 3200 and an information processing device 3100. Note that the configuration of HMD 102 shown in FIG. 1 will be described as HMD 3000, the configuration of HMD 106 as HMD 3200, and the configuration of three-dimensional capture device 104 as three-dimensional capture device 3400.

[0021] The connection may be either wired or wireless, or may be realized by combining wired and wireless connections. First, the information processing device 3100 will be described.

[0022] 4 is a hardware configuration diagram of an information processing device 3100 according to this embodiment. A CPU 401 comprehensively controls each device connected via a bus 400. The CPU 401 reads and executes processing steps and programs stored in a ROM 403 and a RAM 404.

[0023] The storage device 202 stores programs and data for use by the information processing device 3100, including various programs according to this embodiment.

[0024] The ROM 403 stores an operating system (OS), device drivers, and a boot program.

[0025] The RAM 404 temporarily stores programs and data loaded from the storage device 202 and the ROM 403, and has a work area where the CPU 401 executes each process as appropriate.

[0026] The input I / F 405 receives input signals from external devices including the HMD 3000 in a format that can be processed by the information processing device 3100 .

[0027] The output I / F 406 outputs the signal in a format that can be processed by an external device including the HMD 3000.

[0028] The gaze image acquisition unit 3110 acquires an image of the real world (gaze image) captured by the imaging unit 3010 of the HMD 3000. The acquired image is input to the gaze position / direction calculation unit 3120, the image synthesis unit 3160, and the transmission unit 3170.

[0029] The gaze position / direction calculation unit 3120 performs image processing using the real image acquired by the gaze image acquisition unit 3110, and measures the gaze position / orientation of the first user 101 by extracting characteristic information such as points and lines in the image. The gaze direction of the first user 101 may be calculated from the direction the user is facing, with the position of the HMD 3000 as the gaze start point, or may be calculated as a ray direction projected from an arbitrary point in the gaze image as the start point. The gaze position / orientation information of the first user 101 is input to the gaze point area calculation unit 3130.

[0030] The 3D model generation unit 3140 generates a virtual object 107 and measures the position of the virtual object 107 using the 3D information of the real object 103 acquired by capturing an image with the 3D capture device 3400. Note that the 3D capture device 3400 may include multiple cameras, a depth sensor, or a combination of a camera and a depth sensor.

[0031] The position of the virtual object 107 may be measured from multi-viewpoint images obtained from multiple cameras, or the position and orientation may be measured using infrared light, or the position and orientation may be measured using an ultrasonic sensor, a magnetic sensor, or a depth sensor. Alternatively, the position and orientation may be measured by displaying an index in space and estimating the position and orientation of the index displayed on the line-of-sight image. Alternatively, the position and orientation may be measured using a global navigation satellite system (GNSS) such as a global positioning system (GPS).

[0032] Furthermore, the virtual object 107 may use data (shape information and position and orientation information) relating to the virtual objects that make up the virtual space, data relating to the light source that illuminates the virtual space, and the like that is stored in advance.

[0033] The virtual object 107 is input to a transmission unit 3170 , and the position information of the virtual object 107 is input to a gaze point region calculation unit 3130 .

[0034] The gaze point region calculation unit 3130 calculates the gaze point region of the first user based on the gaze position and orientation information of the first user and the position information of the virtual object 107. Note that instead of the position information of the virtual object, position information of a real object corresponding to the virtual object may be used.

[0035] For example, the gaze area may be a spherical area having an arbitrary radius centered on the collision point (also called the contact point) between the ray in the gaze position and orientation direction and the virtual object 107 (which may be a real object). In this embodiment, calculation of the gaze area based on the collision point between the gaze ray of the first user and the virtual object 107 will be described, but the present invention is not limited to this.

[0036] It may be determined from the collision point between a ray cast from an arbitrary point on the gaze image and the virtual object 107, or it may be a point at an arbitrary distance in the forward direction determined from the gaze position and orientation instead of the collision point. The gaze point area of the first user is input to the image synthesis unit 3160.

[0037] 5 is a diagram showing a line-of-sight area of a first user in this embodiment. In an MR space where a real object 103 exists, the first user wears an HMD 102 mounted on his / her head and experiences the experience.

[0038] A gaze point area 501 calculated from the collision point between a ray sent from the HMD 102 in the line of sight and a virtual object 107 simulating a real object 103 acquired by a three-dimensional capture device 104 is displayed on the surface of the real object 103.

[0039] The transmission unit 3170 inputs the gaze image of the first user, the gaze point region 501 of the first user, and information on the generated virtual object 107 to the reception unit 3310 of the information processing device 3300.

[0040] The virtual image generation unit 3150 generates an image (virtual image) in which the calculated gaze point area 501 is arranged as a schematic spherical model as a visible area in the virtual space and is placed in the virtual space visible from the position and orientation calculated by the gaze position and direction calculation unit 3120. The generated virtual image is input to the image synthesis unit 3160.

[0041] The technology for generating an image of a virtual space that can be seen from a predetermined position is known, and therefore a detailed description thereof will be omitted.

[0042] The image synthesis unit 3160 generates an image of the mixed reality space (mixed reality image) by superimposing the virtual image generated by the virtual image generation unit 3150 on the real image acquired by the gaze image acquisition unit 3110. The generated mixed reality image is output to the display unit 3020 of the HMD 3000.

[0043] Next, the HMD 3000 will be described.

[0044] The HMD 3000 is a display device configured with a liquid crystal display or the like, with one screen for each of the right and left eyes, and is attached so as to be positioned in front of the right and left eyes of a user wearing the HMD 3000 on their head. Images with parallax are displayed in stereo on the left and right screens, respectively. In this embodiment, the HMD 3000 is described as a video see-through HMD that displays mixed reality images generated based on images captured by an imaging device on a display device. However, the present invention is not limited to this. The HMD may also be an optical see-through HMD that displays a virtual space image superimposed on a display medium that allows observation through real space.

[0045] The imaging unit 3010 captures an image of the real space displayed on the HMD 3000 and inputs the captured real image to the information processing device 3100.

[0046] The display unit 3020 displays the mixed reality image generated by the information processing device 3100.

[0047] Next, an information processing device 3300 will be described. Components having the same functions as those of the above-described information processing device 3100 are assigned the same reference numerals, and descriptions thereof will be omitted. Specifically, the position and orientation calculation unit 3110, gaze position and direction calculation unit 3120, and image synthesis unit 3160 are as described above, and descriptions thereof will be omitted.

[0048] The receiving unit 3310 is an example of an acquiring unit. The receiving unit 3310 acquires, for example, from the transmitting unit 3170 of the information processing device 3100, a gaze image of the first user, position and orientation information of the gaze point region 501 of the first user, and information of the generated virtual object 107.

[0049] The gaze image of the first user is input to the image synthesis unit 3160 , the position and orientation information of the gaze point area 501 of the first user is input to the gaze image display determination unit 3330 , and the information of the virtual object 107 is input to the gaze point area calculation unit 3320 .

[0050] The gaze point region calculation unit 3320 of the information processing device 3300 calculates the gaze point region of the second user 105 based on the position information of the virtual object 107 acquired from the receiving unit 3310 and the gaze position direction of the second user 105 acquired from the gaze position direction calculation unit 3120. The calculation method is the same as that of the information processing device 3100, and therefore will not be described.

[0051] The position and orientation information of the gaze point area of the second user 105 is input to the gaze image display determination unit 3330 and the gaze image display position and orientation calculation unit 3340 .

[0052] 6 is a diagram showing the line-of-sight area and MR space of the second user 105 in this embodiment. In the MR space where the virtual object 107 acquired by the receiving unit 3310 exists, the second user wears the HMD 106 mounted on his head and experiences the virtual object 107.

[0053] A gaze point area 601 calculated from the collision point between a ray sent from the HMD 106 in the line of sight and a virtual object 107 simulating a real object 103 acquired by a three-dimensional capture device 3400 is displayed on the surface of the virtual object 107.

[0054] The gaze image display determination unit 3330 determines whether to present the gaze image of the first user to the second user 105 based on the gaze point region 501 of the first user and the gaze point region 601 of the second user 105. For example, when the gaze point region 501 of the first user and the gaze point region 601 of the second user 105 satisfy a predetermined criterion, the gaze image display determination unit 3330 determines to present the gaze image of the first user to the second user 105. Furthermore, when the gaze point region 501 of the first user and the gaze point region 601 of the second user 105 do not satisfy the predetermined criterion, the gaze image display determination unit 3330 determines not to present the gaze image of the first user to the second user 105.

[0055] 12 is a diagram showing the overlapping area (common area) of two spheres, the fixation area 501 of the first user and the fixation area 601 of the second user 105, on the xy plane. When viewed on the xy plane, the volume of the body of revolution obtained by rotating the shaded area 1201 where the fixation area 501 and the fixation area 601 overlap, around the x-axis, is the volume of the common area of the two fixation areas. Since the method for calculating the volume of the common area between spheres is a known technique, a detailed description thereof will be omitted.

[0056] If the volume of this common area is larger than a pre-specified threshold, a gaze image display determination flag is turned ON to present the gaze image of the first user to the second user 105. In this embodiment, the gaze point areas of the first user and the second user 105 are assumed to be spherical, and whether to turn the flag ON is determined based on the volume of the common area, but the present invention is not limited to this.

[0057] This determination may be made based on a collision between the gaze area 501 of the first user and the gaze area 601 of the second user 105. Alternatively, the position and orientation of the surface where the virtual object 107 and the gaze area come into contact may be calculated in advance, and the determination may be made based on the common area between the surface of the gaze area 601 of the first user and the surface of the second user 105.

[0058] The starting position of the viewpoint of the first user 101 and the starting position of the viewpoint of the second user 105 may be acquired, and the determination may be made from the distance between the two points in the virtual space.

[0059] It may be determined whether or not the imaging range of the line-of-sight image of second user 105 includes gaze point region 501 of first user 101.

[0060] The virtual object 107 or a more detailed area within the virtual object 107 on which the gaze image is to be displayed may be specified in advance. Then, the determination criterion may be whether the gaze point areas of the first user 101, the second user 105, or both of them are gazing at the virtual object 107.

[0061] Alternatively, the ray direction of the first user 101 and the ray direction of the second user 105 may be acquired, and the angle formed between them may be used as the criterion for judgment.

[0062] Alternatively, the elapsed time when the common area of the gaze point areas is equal to or greater than a threshold may be used as the judgment criterion. The time when the common area of the gaze point areas is equal to or greater than a threshold may be used as the start time, and the area being maintained for a certain period of time or more may be used as the judgment criterion. Furthermore, a plurality of the judgment criteria described above may be combined. Gaze image display judgment flag information is input to the gaze image display position and orientation calculation unit 3340.

[0063] When the gaze image display determination flag information is ON, the gaze image display position and orientation calculation unit 3340 calculates the position and orientation of the gaze image of the first user 101 to be presented to the second user 105.

[0064] A plane perpendicular to the line of sight direction of second user 105 is calculated as the line of sight image display position and orientation. At this time, based on the position and orientation of virtual object 107 visible from the position of second user 105, the line of sight image display position may be adjusted to a position that does not hide the display of virtual object 107.

[0065] Furthermore, a depth value of virtual object 107 relative to the line of sight of the second user may be acquired, and control may be performed so that the virtual object is displayed in front of virtual object 107. Furthermore, a plane perpendicular to the line of sight of first user 105 may be calculated as the line of sight image display position and orientation, and displayed. The calculated line of sight image display position and orientation is input to virtual image generation unit 3350.

[0066] The virtual image generation unit 3350 of the information processing device 3300 is an example of a generation unit, and when the gaze image display determination flag information is ON, generates a composite image including at least the gaze image of the first user 101. Furthermore, when the gaze image display determination flag information is ON, the virtual image generation unit 3350 may generate a composite image including the gaze image of the first user 101 and a virtual object.

[0067] In addition, the virtual image generation unit 3350 generates a virtual image in which the gaze point area 601 of the second user 105 is placed in the virtual space as a schematic spherical model as a visible area in the virtual space, as seen from the position and orientation calculated by the gaze position and direction calculation unit 3120.

[0068] In addition, when the virtual object 107 is placed at a predetermined position and orientation in the MR space, a virtual image is generated that is placed in the virtual space that can be seen from the position and orientation calculated by the gaze position and direction calculation unit 3120.

[0069] If the gaze image display determination flag information is ON, a virtual image is generated in which the gaze image of first user 101 is arranged at the gaze image display position and orientation. The generated virtual image is input to image synthesis unit 3160.

[0070] 7A and 7B are schematic diagrams illustrating the gaze areas of the first user 101 and the second user 105 as viewed from the MR space of the second user in this embodiment. FIG. 7A shows the gaze areas when the first user 101 and the second user 105 are observing the same object. In this case, the common area between the gaze areas of the first user 101 and the second user 105 is larger than a pre-specified threshold. Therefore, the gaze image display determination unit 3330 turns on the gaze image display determination flag, and a gaze image 701 of the first user 101 is displayed in the virtual space of the second user 105.

[0071] 7(B) shows the gaze areas when the first user 101 and the second user 105 are observing different objects. In this case, since there is no common area between the gaze areas of the first user 101 and the second user 105, the gaze image display determination flag is not turned ON, and the gaze image 701 of the first user 101 is not displayed in the virtual space of the second user 105.

[0072] 8A and 8B are schematic diagrams showing the gaze images of the first user 101 and the second user 105 in this embodiment. Fig. 8A shows the gaze image 701 of the first user 101 looking at the real object 103 when the gaze image display determination unit 3330 turns on the gaze image display determination flag.

[0073] 8(B) shows the gaze image of the second user 105 looking at the virtual object 107 when the gaze image display determination flag is turned ON. Based on the gaze image display position and orientation calculated by the gaze image display position and orientation calculation unit 3340, the gaze image 701 of the first user 101 is synthesized onto the gaze image being viewed by the second user 105. At this time, if a part of the real object 103 is missing from the virtual object 107, a discrepancy in communication may occur between the first user and the second user. However, the second user 105 can observe the real object 103 by looking at the gaze image 701 of the first user 101.

[0074] 13 is an example of a flowchart showing the process of displaying the gaze image 701 of the first user acquired by the information processing device 3100 to the second user 105 via the information processing device 3300. Hereinafter, each process (step) will be explained by adding an S to the beginning of the reference numeral.

[0075] First, in step S13010, the gaze image acquisition unit 3110 acquires a gaze image from the imaging unit 3010. Then, the process proceeds to step S13020.

[0076] In step S13020, gaze position / direction calculation section 3120 performs image processing using the gaze image to measure the gaze position / orientation of first user 101. Then, the process proceeds to step S13030.

[0077] In step S13030, the three-dimensional model generation unit 3140 generates the virtual object 107 using the three-dimensional information of the real object 103 acquired by the three-dimensional capture device, and measures the position of the virtual object 107. Then, the process proceeds to step S13040.

[0078] In step S13040, the gaze point region calculation unit 3130 calculates the gaze point region of the first user based on the gaze position and orientation information of the first user and the position information of the virtual object 107. Then, the process proceeds to step S13050.

[0079] In step S13050, the transmission unit 3170 inputs the gaze image of the first user, the gaze point region of the first user, and information about the generated virtual object 107 to the reception unit 3310 of the information processing device 3300. Then, the process proceeds to step S13060.

[0080] In step S13060, the gaze image display determination unit 3330 determines whether to present the gaze image of the first user to the second user 105, based on the gaze area of the first user and the gaze area of the second user 105. If the gaze image of the first user is to be presented, the process proceeds to step S13070; if not, the process proceeds to step S13090.

[0081] In step S13070, gaze image display position and orientation calculation unit 3340 calculates, as the gaze image display position and orientation, a plane perpendicular to the gaze direction of second user 105. Then, the process proceeds to step S13080.

[0082] In step S13080, virtual image generation unit 3350 generates a virtual image in which the gaze image of first user 101 is arranged at the gaze image display position and orientation, and then the process proceeds to step S13090.

[0083] In step S13090, the virtual image generation unit 3350 generates a virtual image in which the virtual object 107 is placed at a predetermined position and orientation in the MR space, and then the process proceeds to step S13100.

[0084] In step S13100, the image synthesis unit 3160 generates a mixed reality image by superimposing the virtual image generated by the virtual image generation unit 3150 on the real image. Then, the process proceeds to step S13110.

[0085] In step S13110, the generated mixed reality image is output to the display unit 3220 of the HMD 3200.

[0086] Next, the process of determining whether to display the line-of-sight image of the first user to the second user in step S13060 will be described.

[0087] FIG. 9 is an example of a flowchart showing the processing performed by the gaze image display determination unit 3330 that determines whether or not to display the gaze image 701 of the first user to the second user.

[0088] First, in step S9100, line-of-sight image display determination unit 3330 acquires the gaze point area of first user 101 from receiving unit 3310. Then, the process proceeds to step S9200.

[0089] In step S9200, the line-of-sight image display determination unit 3330 acquires the gaze point region of the second user 105 from the gaze point region calculation unit 3320. Then, the process proceeds to step S9300.

[0090] In step S9300, line-of-sight image display determination unit 3330 calculates the volume of the common area where two spheres, the fixation point area of first user 101 and the fixation point area of second user 105, overlap. Then, the process proceeds to step S9400.

[0091] In step S9400, if the volume of the common area is greater than a predetermined threshold, the line-of-sight image display determination unit 3330 proceeds to step S9500. If the volume of the common area is smaller than a predetermined threshold, the line-of-sight image display determination unit 3330 proceeds to step S9600.

[0092] In step S9500, the gaze image display determination unit 3330 turns on a gaze image display determination flag for presenting the gaze image of the first user 101 to the second user 105. Then, the processing of the gaze image display determination unit 3330 ends, and the processing is transferred to the gaze image display position and orientation calculation unit 3340.

[0093] In step S9600, the gaze image display determination unit 3330 turns off the gaze image display determination flag, and then ends the processing of the gaze image display determination unit 3330, and transfers the processing to the gaze image display position and orientation calculation unit 3340.

[0094] FIG. 10 is an example of a flowchart showing the processing performed by the gaze image display position and orientation calculation unit 3340 for calculating the position and orientation of the gaze image 701 of the first user 101 to be presented to the second user 105.

[0095] In step S10100, the gaze image display position and orientation calculation unit 3340 acquires the gaze start point and gaze direction of the second user 105 from the gaze point area calculation unit 3320. Then, the process proceeds to step S10200.

[0096] In step S10200, the gaze image display position and orientation calculation unit 3340 acquires gaze image display determination flag information from the gaze image display determination unit 3330. Then, the process proceeds to step S10300.

[0097] In step S10300, the gaze image display position and orientation calculation unit 3340 determines whether the gaze image display determination flag information is ON. If the gaze image display determination flag information is ON, the process proceeds to step S10400. If the gaze image display determination flag information is OFF, the process of the gaze image display position and orientation calculation unit 3340 ends, and the process proceeds to the virtual image generation unit 3350.

[0098] In step S10400, the gaze image display position and orientation calculation unit 3340 calculates a plane perpendicular to the gaze direction of the second user 105 as the gaze image display position and orientation. At this time, the gaze image display position may be adjusted to a position that does not obscure the display of the virtual object 107, based on the position and orientation of the virtual object 107 visible from the position of the second user 105. A position that does not obscure the display of the virtual object 107 is, in other words, a position that does not overlap with the display position of the virtual object 107. Then, the process proceeds to step S10500.

[0099] In step S10500, the gaze image display position and orientation calculation unit 3340 inputs the calculated gaze image display position and orientation to the virtual image generation unit 3350. Then, the processing of the gaze image display position and orientation calculation unit 3340 ends, and the processing is transferred to the virtual image generation unit 3350.

[0100] As a result, the composite image displayed on display unit 3220 of second user 105 can include line-of-sight image 701 of first user.

[0101] According to this embodiment, a viewpoint image of a user who has a real object 103 can be displayed as a two-dimensional image in the virtual space of a user who does not have the real object 103. This enables suitable communication in the MR experience.

[0102] (Variation 1) In the first embodiment, an example has been described in which the gaze image display determination unit 3330 determines whether or not to display the gaze image 701 based on the gaze point area calculated from the gaze positions and orientations of the first user 101 and the second user 105, but the present invention is not limited to this. The user may be able to set the determination of gaze image display.

[0103] That is, the line-of-sight image display determination flag may be operated by an operation input from the user.

[0104] This can be achieved by changing the gaze image display determination flag through an operational input from the user, which may be input by one or more of the following means: button input from a button-type device such as a controller device or a keyboard, gesture input, voice input, etc.

[0105] FIG. 11 is an example of a flowchart for controlling the processing performed by the gaze image display determination unit 3330, which determines whether or not to display the gaze image 701 of the first user 101 to the second user 105, by operation input from the user.

[0106] First, in step S11100, if the operation input by second user 105 is an operation to turn on the gaze image display determination flag, the gaze image display determination unit 3330 proceeds to step S11300. If the operation input by second user 105 is an operation to turn off the gaze image display determination flag, the gaze image display determination unit 3330 proceeds to step S11400.

[0107] In step S11300, the gaze image display determination unit 3330 turns on a gaze image display determination flag for presenting the gaze image of the first user 101 to the second user 105. Then, the processing of the gaze image display determination unit 3330 ends, and the processing is transferred to the gaze image display position and orientation calculation unit 3340.

[0108] In step S11400, the gaze image display determination unit 3330 turns OFF the gaze image display determination flag for presenting the gaze image of the first user 101 to the second user 105. Then, the processing of the gaze image display determination unit 3330 ends, and the processing is transferred to the gaze image display position and orientation calculation unit 3340. The subsequent processing is the same as in the first embodiment, and therefore will be omitted.

[0109] This makes it possible to determine whether or not the composite image displayed on display unit 3220 of second user 105 includes line-of-sight image 701 of first user, based on an operational input from the user.

[0110] (Variation 2) In the first embodiment, the capacity and size of the image may be set for the first user's line of sight image 701 to be transmitted to the second user 105.

[0111] That is, the data amount of the first user's gaze image 701 may be changed by an operational input from the user. This can be achieved by changing the gaze image display determination flag by an operational input from the user. The operational input from the user may be input by one or more of button input from a button-type device such as a controller device or a keyboard, gesture input, voice input, etc.

[0112] Furthermore, the gaze image 701 may be trimmed by a user's operational input. This can be achieved by allowing the user to set a trimming area in the gaze image by a user's operational input. The gaze image may also be controlled according to the network bandwidth. For example, if the network bandwidth is narrow, the image quality of the gaze image 701 may be intentionally reduced to reduce the image size.

[0113] The user's operation input may be input by one or more of the following means: button input from a controller device, a button-type device such as a keyboard, gesture input, and voice input.

[0114] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0115] (Other embodiments) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0116] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0117] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0118] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0119] [Configuration 1] An information processing device that controls a display on a display device worn by a user, The information processing device includes: an acquisition unit that acquires a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; a generating unit that generates a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; An information processing device comprising:

[0120] [Configuration 2] The generation unit generating a composite image including at least the virtual object and a gaze image of the other user when a line of sight of the user with respect to the virtual object displayed on the display device and a line of sight of the other user with respect to the real object satisfy a predetermined criterion; 2. The information processing device according to configuration 1.

[0121] [Configuration 3] The generation unit When a gaze point area of the user with respect to the virtual object and a gaze point area of the other user with respect to the real object satisfy a predetermined criterion, a composite image including at least the virtual object and a gaze image of the other user is generated. 3. The information processing device according to configuration 2.

[0122] [Configuration 4] the user's gaze area is an area based on the user's line of sight and a contact point of the virtual object, The gaze area of the other user is an area based on the line of sight of the other user and a contact point of the virtual object. 4. The information processing device according to any one of configurations 1 to 3.

[0123] [Configuration 5] the user's gaze area is a spherical area centered on a point of contact between the user's line of sight and the virtual object, The gaze area of the other user is a spherical area centered on the line of sight of the other user and the contact point of the virtual object. 5. The information processing device according to configuration 4.

[0124] [Configuration 6] The generation unit When a volume of a common area between the gaze point area of the user and the gaze point area of the other user is equal to or larger than a threshold, a composite image including at least the virtual object and a gaze image of the other user is generated. 6. The information processing device according to configuration 5.

[0125] [Configuration 7] the user's gaze point area is an area based on gaze position and orientation information of the user and position information of the real object or the virtual object, The gaze point area of the other user is an area based on gaze position and orientation information of the other user and position information of the virtual object. 7. The information processing device according to any one of configurations 4 to 6.

[0126] [Configuration 8] The information processing device includes: a transmission unit that transmits the composite image generated by the generation unit to the display device; 8. The information processing device according to any one of configurations 1 to 7, further comprising:

[0127] [Configuration 9] the virtual object is generated based on three-dimensional information acquired by a camera placed in a space where the other user exists photographing the real object; 9. The information processing device according to any one of configurations 1 to 8.

[0128] [Configuration 10] the gaze image of the other user included in the composite image is disposed at a position that does not overlap with the display position of the virtual object; 10. The information processing device according to any one of configurations 1 to 9.

[0129] [method] An information processing method for controlling a display of a display device worn by a user, comprising: acquiring a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; generating a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; 2. A computer-implemented information processing method, comprising:

[0130] [program] Computer, An information processing device that controls a display on a display device worn by a user, an acquisition unit that acquires a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; a generating unit that generates a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; A program for causing the device to function as an information processing device.

Claims

1. An information processing device that controls a display on a display device worn by a user, The information processing device includes: an acquisition unit that acquires a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; a generating unit that generates a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; An information processing device comprising:

2. The generation unit generating a composite image including at least the virtual object and a gaze image of the other user when a line of sight of the user with respect to the virtual object displayed on the display device and a line of sight of the other user with respect to the real object satisfy a predetermined criterion; The information processing device according to claim 1 .

3. The generation unit When a gaze point area of the user with respect to the virtual object and a gaze point area of the other user with respect to the real object satisfy a predetermined criterion, a composite image including at least the virtual object and a gaze image of the other user is generated. The information processing device according to claim 2 .

4. the user's gaze area is an area based on the user's line of sight and a contact point of the virtual object, The gaze area of the other user is an area based on the line of sight of the other user and a contact point of the virtual object. The information processing device according to claim 1 .

5. the user's gaze area is a spherical area centered on a point of contact between the user's line of sight and the virtual object, The gaze area of the other user is a spherical area centered on the line of sight of the other user and the contact point of the virtual object. The information processing device according to claim 4 .

6. The generation unit When a volume of a common area between the gaze point area of the user and the gaze point area of the other user is equal to or larger than a threshold, a composite image including at least the virtual object and a gaze image of the other user is generated. The information processing device according to claim 5 .

7. the user's gaze point area is an area based on gaze position and orientation information of the user and position information of the real object or the virtual object, The gaze point area of the other user is an area based on gaze position and orientation information of the other user and position information of the virtual object. The information processing device according to claim 4 .

8. The information processing device includes: a transmission unit that transmits the composite image generated by the generation unit to the display device; The information processing device according to claim 1 , further comprising:

9. the virtual object is generated based on three-dimensional information acquired by a camera placed in a space where the other user exists photographing the real object; The information processing device according to claim 1 .

10. the gaze image of the other user included in the composite image is disposed at a position that does not overlap with the display position of the virtual object; The information processing device according to any one of claims 1 to 9.

11. An information processing method for controlling a display of a display device worn by a user, comprising: acquiring a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; generating a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; 2. A computer-implemented information processing method, comprising:

12. Computer, An information processing device that controls a display on a display device worn by a user, an acquisition unit that acquires a line-of-sight image of another user wearing another display device and information about a virtual object corresponding to a real object placed in a space where the other user is present; a generating unit that generates a composite image including at least a gaze image of the other user when a gaze of the user with respect to the virtual object displayed on the display device and a gaze of the other user with respect to the real object satisfy a predetermined criterion; A program for causing the device to function as an information processing device.

Images