Information processing device, information processing method, and program

Abstract

To provide a user with a more suitable MR or VR experience by controlling a masking CG rendering method.SOLUTION: An information processing device, which controls a display of a display device worn by a certain user, includes: acquisition means that acquires virtual object information, which is information on a virtual object to be displayed on the display device; and control means that controls, on the basis of the virtual object information acquired by the acquisition means, a display of an image of the virtual object. The control means controls, when the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, the display device to display an image different from the transparent mask image.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Mixed reality (MR) technology is known as a technology that blends the real world and the virtual world in real time. This technology seamlessly blends real space with a computer-generated MR space. MR uses a head-mounted display (HMD) to allow users to experience the MR space from a perspective closer to reality.

[0003] Conventional general mixed reality presentation methods simply superimpose a CG (Computer Graphics) image on a real video image, without taking into consideration the depth relationship between a real object and a CG object. Therefore, a conventional method detects the overlapping area between the real object and the CG object, and masks the CG in the detected area to display the real object so that it is visible (Patent Document 1).

[0004] As a CG masking method, the image area where a real object is photographed is set in the CG stencil buffer or the depth buffer (Z buffer) is set, thereby obtaining the depth of the transparent CG and preventing the CG from being drawn in that area. In this way, the front-to-back relationship with the virtual object can be obtained.

[0005] There is also known a technology that combines real-time sharing of 3D models of people and objects with an XR device, allowing remote users to communicate in an MR space where the same virtual objects are shared in different real spaces.

[0006] One possible use of this technology is to check the work in an MR space, for example, by holding a CG image of a prototype factory assembly equipment and the actual object that will be used after the equipment is completed.

[0007] Even in this case, the user holding the real object can obtain a transparent CG mask image that simulates the front-to-back depth relationship between the real object and the virtual object. On the other hand, the masking CG is shared with the remote user who does not have the real object, so the remote user's real space is visible through the masking CG position where the real object should be drawn. This can lead to situations where the intended message is not conveyed when communicating with a real object held in one's hand.

[0008] In addition, XR devices can operate in either VR or MR mode. In the case of data with masking CG, when operating in VR mode, the background image of the virtual space can be seen through the masking CG position.

[0009] In addition, in the case of data with masking CG, depending on the location and time when the user uses the data, there may or may not be a real object that corresponds to the masking CG. If the masking CG is used as is when there is no real object, a situation will occur in which real space is visible through the masking CG position where the real object should be drawn. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] JP 2003-296759 A Summary of the Invention [Problem to be solved by the invention]

[0011] When a first user applies masking CG to a real object and shares it with a second user in a remote location, the second user cannot see the real object, but can see the real object through the masking CG, which disrupts the user experience.

[0012] Additionally, depending on the operating mode of the device displaying the XR, the image beyond the masking CG can be seen, disrupting the user experience.

[0013] In addition, when data with masking CG is used in a different location or at a different time, if there is no real object that corresponds to the masking CG, the image beyond the masking CG will be visible, disrupting the user experience.

[0014] Therefore, an object of the present invention is to provide a technology that provides a user with a more suitable MR experience by controlling the rendering method of masking CG. [Means for solving the problem]

[0015] In order to achieve the above object, according to one aspect of the present invention, an information processing device that controls display of a display device includes: an acquisition means that acquires virtual object information, which is information about virtual objects to be displayed on the display device; and a control means that controls display of an image of the virtual object based on the virtual object information acquired by the acquisition means, wherein when the virtual object information includes a setting for displaying a real object in front of the virtual object, the control means determines a setting for displaying the real object in front of the virtual object or a setting for displaying the virtual object in front based on an environment in which the virtual object information was acquired, and controls the display device to display a display image created based on the determined setting. [Effects of the Invention]

[0016] According to the present invention, by controlling the method of rendering masking CG, it is possible to provide the user with a more suitable MR experience. [Brief explanation of the drawings]

[0017] [Figure 1] This is a diagram showing the MR space when a user superimposes masking CG on a real object. [Figure 2]This is a diagram showing the positional relationships when multiple users share an MR space. [Figure 3] 1 is a block diagram showing an example of the functional configuration of a system according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the configuration of CG model data for one virtual object. [Figure 5] 10 is a flowchart of a masking determination process according to the first embodiment. [Figure 6] FIG. 2 is a diagram illustrating the positional relationship between a first user and a second user during implementation of the first embodiment. [Figure 7] FIG. 10 is a diagram showing the positional relationship between a first user and a second user according to a modified example of the first embodiment. [Figure 8] 10 is a flowchart of a masking determination process according to a modified example of the first embodiment. [Figure 9] FIG. 10 is a diagram showing the positional relationship between a first user and a second user according to the second embodiment. [Figure 10] FIG. 10 is a block diagram showing an example of the functional configuration of a system according to a second embodiment. [Figure 11] FIG. 10 is a block diagram showing an example of the functional configuration of a system according to a third embodiment. [Figure 12] 10 is a flowchart showing a processing procedure of an information processing device according to a third embodiment. [Figure 13] 10A to 10C are diagrams illustrating an example of control of masking CG in the information processing device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the embodiments described below are examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. Furthermore, it is also possible to combine the various embodiments as appropriate.

[0019] Figure 1 shows a virtual space in which the overlapping area between a real object (also called a real object) and a CG object (also called a virtual object) is detected, and the CG in the detected area is masked so that the real object is visible. Note that CG masking is an example of a setting in which a real object is displayed in front of a virtual object, and any method can be used as long as it can realize control to display a real object in front of each virtual object, or behind the virtual object.

[0020] The superimposition technology for superimposing masking CG on a real object includes a real object detection means and a CG masking means. It is desirable for the real object detection means to accurately display the overlap between a real object 102 held by a user 101 in his / her hand and a virtual object 103, as shown in FIG. 1(a). To achieve this, it is necessary to detect the display area of ​​the real object 102 held by the user 101 in his / her hand in the real image. Therefore, it is sufficient to determine the position and orientation of the real object in the MR space and detect the area in which the real object is captured.

[0021] Fig. 1(b) shows the positional relationship between the user 101, the real object 102, and the virtual object 103 when Fig. 1(a) is viewed from above. Fig. 1(c) shows an image from the user's viewpoint when the real object is masked in the positional relationship shown in Fig. 1(b).

[0022] As a position and orientation detection method, one method for acquiring the actual viewpoint position and orientation is to photograph markers placed in space and estimate the actual viewpoint position and orientation from the arrangement of feature points of the markers in the image. Another method is to use Simultaneous Localization and Mapping (SLAM), which uses natural feature points in the actual image to simultaneously estimate self-position and create an environmental map. External measurement devices such as motion capture may also be used.

[0023] The CG masking means, for example, sets an image area where a real object 102 is determined to be captured by a real object detection means in a CG stencil buffer or a depth buffer (Z-buffer). This allows the depth of the transparent CG 104 to be obtained and the CG not to be drawn in that area, thereby obtaining the front-to-back relationship with the virtual object 103. As a result, as shown in FIG. 1(c), the real object 102 is visible in the area of ​​the transparent CG 104, and a composite image can be obtained in which the front-to-back depth relationship of the virtual object 103 is reproduced in a pseudo manner. The transparent CG is an example of a transparent mask image.

[0024] FIG. 2 is a schematic diagram showing a case where a user 101 and a remote user share the same virtual object 103 in different real spaces and communicate in an MR space.

[0025] FIG. 2(a) shows a case where a remote user 201 who does not have a real object 102 and a user 101 who has the real object 102 share an MR space.

[0026] 2(b) shows the positional relationship between users 101 and 201 when viewing FIG. 2(a) from above, a masking CG 104 drawn to mask the real object, and the virtual object 103. In this case, user 101 can obtain a transparent CG mask image that artificially reproduces the front-to-back depth relationship between the real object and the virtual object 103, as shown in FIG. 1(c).

[0027] On the other hand, a user 201 who does not hold the physical object 102 can see the physical object through the masking CG 104, as shown in Figure 2(c). This can lead to situations where the intended message is not conveyed when communicating about the physical object 102 being held in the hand. Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[0028] [First embodiment] The mixed reality presentation system according to the present embodiment, which will be described below, is intended to present to a user a well-known mixed reality space (hereinafter referred to as MR space) that combines a real space and a virtual space.

[0029] In the system according to this embodiment, when an image in which real space and MR space are combined is presented to a user, the image in real space is basically drawn first, followed by the image in MR space, and as in the conventional example, the area of ​​the virtual object that overlaps with the real object is masked. However, in this embodiment, when a masked virtual object is shared, the receiving user can set whether or not to mask it for each user, which is different from the conventional example.

[0030] The mixed reality presentation system according to this embodiment will be described below.

[0031] 3 is a diagram showing the basic configuration of a mixed reality presentation system according to this embodiment. This system is composed of, for example, a computer 3100, a computer 3200, an HMD 3000, an HMD position and orientation sensor 3310, a real object position and orientation sensor 3320, and a sensor controller 3300. Each of these components will be described below. First, the HMD 3000 will be described.

[0032] The HMD3000 is an example of a display device, and is a device worn on the head of a user experiencing an MR space, and is worn so that the display unit 3020 (including a display screen) provided in the HMD3000 is positioned in front of the user's eyes.

[0033] An imaging unit 3010 is fixed to the HMD 3000 so that it can capture an image in the line of sight of the user when the user wears the HMD 3000 on their head. Therefore, the imaging unit 3010 can capture an image of the real space that is visible according to the position and orientation of the HMD 3000.

[0034] An HMD position and orientation sensor 3310 may also be fixed to the HMD 3000. The HMD position and orientation sensor 3310 may be configured with a magnetic sensor, an ultrasonic sensor, or the like, and may measure its own position and orientation and output the measurement result as a signal to the sensor controller 3300. In this embodiment, the HMD position and orientation sensor 3310 will be described as measuring its own position and orientation in a world coordinate system (a space with one point in real space as the origin and three axes orthogonal to each other at this origin: the x-axis, y-axis, and z-axis).

[0035] The results of measurement by the HMD position and orientation sensor 3310 are output as signals to the sensor controller 3300, and the sensor controller 3300 outputs to the computer 3100 a numerical value according to the strength of the received signal.

[0036] The physical object position and orientation sensor 3320 is used to change the position and orientation of a virtual object in the MR space held by the user experiencing the MR space, and may be the same sensor as the HMD position and orientation sensor 3310.

[0037] That is, the physical object position and orientation sensor 3320 measures its own position and orientation in the world coordinate system, and outputs the measurement result as a signal to the sensor controller 3300. Similarly, the sensor controller 3300 outputs the measurement result as numerical data to the computer 3100 in accordance with the strength of the received signal. Next, the computer 3100 will be described.

[0038] The computers 3100 and 3200 are examples of information processing devices, and are generally configured as a PC (personal computer) or a WS (workstation). They may also be configured as dedicated hardware, or as a mobile terminal such as a smartphone or tablet. The computers 3100 and 3200 may be configured separately from the HMD 3000, or may be installed in the HMD 3000.

[0039] The operation unit 3400 can input various instructions to the computer. The operation unit 3400 may be configured by one or more of a controller device, button input from a button-type device such as a keyboard, gesture input, voice input, and the like.

[0040] The imaging unit 3010 captures moving images of the real space that are visible according to the position and orientation of the HMD 3000, and the images of each frame that make up this moving image (real space images) are sequentially input to the computer 3100. Therefore, the real space image acquisition unit 3110 acquires the real space images from the imaging unit 3010.

[0041] To obtain the position and orientation of the image capture unit 3010 in the world coordinate system, the position and orientation detection unit 3120 acquires the gaze position and orientation measured by the HMD position and orientation sensor 3310 and converted into numerical data by the sensor controller 3300. The position and orientation detection unit 3120 may then obtain the position and orientation of the image capture unit 3010 in the world coordinate system from the results obtained by the HMD position and orientation sensor 3310. At this time, the position and orientation detection unit 3120 may perform well-known conversion processing using the position and orientation relationship between the image capture unit 3010 and the HMD position and orientation sensor 3310. It is assumed that the position and orientation relationship between the image capture unit 3010 and the HMD position and orientation sensor 3310 has been measured in advance.

[0042] Also, the results measured by the physical object position and orientation sensor 3320 and converted into numerical data by the sensor controller 3300 are acquired.

[0043] The physical object detection unit 3140 performs processing to detect an area occupied by a specific physical object, for example, the physical object 102 in FIG. 1, from the physical image acquired by the imaging unit 3010.

[0044] For example, the area occupied by the physical object 102 can be obtained by detecting a group of pixels having the area occupied by the physical object from specific position and orientation information in the physical space image. The physical object detection unit 3140 acquires an image based on the detected group of pixels (an image of the area in the physical space image where the specific physical object is located).

[0045] Furthermore, a specific physical object may be detected by detecting a group of pixels that exhibits a specific color or a group of pixels that has a specific shape in the physical image.

[0046] In addition, a new physical object may be detected from the physical image acquired by the image capture unit 3010 and registered in the virtual object information. Also, detailed information about the physical object, such as color information, glossiness, haze, image clarity, and diffusion, may be detected and registered in the virtual object information, or information may be added to or updated in the registered virtual object information. Furthermore, the position and orientation of the physical object may be calculated from the information about the detected physical object.

[0047] The masking target CG designation unit 3130 references virtual object model data for rendering virtual objects that constitute the MR space and selects data of the virtual object to be masked. Virtual object model data is provided for each virtual object. For example, the virtual object model data may be stored in a storage unit (not shown) of the computers 3100 and 3200, or may be stored in another database server. For example, the masking target CG designation unit 3130 may identify a virtual object corresponding to a real object using a marker in an image, or may identify a virtual object corresponding to the real object based on the shape or color information of the real object. Alternatively, the user may manually designate the virtual object.

[0048] Next, the masking process in the masking target CG designation section will be described.

[0049] 4 is a diagram showing an example of the configuration of CG model data for one virtual object. Virtual object information (hereinafter also referred to as virtual object configuration information 400) of a virtual object (CG object) rendered in this embodiment includes position and orientation information 403 indicating the position and orientation of the virtual object (position (x, y, z) and orientation (roll, pitch, yaw)). Furthermore, the virtual object information includes model information 404, which is visual information such as the color and shape of the virtual object, as well as a masking target flag 402 and a masking control flag 401, which indicate whether or not this virtual object is a target of masking.

[0050] This masking target flag 402 can be expressed by one bit indicating ON / OFF. For example, when the value of the masking target flag 402 is "1" (ON), it indicates that "it is a target for masking." When the value of the masking target flag 402 is "0" (OFF), it indicates that "it is not a target for masking." In other words, when the value of the masking target flag 402 included in virtual object information is "1," it indicates that the virtual object information includes a masking flag that superimposes a transparent mask image on a physical object. Note that the masking target flag 402 is an example of masking flag information.

[0051] The masking target flag 402 may be set in advance to ON / OFF (yes / no) by the user of the computer 3100 using the operation unit 3400. Alternatively, it may be determined to dynamically change the masking target flag to always maintain it as OFF, such as for an operation panel (virtual object as a GUI) that must always be presented to the user.

[0052] In this case, the actual flag value setting process is performed by the masking target CG designation unit 3130.

[0053] The masking control flag 401, like the masking target flag 402, can be expressed by one bit indicating ON / OFF. For example, when the value of the masking control flag 401 is "1" (ON), it indicates that "the received masking target flag has been changed from ON to OFF." On the other hand, when the value of the masking control flag 401 is "0" (OFF), it indicates that "the received masking target flag has not changed." The masking control flag 401 is an example of display control information.

[0054] This masking control flag 401 is a flag used in the processing of the computer 3200, which will be described later, and is not used in the processing of the computer 3100.

[0055] The masking target CG designation unit 3130 refers to the value of the masking target flag 402 in the CG model data of each virtual object, and designates only the CG model data (model image) for which the masking target flag 402 is OFF as the drawing target. Then, the virtual image generation unit 3150 generates (renders) an image of the virtual object that has been designated as the drawing target, in accordance with the model information 404 in the CG model data.

[0056] During this rendering, an image of the virtual object seen from the image capturing unit 3010 is generated using gaze position and orientation information and position and orientation information in the CG model data.

[0057] Furthermore, when rendering a virtual object whose position and orientation change dynamically (for example, a virtual object placed at the position of the physical object position and orientation sensor 3320), the position and orientation information of this virtual object is updated as appropriate before the rendering process. Then, an image of the virtual object as seen from the image capturing unit 3010 is generated using the updated position and orientation information and the gaze position and orientation information.

[0058] The rendering technology for a virtual object image seen from a predetermined viewpoint position and orientation is well known, and therefore a description thereof will be omitted here.

[0059] The image synthesis unit 3160 superimposes the virtual image generated by the virtual image generation unit 3150 on the real space image acquired by the real image acquisition unit 3110. As a result, the real image is drawn with virtual objects whose masking target flags 402 are ON masked in the masked portions of the synthesized image. In the image where the masking target flags 402 are OFF, unmasked virtual objects are drawn.

[0060] Next, a description will be given of the computer 3200. The same components as those in the computer 3100 are given the same reference numerals, and a description thereof will be omitted.

[0061] The transmitting unit 3170 of the computer 3100 inputs the virtual object data to the masking target determining unit 3220 of the receiving unit 3210 of the computer 3200. The receiving unit 3210 is an example of an acquiring means. Here, the masking determination process in the masking target determining unit 3220 will be described.

[0062] 5 shows the masking determination process for one virtual object. First, in step S5100, the configuration information 400 of the virtual object is acquired.

[0063] In step S5200, it is determined whether the masking target flag 402, which indicates whether the virtual object is a target for masking, is ON. If the target flag 402 is ON, the process proceeds to step S5300. If the masking target flag 402 is OFF, the configuration information 400 of the virtual object is input to the image generation method control unit 3230.

[0064] In step S5300, the masking target determination unit 3220 turns off the masking target flag 402, and the process proceeds to step S5400.

[0065] In step S 5400 , the masking target determination unit 3220 turns on the masking control flag 401 , and inputs the virtual object configuration information 400 to the image generation method control unit 3230 .

[0066] The image generation method control unit 3230 references the value of the masking control flag 401 and the value of the masking target flag 402 in the CG model data of each virtual object, and generates an image.

[0067] That is, when the value of the masking control flag 401 is ON, this means that the masking target flag 402 of the received virtual object has been changed from ON to OFF, and this CG model data is the rendering target. As a rendering method, for example, a single color or multiple colors preset by the user of the computer 3200 may be specified using the operation unit 3400. For example, as a rendering method, a color specified by the user may be applied to a transparent mask image. Alternatively, a virtual object similar to or identical to the virtual object indicated by the received virtual object information may be prepared in advance, and when the value of the masking control flag 401 is ON, this virtual object may be the rendering target. Furthermore, information such as color information, glossiness, haze, clarity, and diffusion of the real object may be acquired from the real object detection unit 3140 and stored and used for rendering. That is, rendering that looks similar to the real object in virtual space may be performed. For example, a texture image may be converted from the real object and applied to the shape of the virtual object.

[0068] For CG model data in which the masking target flag 402 is OFF, the image generation method control unit 3230 may generate an image of the virtual object to be rendered in accordance with the model information 404 in the CG model data. For CG model data in which the masking control flag 401 has an ON value, the image generation method control unit 3230 generates an image of the virtual object in accordance with the rendering method and the model information 404 in the CG model data.

[0069] During this rendering, the image generation method control unit 3230 generates an image of the virtual object seen from the image capturing unit 3010 using the gaze position and orientation information and the position and orientation information in the CG model data.

[0070] The mask control image synthesis unit 3240 superimposes the virtual image generated by the image generation method control unit 3230 on the real space image acquired by the real image acquisition unit 3110. Specifically, the mask control image synthesis unit 3240 is an example of an acquisition means, and when the masking target flag 402 is OFF (i.e., the masking control flag 401 is ON), it displays an image different from the transparent CG 104 on the HMD 3000. As a result, an image of a virtual object whose masking target flag 402 is ON is not drawn in the portion of the real space image masked by the computer 3200, but a virtual object is drawn in an image whose masking target flag 402 is OFF. A schematic diagram of a case where this embodiment is implemented is shown in FIG. 6.

[0071] FIG. 6(a) shows the MR space of a remote user 201 who does not have a real object when this embodiment is implemented.

[0072] A virtual object 601 that is to be masked from a physical object 102 owned by the user 101 is shared with the user 201, and the virtual object 601 is visualized and displayed as a visible image by the masking object determination unit 3220 and the image generation method control unit 3230.

[0073] 6(b) shows the viewpoint image of the user 201. The user 201 can obtain a CG image that artificially reproduces the front-to-back depth relationship of the virtual object 103 at the position of the real object 102 that the user 101 is holding.

[0074] As explained above, in this embodiment, when a masking CG is received, the method of rendering the masking CG is controlled, thereby making it possible to provide a suitable MR experience.

[0075] [Modification 1 of the First Embodiment] In the above-described embodiment, when a virtual object is displayed using a controlled masking CG rendering method and used by multiple users, a user may move the position of the virtual object using a controlled masking CG rendering method. In this case, a user who displays a virtual object using a controlled masking CG rendering method can obtain a suitable display. On the other hand, for a user who displays a real object using a masking CG, the location of the masking CG may be moved, which may interfere with the display of the real object. Therefore, a virtual object displayed using a controlled masking CG rendering method may be controlled so that its position and orientation cannot be changed. Furthermore, when a user operates to move a virtual object displayed using a controlled masking CG rendering method, the virtual object may not be moved, but a duplicate of the same virtual object may be displayed at the location specified by the user's operation. Then, by setting the duplicated virtual object to be shared by multiple users, it becomes possible to move the virtual object displayed using a controlled masking CG rendering method.

[0076] [Modification 2 of the First Embodiment] In the first embodiment, it was explained that when masking CG is received, the method of drawing the masking CG is controlled, but depending on the situation of the user who received it, it may not necessarily be necessary to update the value of the masking control flag 401.

[0077] FIG. 7 is a diagram showing a case where a user 101 who has a real object 102 and a user 701 who does not have the real object 102 share an MR space in the same location.

[0078] FIG. 7(b) shows the positional relationship between the user 101 and user 701, the masking CG 104 drawn to mask the real object, and the virtual object 103 when viewed from above in FIG. 7(a). In this case, the user 101 can obtain a masking image that simulates the front-to-back depth relationship between the real object 102 and the virtual object 103, as shown in FIG. 7(c). On the other hand, the masking CG 104 is shared with the user 701, who does not hold the real object 102, but at this time, the user 701 can see the real object 102 held by the user 101. Therefore, it is not necessary to control masking using the masking control flag 401 and perform image generation control; instead, a masking image may be drawn in the same way as for the user 101, as shown in FIG. 7(c). Here, the masking determination process in the masking target determination unit 3220 in this modification will be described.

[0079] FIG. 8 is a diagram showing the masking determination process for one virtual object in the modified example.

[0080] The same processes as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0081] In step S5100, the configuration information 400 of the virtual object is acquired.

[0082] In step S8200, it is determined whether the user who transmitted the virtual object configuration information 400 and the user who received it are in different locations. If they are connected in different locations, the process proceeds to step S5200. If they are connected in the same location, the virtual object configuration information 400 is input to the image generation method control unit 3230.

[0083] Determining whether the user who transmitted the virtual object configuration information 400 and the user who received it are in the same location may be determined, for example, by whether the users acquire their positions and orientations from the same sensor controller. Determining whether the user who transmitted the virtual object configuration information 400 and the user who received it are in the same location may also be determined by whether the positions and orientations are calculated based on the same world coordinate system. Determining whether the position and orientation information of the real object to be masked has been acquired in advance from a sensor controller may also be determined based on the user's viewpoint image, such as whether the HMD of the user who transmitted the virtual object configuration information 400 is included in the user's viewpoint image, or whether the real object to be masked is included in the user's viewpoint image. The user may also set users with whom the transparent CG will be shared in advance using the operation unit 3400, or preset location information or GPS information may be used.

[0084] As described above, even when there is no need to control masking for connecting at the same location, MR images can be generated flexibly according to the situation in the MR space.

[0085] [Second embodiment] In the above embodiment, the configuration information of all virtual objects, including the virtual objects designated as masking targets by the masking target CG designation unit, is shared between users, but there are cases where it is not necessary to share all virtual objects. In such cases, the virtual objects to be shared may be switched according to the user's wishes.

[0086] 9 shows a case where a user 101 who has a real object 102 and a remote user 201 who does not have the real object 102 share an MR space in the same location. A real stationary object 901 (e.g., a shelf) may exist in the environment of the user 101 who has the real object 102. In this case, in consideration of the safety of the user 101 during the MR space experience, the real stationary object 901 is designated as a masking target so that it is not obscured by the virtual object 103.

[0087] Figure 9(b) shows the positional relationship between user 101 and user 201 when viewed from above in Figure 9(a), the masking CG 104 drawn to mask the real object, and the virtual object and real stationary object 901 in the user 101 environment.

[0088] In this virtual experience, the real stationary object 901 is not the subject of the virtual experience, and only needs to be visible to the user 101, and the virtual object superimposed on the real stationary object 901 does not need to be shared with the user 201.

[0089] In this case, the user 101 can obtain an image as shown in Fig. 9(c) in which the real object 102 held in the hand and the real still object 901 are superimposed, which simulates the front-to-back depth relationship between the real object 102 and the virtual object 103. On the other hand, the user 201 who does not have the real object 102 does not need to share the virtual object that masks the real still object 901, and can simply obtain an image as shown in Fig. 9(d).

[0090] Here, a second current mixed reality presentation system that realizes the above will be described.

[0091] FIG. 10 is a diagram showing the basic configuration of a mixed reality presentation system according to the second embodiment.

[0092] The same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0093] The shared virtual object determination unit 10010 acquires virtual object data for each virtual object from the masking target CG designation unit 3130. From the virtual object data, a transmission unit 3170 determines which virtual objects are to be shared with other users.

[0094] The virtual objects to be shared may be determined, for example, by having the user set the virtual objects to be shared in advance using the operation unit 3400, or by using the amount of movement of the position and orientation of a physical object per unit time. Alternatively, the determination may be made based on whether the position of the virtual object to be masked and the position of another virtual object are separated by a threshold or more.

[0095] The transmission unit 3170 shares with other users only the configuration information 400 of the virtual object that has been set by the shared virtual object determination unit 10010 to be shared with other users.

[0096] As explained above, by not sharing masking information for virtual objects that do not need to share masking information, synthetic images can be generated flexibly according to the situation in the MR space.

[0097] [Third embodiment] In the above embodiment, the control of the masking target when a virtual space is shared by multiple people has been described, but the present invention is not necessarily limited to sharing by multiple people.

[0098] That is, while a user is experiencing a virtual space alone, the masking ON / OFF of the masking CG designated as the masking target may be dynamically changed. Specifically, the masking ON / OFF may be dynamically changed depending on whether the image displayed to the user is a virtual reality space (VR space) or a mixed reality space (MR space). Furthermore, when using virtual object information containing a masking flag stored in an information processing device, there may or may not be a real object corresponding to the virtual object information containing the masking flag depending on the location and time of the user's use. In such cases, the masking flag may be dynamically changed.

[0099] Fig. 11 is a diagram showing an example of the configuration of a system according to this embodiment. Fig. 11 shows an image processing system for presenting a virtual reality space (VR space) or a mixed reality space (MR space) that combines real space and virtual space to a person experiencing the system (user). The same parts as in Fig. 3 are numbered the same, and their explanations will be omitted.

[0100] A masking target CG control unit 1101 sets a masking control flag, which serves to control whether or not a virtual object whose masking target flag is ON, to ON or OFF.

[0101] The image generation unit 1102 constructs a virtual space in accordance with the setting of the masking control flag set by the masking target CG control unit 1101. The virtual space data includes data related to each virtual object constituting the virtual space and data related to a light source illuminating the virtual space. Then, the position and orientation of the virtual viewpoint corresponding to the position and orientation of the HMD acquired by the position and orientation detection unit 3120 is set. Then, an image of the virtual space seen from the viewpoint (virtual space image) is generated. Then, if the operation of the display device is to display an MR image, the process proceeds to processing by the image synthesis unit 1103. If the operation of the display device is to display an image of the virtual space, the generated image of the virtual space is output to the display unit 3020. Note that the technology for generating an image of the virtual space seen from a viewpoint having a predetermined position and orientation is well known, and therefore a detailed description thereof will be omitted.

[0102] The image synthesis unit 1103 generates an MR image by synthesizing the image of the virtual space generated by the image generation unit 1102 and the image of the real space captured by the imaging unit 3010. Then, the generated MR image is output to the display unit 3020.

[0103] The operation mode designation unit 1104 designates whether the image to be displayed on the display device HMD 3000 is a VR space or an MR space. The method of designating the operation mode is, for example, such that the operation mode designation unit 1104 inquires about the operation modes that can be set by the display device HMD 3000, and designates the operation mode based on information returned from the display device HMD 3000. In the case of a display device that can display both VR space and MR space, the operation mode may be designated according to a user designation.

[0104] FIG. 12 is a flowchart of processing performed by the information processing device 1100 to generate VR video or MR video and output it to the HMD 3000.

[0105] First, in step S1201, the position and orientation detection unit 3120 calculates the position and orientation information of the HMD 3000 and the physical object.

[0106] Next, in step S1202, the physical object detection unit 3140 detects physical objects that exist in the physical space.

[0107] Next, in step S1203, the physical object detection unit 3140 determines whether or not information about a physical object existing in the physical space has been updated. If it is determined that information about the physical object has been updated, the process proceeds to step S1204, and if it is determined that information about the physical object has not been updated, the process proceeds to step S1205.

[0108] Next, in step S1204, the physical object detection unit 3140 registers, adds, or updates information about the physical object in the virtual object information.

[0109] Next, in step S1205, the masking target CG designation unit 3130 sets the masking target flag to ON for the virtual object information to which information about the real object has been registered, added, or updated.

[0110] Next, in step S1206, the operation mode designation unit 1104 determines whether the operation mode is a mode for displaying MR images. If it is determined that the operation mode is a mode for displaying MR images, the process proceeds to step S1207, and if it is determined that the operation mode is not a mode for displaying MR images, i.e., a mode for displaying VR images, the process proceeds to step S1215.

[0111] Next, in step S1207, the masking target CG control unit 1101 sets k=1 as an initial setting. Then, in step S1208, the value of k is compared with the number of virtual objects whose masking target flags are ON. If it is determined that the value of k is less than the number of virtual objects whose masking target flags are ON, the process proceeds to step S1209, and if it is determined that the value of k is equal to or greater than the number of virtual objects whose masking target flags are ON, the process proceeds to step S1213.

[0112] Next, in step S1209, it is determined whether or not a physical object corresponding to the k-th virtual object with the masking target flag set to ON exists. If it is determined that a physical object corresponding to the k-th virtual object with the masking target flag set to ON exists, the process proceeds to step S1210, and if it is determined that a physical object corresponding to the k-th virtual object with the masking target flag set to ON does not exist, the process proceeds to step S1211.

[0113] Next, in step S1210, the masking target CG control unit 1101 sets the masking control flag of the k-th virtual object whose masking target flag is ON to ON.

[0114] Next, in step S1211, the masking target CG control unit 1101 sets the masking control flag of the k-th virtual object whose masking target flag is ON to OFF.

[0115] Next, in step S1212, the masking target CG control unit 1101 increments the value of k by 1. Then, the process returns to step S1208.

[0116] Next, in step S1213, the image generating unit 1102 generates an image of the virtual space using the position and orientation information of the virtual viewpoint calculated in step S1201.

[0117] Next, in step S1214, the image synthesis unit 1103 synthesizes the real image captured by the imaging unit 3010 with the image of the virtual space generated in step S1213.

[0118] Next, in step S1215, the masking target CG control unit 1101 sets the masking control flag of the virtual object whose masking target flag is ON to OFF.

[0119] Next, in step S1216, the image generating unit 1102 generates an image of the virtual space using the position and orientation information of the virtual viewpoint calculated in step S1201.

[0120] Next, if an instruction to end this process is input or the conditions for ending this process are satisfied, this process is terminated. On the other hand, if an instruction to end this process is not input or the conditions for ending this process are not satisfied, the process returns to step S1201 via step S1217.

[0121] 13 is an image showing the image synthesis or image generation result generated by the image synthesis unit 1103 and the image generation unit 1102. It shows that by controlling the CG of the masking target, a suitable MR image or VR image is displayed depending on the operation mode of the display device or the presence or absence of a real object.

[0122] An image 1301 represents a real space and is an image captured by the image capturing unit 3010.

[0123] Reference numeral 1302 denotes the user's hand. Reference numerals 1303 and 1304 denote real objects existing in real space, and CG images to be masked (not shown) are placed in the same locations. Reference numerals 1305, 1306, 1307, and 1308 denote virtual objects. Reference numerals 1309 and 1310 denote virtual objects that control the CG images to be masked, and reference numeral 1311 denotes a virtual object that controls the CG images to be masked with respect to the user's hand.

[0124] 13A is a diagram showing the image synthesis result generated by the image synthesis unit 1103. By placing the masking target CG (not shown) in the same location as the real objects 1303 and 1304, the image synthesis unit 1103 generates an MR image that correctly represents the depth relationship.

[0125] Fig. 13(B) is a diagram showing the result of image generation generated by the image generation unit 1102 when the operation mode is VR image display. The masking target CG is controlled to display virtual objects 1309, 1310, and 1311. By this control, a VR image is generated that correctly expresses the depth relationship of the real objects or virtual objects displayed in Fig. 13(A) using virtual objects, even when the operation mode is VR image display.

[0126] Fig. 13(C) is a diagram showing the image synthesis result generated by the image synthesis unit 1103 using the same data as Fig. 13(A) when a physical object 1303 does not exist from the state of Fig. 13(A). Because the physical object 1303 does not exist, the masking target CG is controlled to display a virtual object 1309. This control makes it possible to use the data in which the masking target CG is set in the same way as when a physical object exists, even when no physical object exists.

[0127] [Variations] In the above embodiment, the display device is described as an HMD, but the display device is not limited to an HMD and may be a handheld display (HHD). An HHD is a handheld display. In other words, the HMD may be the aforementioned ordinary monitor. It may also be a display that the user holds in their hand and peers into like binoculars to observe images. Furthermore, the display device 1000 may be a display terminal such as a tablet, smartphone, or ordinary monitor. Also, the display device connected to the information processing device may be switched, or multiple display devices may be connected and the display device to which the information processing device outputs may be switched.

[0128] (supplement) The present invention can also be realized by executing the following process: software (program) that realizes the functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and the computer (or control unit, MPU, etc.) of the system or device reads and executes the program code. In this case, the program and the storage medium storing the program constitute the present invention.

[0129] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0130] Note that each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, an apparatus may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the apparatus may be realized by the processor reading and executing the control program from the memory.

[0131] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0132] [Configuration 1] An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. Information processing device.

[0133] [Configuration 2] the transparent mask image is a transparent image based on model information included in the virtual object information. 2. The information processing device according to configuration 1.

[0134] [Configuration 3] The control means When the masking flag is not included in the virtual object information acquired by the acquisition means, control is performed so that a visible image based on model information included in the virtual object information is displayed on the display device. 3. The information processing device according to configuration 1 or 2.

[0135] [Configuration 4] the image different from the transparent mask image is an image obtained by adding color to the transparent mask image; 4. The information processing device according to any one of configurations 1 to 3.

[0136] [Configuration 5] The color to be applied to the transparent mask image is a pre-designated color. 5. The information processing device according to configuration 4.

[0137] [Configuration 6] the color assigned to the transparent mask image is a color based on color information of the real object acquired by the other display device; 5. The information processing device according to configuration 3 or 4.

[0138] [Configuration 7] 2. The information processing device according to configuration 1, wherein the image different from the transparent mask image is a model image based on the transparent mask image.

[0139] [Configuration 8] The control means When the display device and the other display device are present in the same space, control is performed to superimpose the transparent mask image on the real object regardless of whether the masking flag is included in the virtual object information. 8. The information processing device according to any one of configurations 1 to 7.

[0140] [Configuration 9] whether the display device and the other display device are present in the same space is determined based on whether position and orientation information is acquired from the same sensor controller; 9. The information processing device according to configuration 8.

[0141] [Configuration 10] The information processing device is mounted on the display device. 10. The information processing device according to any one of configurations 1 to 9.

[0142] [Configuration 11] A control method for controlling a display of a display device worn by a user, comprising: acquiring virtual object information that is information about a virtual object to be displayed on the display device; Controlling display of an image of the virtual object based on the acquired virtual object information; Equipped with Controlling the display of the image of the virtual object includes: When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. A computer-implemented control method.

[0143] [Configuration 12] Computer, An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. A program that functions as an information processing device.

[0144] [Configuration 13] An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring an image obtained by the display device; a determining means for determining whether or not to superimpose a transparent mask image on the real object on the display screen of the display device based on virtual object information of a virtual object corresponding to the real object included in the image; a transmitting means for transmitting the virtual object information to another information processing device; An information processing device comprising:

[0145] [Configuration 14] the other information processing device is a device that controls a display of another display device worn by the other user, 14. The information processing device according to configuration 13.

[0146] [Configuration 15] the virtual object information includes masking flag information indicating whether or not a transparent mask image is to be superimposed on a real object on the display screen of the display device, and display control information for controlling display of the virtual object on the display screen of the other display device. 15. The information processing device according to configuration 14.

[0147] [Configuration 16] the display control information is set based on the masking flag information. 16. The information processing device according to configuration 15.

[0148] [Configuration 17] the display control information indicates that, when the masking flag information indicates that a transparent mask image is to be superimposed on a real object on the display screen of the display device, an image different from the transparent mask image is to be generated on the display screen of the other display device; 17. The information processing device according to configuration 16.

[0149] [Configuration 18] The transmitting means not transmitting the virtual object information of the virtual object for which a setting indicating that the virtual object should not be shared with the other information processing device has been made to the other display device; 18. The information processing device according to any one of configurations 13 to 17.

[0150] [Configuration 19] An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means If the virtual object information includes a masking flag indicating that a mask image is to be superimposed on a real object, the display of the mask image is controlled based on an environment in which the virtual object information is acquired. Information processing device.

[0151] [Configuration 20] The environment for acquiring the virtual object information is based on an operation mode of the display device. 20. The information processing device according to claim 19.

[0152] [Configuration 21] The operation mode of the display device includes a mode for displaying MR images. 21. The information processing device according to configuration 20.

[0153] [Configuration 22] The environment in which the virtual object information is acquired is an environment including either an environment in which the real object exists or an environment in which the real object does not exist. 22. The information processing device according to any one of configurations 19 to 21.

[0154] [Configuration 23] a real object acquisition means for acquiring information about the real object, the control means assigns a setting to display the real object in front of the virtual object based on the information about the real object and the information about the virtual object; The information processing device according to any one of the nineteenth to twenty-second configurations.

[0155] [Configuration 24] The information on the real object includes at least one of shape information, color information, glossiness, haze, image clarity, and diffusion degree. 24. The information processing device according to claim 23.

[0156] [Configuration 25] the real object acquisition means updates the virtual object information including the masking flag based on the assigned setting. 25. The information processing device according to configuration 23 or 24.

[0157] [Configuration 26] The environment for acquiring the virtual object information is based on the location of each of the plurality of users. 26. The information processing device according to any one of configurations 19 to 25.

Claims

1. An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. Information processing device.

2. the transparent mask image is a transparent image based on model information included in the virtual object information. The information processing device according to claim 1 .

3. The control means When the masking flag is not included in the virtual object information acquired by the acquisition means, control is performed so that a visible image based on model information included in the virtual object information is displayed on the display device. The information processing device according to claim 2 .

4. the image different from the transparent mask image is an image obtained by adding color to the transparent mask image; The information processing device according to claim 1 .

5. The color to be applied to the transparent mask image is a pre-designated color. The information processing device according to claim 4 .

6. the color assigned to the transparent mask image is a color based on color information of the real object acquired by the other display device; The information processing device according to claim 4 .

7. the image different from the transparent mask image is a model image based on the transparent mask image; The information processing device according to claim 1 .

8. The control means When the display device and the other display device are present in the same space, control is performed to superimpose the transparent mask image on the real object regardless of whether the masking flag is included in the virtual object information. The information processing device according to claim 1 .

9. whether the display device and the other display device are present in the same space is determined based on whether position and orientation information is acquired from the same sensor controller; The information processing device according to claim 8 .

10. The information processing device is mounted on the display device. The information processing device according to any one of claims 1 to 9.

11. A control method for controlling a display of a display device worn by a user, comprising: acquiring virtual object information that is information about a virtual object to be displayed on the display device; Controlling display of an image of the virtual object based on the acquired virtual object information; Equipped with Controlling the display of the image of the virtual object includes: When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. A computer-implemented control method.

12. Computer, An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means When the virtual object information includes a masking flag indicating that a transparent mask image is to be superimposed on a real object on a display screen of another display device worn by another user, control is performed so that an image different from the transparent mask image is displayed on the display device. A program that functions as an information processing device.

13. An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring an image obtained by the display device; a determining means for determining whether or not to superimpose a transparent mask image on the real object on the display screen of the display device based on virtual object information of a virtual object corresponding to the real object included in the image; a transmitting means for transmitting the virtual object information to another information processing device; An information processing device comprising:

14. the other information processing device is a device that controls a display of another display device worn by the other user, The information processing device according to claim 13.

15. the virtual object information includes masking flag information indicating whether or not a transparent mask image is to be superimposed on a real object on the display screen of the display device, and display control information for controlling display of the virtual object on the display screen of the other display device. The information processing device according to claim 14.

16. the display control information is set based on the masking flag information. The information processing device according to claim 15.

17. the display control information indicates that, when the masking flag information indicates that a transparent mask image is to be superimposed on a real object on the display screen of the display device, an image different from the transparent mask image is to be generated on the display screen of the other display device; The information processing device according to claim 16.

18. The transmitting means not transmitting the virtual object information of the virtual object for which a setting indicating that the virtual object should not be shared with the other information processing device has been made to the other display device; 18. The information processing device according to claim 13.

19. An information processing device that controls a display of a display device worn by a user, an acquisition means for acquiring virtual object information, which is information on a virtual object to be displayed on the display device; a control means for controlling display of an image of the virtual object based on the virtual object information acquired by the acquisition means; Equipped with The control means If the virtual object information includes a masking flag indicating that a mask image is to be superimposed on a real object, the display of the mask image is controlled based on an environment in which the virtual object information is acquired. Information processing device.

20. The environment for acquiring the virtual object information is based on an operation mode of the display device. The information processing device according to claim 19.

21. The operation modes of the display device include a mode for displaying MR images. The information processing device according to claim 20.

22. The environment in which the virtual object information is acquired is an environment including either an environment in which the real object exists or an environment in which the real object does not exist.

22. The information processing device according to claim 19.

23. a real object acquisition means for acquiring information about the real object, the control means assigns a setting to display the real object in front of the virtual object based on the information about the real object and the information about the virtual object; 23. The information processing device according to claim 19.

24. The information on the real object includes at least one of shape information, color information, glossiness, haze, image clarity, and diffusion degree. The information processing device according to claim 23.

25. the real object acquisition means updates the virtual object information including the masking flag based on the assigned setting.

25. The information processing device according to claim 23 or 24.

26. The environment for acquiring the virtual object information is based on the location of each of the plurality of users.

26. The information processing device according to claim 19.

Images