Information processing system, control method, and program

Abstract

To provide a mechanism capable of looking at a virtual object without causing discomfort to a user.SOLUTION: An information processing system 10 incudes a client device 100 with an HMD 105 connected thereto and a server device 120. A display part 1060 of the HMD 105 displays a composite image obtained by superimposing an image of a virtual object on an image in a real space obtained by photographing a studio 130 being the real space. The information processing system 10 controls a luminous environment in the studio 130 on the basis of illumination information of the virtual object displayed on the display part 1060.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method, and a program.

Background Art

[0002] As technologies for fusing the real world and the virtual world in real time, mixed reality (MR) technology and augmented reality (AR) technology are known. These technologies are technologies for seamlessly fusing the real space and the virtual space (virtual objects) created by a computer.

[0003] As display devices for making a user feel that virtual objects exist in the real space, there are a video see-through type head-mounted display device and an optical see-through type head-mounted display device. The head-mounted display device is, for example, an HMD (Head Mounted Display) (see, for example, Patent Document 1). In the case of the video see-through method, for example, an image of the real space captured by an imaging device (for example, a video camera) is displayed in real time. Then, an image of a virtual object is superimposed and displayed on the image of the real space. In the case of the optical see-through method, for example, the head-mounted display device has lenses similar to those of ordinary glasses, and an image of a virtual object is projected onto the lenses.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a virtual reality (VR) system, in a virtual space independent of the real space, virtual objects are displayed in harmony with the virtual space, so users do not feel discomfort with the appearance of the virtual objects.

[0006] On the other hand, in an MR or AR system, users will see virtual objects placed in the real space. There is a pre-assumed optimal environment in which the virtual objects look natural. If this environment is different from the environment of the real space that the user actually observes, the user will feel discomfort with the appearance of the virtual objects.

[0007] An object of the present invention is to provide a mechanism that allows users to view virtual objects without feeling discomfort.

Means for Solving the Problem

[0008] In order to achieve the above object, the information processing system of the present invention is characterized by including means for controlling the lighting environment in the real space based on the lighting information of virtual objects displayed superimposed on the real space in a display device.

Effect of the Invention

[0009] According to the present invention, users can view virtual objects without feeling discomfort.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail based on the accompanying drawings. First, Embodiment 1 of the present invention will be described. In Embodiment 1, an information processing system combining an HMD and a studio capable of controlling the lighting environment will be described. Also, in Embodiment 1, as an example, a configuration in which the HMD is externally connected to the information processing device will be described as an example. In Embodiment 1, the information processing device is a client device described later, and for example, it is a controller or a PC (Personal Computer).

[0012] Note that the information processing device may be built into the HMD. Therefore, the present invention is also applicable to an HMD which is a type of head-mounted display device. Also, the present invention is applicable to other head-mounted display devices. For example, a hand-held display device that a user holds with their hand and wears on their head is a type of head-mounted display device, and the present invention is applicable to a hand-held display device. The hand-held display device is, for example, a smartphone or a tablet terminal. Smart glasses (AR (Augmented Reality) glasses) are also a type of head-mounted display device, and the present invention is applicable to smart glasses. Also, the present invention is applicable to both head-mounted display devices that a user views images with both eyes and head-mounted display devices that a user views images with one eye. A smartphone mounted on a head-mounted adapter (for example, VR (Virtual Reality) goggles) is a type of head-mounted display device. The present invention is applicable to display devices other than head-mounted display devices (for example, stationary display devices).

[0013] The present invention is applicable to both the video see-through method and the optical see-through method. In the case of the video see-through method, for example, an image of the real space (the external world) captured by an imaging device (e.g., a video camera) is displayed in real time on a display surface (a display surface that does not transmit light from the real space). Then, graphics (e.g., an image of a virtual object) are superimposed (composited) on the image of the real space and displayed. In this case, although the user cannot directly see the real space, by looking at the displayed image, the user can indirectly see the real space or see the graphics superimposed on the image of the real space. In the case of the optical see-through method, for example, graphics (e.g., an image of a virtual object) are displayed on a display surface (a display surface that transmits light from the real space (the external world)). In this case, the user can directly see the real space (the external world) through the display surface or see the graphics displayed on the display surface.

[0014] FIG. 1 is a diagram showing an example of the functional configuration of an information processing system 10 according to Embodiment 1. The information processing system 10 in FIG. 1 includes a server device 120, a client device 100, an HMD 105, and a studio 130. The HMD 105 is, for example, an HMD of the video see-through type. The client device 100 is connected to the server device 120 via a network, and the HMD 105 is connected to the client device 100.

[0015] The client device 100 can display an image of the real space on the HMD 105 and can also superimpose (composite) an image of a virtual object on the image of the real space and display it. A user of the client device 100 (hereinafter referred to as the "first user") can perform an operation (action) on the virtual object while wearing the HMD 105 and viewing the virtual object.

[0016] The server device 120 has a virtual object holding unit 1070. The virtual object holding unit 1070 holds virtual object information. The virtual object information is information regarding a virtual object, and includes, for example, information indicating the shape of the virtual object, information indicating the arrangement position of the virtual object, and information indicating the texture of the surface of the virtual object. Further, the virtual object information includes an environment map which is illumination information of the virtual object. The environment map includes the position of a virtual light source (a light source that illuminates the virtual object) set for appropriately appreciating the virtual object, the intensity of the virtual light source, the color temperature of the virtual light source, and environmental information around the virtual object. Note that the virtual object information may further include an image obtained by photographing the surroundings of the virtual object.

[0017] The HMD 105 has an imaging unit 1040 and a display unit 1060. The imaging unit 1040 images the surroundings of the HMD 105 in the studio 130. The surroundings of the HMD 105 may be regarded as the surroundings of the client device 100. The imaging unit 1040 may be built in the HMD 105 or may be detachably provided with respect to the HMD 105. The imaging unit 1040 is, for example, a video camera or a digital camera. Note that the imaging unit 1040 may be a stereo camera configured to perform imaging in two directions, a direction close to the visual axis of the left eye and a direction close to the visual axis of the right eye. The display unit 1060 displays an image (video) based on the image data (video data) output from the client device 100. The display unit 1060 is, for example, a small liquid crystal display or a small organic EL display. The image displayed on the display unit 1060 is enlarged by a lens provided in the HMD 105, and the first user wearing the HMD 105 views the image enlarged by this lens.

[0018] The client device 100 has a virtual object communication unit 1010, an environment information acquisition unit 1020, and a composition unit 1050.

[0019] The virtual object communication unit 1010 communicates with the server device 120 via a network. The communication by the virtual object communication unit 1010 is, for example, wired LAN (Local Area Network) communication, wireless LAN communication, Wi-fi (registered trademark) communication, Bluetooth (registered trademark) communication, or infrared communication. The virtual object communication unit 1010 receives (acquires) virtual object information from the virtual object holding unit 1070 of the server device 120 via the network. In the present embodiment, for example, when the virtual object information is updated in the client device 100, the virtual object communication unit 1010 transmits the updated virtual object information to the virtual object holding unit 1070 of the server device 120 via the network. The virtual object information held by the virtual object holding unit 1070 is updated to the updated virtual object information transmitted from the virtual object communication unit 1010.

[0020] The environment information acquisition unit 1020 acquires an environment map, which is illumination information of the virtual object, from the virtual object information obtained by the virtual object communication unit 1010. This environment map is transmitted to the real environment control unit 1310 in the studio 130 via the communication I / F 210 shown in FIG. 2 described later. Further, the environment information acquisition unit 1020 acquires real environment information. The real environment information is information regarding the environment around the client device 100 in the studio 130. Details of the method for acquiring the real environment information will be described later.

[0021] The composition unit 1050 acquires an image of the real space obtained by imaging the surroundings of the client device 100 from the imaging unit 1040 of the HMD 105 (image acquisition). The image of the real space is, for example, a moving image. Further, the composition unit 1050 generates a composite image based on the acquired image of the real space and the virtual object information obtained by the virtual object communication unit 1010, and controls the display unit 1060 to display this composite image. For example, the composition unit 1050 generates a composite image by superimposing an image of the virtual object on the image of the real space, and outputs the data of the composite image to the HMD 105.

[0022] In Studio 130, a real environment control unit 1310 and a lighting device 1320 are arranged. The real environment control unit 1310 controls the lighting environment of Studio 130 based on the environment map included in the virtual object information transmitted by the environment information acquisition unit 1020. In the present embodiment, the real environment control unit 1310 controls the lighting device 1320 within Studio 130.

[0023] The lighting device 1320 is a device composed of a plurality of lights, and can change the brightness, color temperature, position of the light source, etc. according to the instructions from the real environment control unit 1310. In the present embodiment, it is assumed that the client device 100 and the HMD 105 are used within Studio 130. Details of the control method of the lighting device 1320 will be described later.

[0024] In the present embodiment, a display device other than the HMD may be used. For example, the real space may be imaged by a USB (Universal Serial Bus) camera, and the composite image may be displayed on a display separate from the imaging unit. Also, the real space may be imaged by a camera built into a laptop PC, and the composite image may be displayed on the display of the laptop PC. Further, the real space may be imaged by a camera built into a tablet PC, and the composite image may be displayed on the display of the tablet PC.

[0025] Figure 2 is a hardware configuration diagram of the client device 100 in Figure 1. The client device 100 has a CPU 201, a RAM 202, a ROM 203, a keyboard 204, a mouse 205, a display unit 206, an external storage device 207, an internal storage device 208, an output I / F 209, and a communication I / F 210. These are connected to each other via a bus 211. Note that the CPU is an abbreviation for Central Processing Unit. The ROM is an abbreviation for Read Only Memory. The RAM is an abbreviation for Random Access Memory.

[0026] The CPU 201 controls each part of the client device 100. Various types of information are stored in advance in the ROM 203. For example, programs of an OS (Operating System), programs of device drivers, and programs of the processing according to Embodiment 1 are stored in advance in the ROM 203. The RAM 202 temporarily stores various types of information. The CPU 201 expands and executes the programs stored in the ROM 203 in the RAM 202.

[0027] The keyboard 204 and the mouse 205 are operation units (input I / F (Interface)) that receive operations (instructions) from the first user. The first user performs various operations (for example, operations on virtual objects) using at least one of the keyboard 204 and the mouse 205. The keyboard 204 and the mouse 205 output operation signals corresponding to the operations (instructions) from the first user, and the CPU 201 performs processing corresponding to the operation signals. Note that the keyboard 204 and the mouse 205 may be configured to be detachable from the client device 100, or may be configured to be provided in the client device 100.

[0028] The display unit 206 displays various images. Note that the display unit 206 may be configured to be detachable from the client device 100, or may be configured to be provided in the client device 100. The external storage device 207 is a storage device that stores various types of information, and is provided detachably with respect to the client device 100. The internal storage device 208 is a storage device that stores various types of information, and is built in the client device 100.

[0029] The output I / F 209 is used to output image data to an external device (for example, the HMD 105). The output I / F 209 may be used to acquire information from the external device. The communication I / F 210 is used to communicate with an external device (for example, the server device 120 or the real environment control unit 1310).

[0030] The virtual object communication unit 1010, the environmental information acquisition unit 1020, and the synthesis unit 1050 in FIG. 1 are realized by the CPU 201. For example, the CPU 201 communicates with the server device 120 via the communication I / F 210 (and the network) so that the virtual object communication unit 1010 is realized. The CPU 201 acquires a real-space image from the output I / F 209 of the HMD 105 so that the synthesis unit 1050 is realized. Then, the CPU 201 generates a synthesized image and outputs the data of the synthesized image to the HMD 105 via the output I / F 209.

[0031] In addition, the CPU 201 also updates the virtual object information received from the virtual object holding unit 1070 of the server device 120. For example, when the first user performs an operation on the virtual object and the state of the virtual object (e.g., the shape of the virtual object, the arrangement position of the virtual object, or the texture of the surface of the virtual object) changes, the CPU 201 updates the virtual object information to reflect it.

[0032] FIG. 3 is a flowchart showing the control process executed by the information processing system 10 in FIG. 1. In FIG. 3, the processes of S (steps) 301 to S307 are processes by the client device 100, and are realized by the CPU 201 of the client device 100 expanding the program stored in the ROM 203 into the RAM 202 and executing it. Also, the processes of steps S311 to S312 are processes in the studio 130, and are realized by a CPU (not shown) in the real environment control unit 1310 expanding the program stored in the ROM into the RAM and executing it. For example, when the first user performs an operation (instruction) for displaying a virtual object on the client device 100, the control process in FIG. 3 is started.

[0033] In FIG. 3, first, the CPU 201 of the client device 100 receives virtual object information from the server device 120 via the network (S301). The CPU 201 acquires an environment map from the received virtual object information. As described above, the environment map is the position of a virtual light source, the intensity of the virtual light source, the color temperature of the virtual light source, and the environmental information around the virtual object, which are set for appropriately appreciating the virtual object. For example, when the environment map is not included in the virtual object information, the CPU 201 of the client device 100 may create an environment map based on information such as the brightness and shadow direction of the image of the virtual object based on the virtual object information. Another information other than the above-described information may be added to the environment map generated in this way.

[0034] Next, the CPU 201 acquires an image of the real space obtained by imaging the surroundings of the client device 100 from the imaging unit 1040 of the HMD 105 (S302).

[0035] Next, the CPU 201 acquires real environment information, which is information about the environment around the client device 100 in the studio 130 (S303). This real environment information includes, for example, the position of a light source, the intensity of the light source, the color temperature of the light source, the brightness of the environment, the exposure information of the imaging unit 1040, and any one of the images of the real space obtained by the imaging unit 1040 in the environment around the client device 100.

[0036] For example, the position of the light source may be obtained (determined) based on the position of the high-brightness region in the image of the real space. The intensity of the light source may be obtained (determined) based on the luminance value of the high-brightness region in the image of the real space. The color temperature of the light source, the brightness of the environment, and the exposure information of the imaging unit 1040 may be obtained from the imaging unit 1040 together with the image of the real space as supplementary information. The color temperature of the light source may be obtained (determined) based on the color information of the image of the real space (for example, the balance of R value, G value, and B value). The brightness of the environment may be obtained based on the luminance value of the entire image of the real space. The intensity of the light source and the brightness of the environment may each be obtained using an illuminance meter, or may be obtained (determined) based on the lighting information of the studio 130 obtained from the real environment control unit 1310. Note that the real environment information obtained in S303 may be information regarding the environment around the client device 100, and is not limited to the information described above. Also, the method for obtaining the real environment information is not particularly limited.

[0037] Next, the CPU 201 compares the environment map obtained in S301 with the real environment information obtained in S303, and determines whether it is necessary to change the real environment (S304). In S304, for example, when the environment map and the real environment information match in a predetermined item, it is determined that there is no need to change the real environment. In this case, this process proceeds to S306 described later. Note that the predetermined item is, for example, the position of the light source, the intensity of the light source, and the color temperature of the light source in the environment around the client device 100. On the other hand, when the environment map and the real environment information do not match in the predetermined item, it is determined that it is necessary to change the real environment. In this case, this process proceeds to S305.

[0038] In S305, the CPU 201 transmits the environment map to the real environment control unit 1310 via the communication I / F 210.

[0039] In Studio 130, when the real environment control unit 1310 acquires an environmental map from the client device 100 (S311), it controls the lighting environment in Studio 130 based on the acquired environmental map (S312). The process of controlling the lighting environment in Studio 130 will be described later. When the lighting environment in Studio 130 is controlled, this process proceeds to S306.

[0040] In S306, the CPU 201 generates a composite image by causing the composite unit 1050 to composite (overlay) the image of the virtual object onto the image of the real space acquired in S302. In generating the composite image, the composite unit 1050 acquires the position and orientation of the imaging unit 1040 and renders the image of the virtual object so that the real space and the virtual object are geometrically aligned. The position and orientation of the imaging unit 1040 may be acquired (estimated) by any method. For example, SLAM (Simultaneous Localization And Mapping) that detects feature points from the captured image (the image obtained by the imaging unit 1040) and simultaneously estimates the self-position and the map of the feature points may be used. Also, a method of detecting an index with a known position in the real space from the captured image may be used. Also, assuming that camera-specific parameters such as the focal length and the frustum are known, when detecting feature points or an index from the captured image, the distortion of the captured image may be corrected using the camera-specific parameters. Also, an optical or magnetic sensor may be used, or an IMU (Inertial Mesurement Unit) may be used. When using a sensor, information on the positional relationship between the sensor and the imaging unit 1040 is prepared in advance.

[0041] The composite unit 1050 may acquire depth information of the real space (for example, the distance from the imaging unit 1040). The method of acquiring the depth information is not particularly limited. Then, the composite unit 1050 may compare the depth information of the virtual object (for example, the distance from the virtual camera) with the depth information of the real space and generate a composite image so that the object in front among the real object (the object in the real space) and the virtual object is displayed. By controlling in this way, the front-back relationship between the real object and the virtual object can be correctly expressed.

[0042] Next, the CPU 201 outputs the data of the generated composite image to the HMD 105. The display unit 1060 of the HMD 105 displays a composite image based on the data of the composite image output from the composite unit 1050.

[0043] Next, the CPU 201 determines whether to end the control process of FIG. 3 (S307). For example, when the first user performs an operation (instruction) to make the virtual object non-displayed (end the display of the virtual object) on the client device 100, the CPU 201 determines to end the control process of FIG. 3. In this case, this process ends. On the other hand, when the above-described operation has not been performed, the CPU 201 determines not to end the control process of FIG. 3. In this case, this process returns to S301.

[0044] Here, the process of controlling the illumination environment in the studio 130 in S312 will be described.

[0045] FIG. 4(a) is a diagram showing a virtual object (apple). As the environment map of this virtual object, information such as being illuminated with a light bulb light (color temperature 3000K) from the left direction with an illuminance of 1000 lx as seen from the user is attached.

[0046] FIG. 4(b) is a diagram showing the state inside the studio 130 before controlling the illumination environment. In the studio 130, five illuminations 1321 to 1325 are arranged as the illumination device 1320. The illuminations 1321 to 1325 are LED light sources of a plurality of colors, and the illumination intensity and color temperature can be independently controlled. In the studio 130, only the illumination 1322 is lit, and there are shadows in the left direction for the table and cup placed in the center of the studio. When the virtual object of FIG. 4(a) is superimposed on such a studio 130, since the direction of the shadow of the real object, that is, the table and cup placed in the center of the studio, is different from the direction of the shadow of the virtual object, a composite image with a sense of discomfort is obtained.

[0047] Figure 4(c) is a diagram showing the state inside Studio 130 after controlling the lighting environment. By the real environment control unit 1310 controlling the lighting environment inside Studio 130 to turn off lighting 1322 and turn on lighting 1324 and lighting 1325, the direction of the shadows of the table and cup, which are real objects, can be aligned with the direction of the shadow of the apple, which is a virtual object. Also, the real environment control unit 1310 can align the brightness of the lighting device 1320 in Studio 130 with the brightness of the virtual light source by adjusting the color temperature to 3000K and controlling the illuminance to 1000 lx. In this way, a composite image without a sense of incongruity can be reproduced.

[0048] Note that in FIGS. 4(b) and 4(c), an incandescent light source is used as the lighting device 1320 inside Studio 130, but the wall of Studio 130 may use a two-dimensional display device such as an LED monitor. In such a configuration, by displaying an image of the surroundings of the virtual object stored as an environment map on the two-dimensional display device, the same effect as in FIG. 4(c) can be obtained.

[0049] According to the above-described embodiment, the lighting environment in Studio 130 (real space) is controlled based on the environment map, which is the lighting information of the virtual object. Thereby, the lighting environment of Studio 130 can be adjusted so that the virtual object is harmonious, and thus the user can view the virtual object without feeling a sense of incongruity.

[0050] Also, in the above-described embodiment, the environment map includes at least information on the position of the light source that illuminates the virtual object, the intensity of the light source, and the color temperature of the light source. Thereby, the lighting environment of Studio 130 (real space) can be adjusted so that the user can view the virtual object without feeling a sense of incongruity.

[0051] Also, in the above-described embodiment, since the environment map is included in the virtual object information used for displaying the virtual object, a virtual object that is harmonious with the real space can be displayed based on the acquired virtual object information.

[0052] In the above-described embodiment, the environment map is generated based on the image of the virtual object. As a result, even if the environment map is not included in the virtual object information, a virtual object that harmonizes with the real space can be displayed.

[0053] In the above-described embodiment, control is performed to change the lighting environment in the studio 130 to the lighting environment corresponding to the environment map. As a result, the lighting environment of the studio 130 can be made to match the lighting environment of the virtual object set for appropriately appreciating the virtual object.

[0054] In the above-described Embodiment 1, an example of controlling the lighting environment of the studio 130 when the environment map in the virtual object information does not match the real environment information in a predetermined item has been described. However, when the difference between the environment map in the virtual object information and the real environment information is relatively small with respect to the intensity of the light source and the color temperature of the light source, the discomfort of the virtual object in the real space can be suppressed by changing the exposure conditions and white balance of the imaging unit 1040. Therefore, in the present embodiment, the control of the lighting environment of the studio 130 by the real environment control unit 1310 and the exposure control of the imaging unit 1040 that controls the exposure and development parameters of the imaging unit 1040 may be combined. Thereby, the discomfort of the virtual object in the real space can be suppressed.

[0055] However, for the portion where the image of the real space captured by the imaging unit 1040 is blacked out, if the shutter speed, which is an exposure condition, is set long, image blur will occur. Also, if the ISO sensitivity is increased, noise will appear in the image. In such a case, it is effective to perform the control of the lighting environment of the studio 130 described in Embodiment 1 without performing the exposure control of the imaging unit 1040. Also, a configuration may be adopted in which the exposure control of the imaging unit 1040 and the control of the lighting environment of the studio 130 are switched.

[0056] Next, Embodiment 2 of the present invention will be described. In the following, the description of the same points as in Embodiment 1 (for example, the same configuration and processing as in Embodiment 1) will be omitted, and the points different from Embodiment 1 will be described.

[0057] In Embodiment 1, an example in which one client device renders an image of a virtual object was described. In Embodiment 2, a system in which a plurality of users wearing separate HMDS share a virtual object will be described.

[0058] FIG. 5 is a diagram showing an example of the functional configuration of the information processing system 50 according to Embodiment 2. The information processing system 50 in FIG. 5 includes a server device 520, a client device 100, an HMD 105, a studio 130, a client device 500, and an HMD 505.

[0059] The server device 520 includes a virtual object holding unit 5070 and a virtual object generation unit 5030. The virtual object holding unit 5070 has the same function as the virtual object holding unit 1070 in Embodiment 1. The virtual object generation unit 5030 receives real environment information from the client device via the network. When receiving the real environment information from the client device, the virtual object generation unit 5030 also receives the ID (identification information) of this client device, and holds the real environment information and the ID associated with each other. The virtual object generation unit 5030 generates an image of a virtual object based on the virtual object information held by the virtual object holding unit 5070 and the real environment information received from this client device. Then, the virtual object generation unit 5030 transmits the generated image of the virtual object to the client device via the network. The client device to which the image of the virtual object is transmitted is the client device with the ID associated with the real environment information used for the generation of the image of the virtual object.

[0060] The client device 500 includes a communication unit 5010, an environmental information acquisition unit 5020, and a synthesis unit 5050. The environmental information acquisition unit 5020 has the same functions as the environmental information acquisition unit 1020 in Embodiment 1, and the synthesis unit 5050 has the same functions as the synthesis unit 1050 in Embodiment 1. Similar to the virtual object communication unit 1010 in Embodiment 1, the communication unit 5010 communicates with the server device 520 via a network. The communication unit 5010 transmits the real environmental information acquired by the environmental information acquisition unit 5020 and the ID of the client device 500 to the virtual object generation unit 5030 of the server device 520 via the network. Also, when the user (second user) of the client device 500 performs an operation on the virtual object and the state of the virtual object changes, the communication unit 5010 transmits information indicating the state after the change (at least a part of the virtual object information) to the server device 520 via the network. The virtual object information held by the virtual object holding unit 5070 of the server device 520 is updated based on the information transmitted from the communication unit 5010.

[0061] The client device 100, the HMD 105, and the studio 130 have the same configuration as in Embodiment 1.

[0062] It is assumed that the client device 500 and the HMD 505 are used in a space separate from the studio 130.

[0063] Also, the hardware configuration of the client device 500 is the same as the hardware configuration of the client device 100 shown in FIG. 1 described above. Hereinafter, the components of the client device 500 are indicated by adding a "'" to the end of the components of the client device 100.

[0064] FIG. 6 is a flowchart showing the control process executed by the information processing system 50 of FIG. 5. In FIG. 6, the processes of S601 to S605 are processes by the client device 500, and are realized by the CPU 201' of the client device 500 expanding and executing the program stored in the ROM 203' in the RAM 202'. The processes of steps S611 to S612 are processes by the server device 520, and are realized by a CPU (not shown) in the server device 520 expanding and executing the program stored in the ROM in the RAM. For example, when the second user performs an operation (instruction) for displaying a virtual object on the client device 500, the control process of FIG. 6 is started.

[0065] In FIG. 6, first, the CPU 201' acquires real environment information regarding the environment around the client device 500 (real space) (S601). Next, the CPU 201' transmits, via the communication unit 5010, the real environment information acquired in S501 and the ID of the client device 500 to the server device 520 via the network (S602).

[0066] In the server device 520, the virtual object generation unit 5030 generates an image of the virtual object based on the virtual object information held by the virtual object holding unit 5070 and the real environment information transmitted from the client device 500 in S602 (S611). The virtual object information held by the virtual object holding unit 5070 is, for example, virtual object information generated or updated by the client device 500. The virtual object generation unit 5030 adjusts (updates) the virtual object information based on the real environment information. For example, the virtual object generation unit 5030 changes the position of the light source, the intensity of the light source, and the color temperature of the light source in the environment map in the virtual object information to the position of the light source, the intensity of the light source, and the color temperature of the light source in the real environment information so that the virtual object harmonizes with the studio where the client device 500 is arranged. An image of the virtual object is generated by performing rendering based on the adjusted virtual object information.

[0067] Next, the virtual object generation unit 5030 transmits the image of the virtual object generated in S511 to the communication unit 5010 of the client device 500 corresponding to the ID transmitted from the client device 500 in S502 via the network (S612).

[0068] In the client device 500 that has received the image of the virtual object, the CPU 201' acquires the image of the real space obtained by imaging the surroundings of the client device 500 from the imaging unit 5040 of the HMD 505 (S603).

[0069] Next, the CPU 201' synthesizes (superimposes) the image of the virtual object transmitted from the server device 520 in S612 on the image of the real space acquired in S603 by the synthesizing unit 5050 to generate a synthesized image (S604). The synthesizing unit 5050 outputs the data of this synthesized image to the HMD 505. The display unit 5060 of the HMD 505 displays the synthesized image based on the data of the synthesized image output from the synthesizing unit 5050.

[0070] Next, the CPU 201' determines whether to end the control process of FIG. 6 (S605). Note that the determination method in S605 is the same as the determination method in S307 described above. If it is determined that the control process of FIG. 6 is not ended, this process returns to S601. On the other hand, if it is determined that the control process of FIG. 6 is ended, this process ends.

[0071] Note that the processing by the client device 100 and the processing in the studio 130 are the same as the control processing of FIG. 3 described above. However, the virtual object information received by the virtual object communication unit 1010 is adjusted (updated) in S611 based on the real environment information corresponding to the environment around the client device 500.

[0072] Therefore, by executing the control process of FIG. 3, the lighting environment of the studio (real space) where the client device 500 is arranged is reproduced in the studio 130 where the client device 100 is arranged.

[0073] As described above, in the second embodiment, when the client device 500 generates an image of a virtual object, the environment around the client device 500 is taken into consideration. As a result, the same effects as those in the first embodiment can be obtained. Also, when an image of a virtual object is generated (rendered) by the server device 520, an environment map is updated based on the real environment information corresponding to the environment around the client device 500. The virtual object information including the updated environment map is transmitted to the client device 100 via the server device 520. Thereby, a plurality of users in different locations can suitably view the virtual object (in an environment equivalent to the environment around a specific client device). In this way, a plurality of users can view the virtual object without discomfort.

[0074] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

[0075] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in a computer of the system or device read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0076] Note that the disclosure of the present embodiment includes the following configurations and methods. (Configuration 1) An information processing system comprising means for controlling an illumination environment in the real space based on illumination information of a virtual object that is superimposed and displayed in the real space on a display device. (Configuration 2) The information processing system according to Configuration 1, wherein the illumination information of the virtual object includes at least information on a position of a light source that illuminates the virtual object, an intensity of the light source, and a color temperature of the light source. (Configuration 3) The information processing apparatus according to Configuration 1 or 2, further comprising means for acquiring information about a virtual object used for displaying the virtual object, wherein the illumination information of the virtual object is included in the information about the virtual object. (Configuration 4) The information processing system according to Configuration 1 or 2, wherein the illumination information of the virtual object is generated based on an image of the virtual object. (Configuration 5) The information processing system according to any one of Configurations 1 to 4, wherein the means for performing the control performs control to change the illumination environment in the real space to an illumination environment corresponding to the illumination information of the virtual object. (Configuration 6) The information processing system according to any one of Configurations 1 to 5, further comprising image acquisition means for photographing the real space, wherein the display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space obtained by photographing the real space. (Configuration 7) The information processing system according to Configuration 6, wherein both exposure control of the image acquisition means and control of the illumination environment in the real space are performed. (Configuration 8) The information processing system according to any one of Configurations 1 to 7, wherein the illumination information of the virtual object is information updated based on the illumination environment around another display device arranged at a location different from the display device.

Description of Reference Numerals

[0077] 10 Information processing system 50 Information processing system 100 Client device 105 HMD 130 Studio 500 Client device 505 HMD 1010 Virtual object communication unit 1040 Imaging unit 1060 Display unit 1310 Real environment control unit 5060 Display unit

Claims

1. An information processing system, comprising means for controlling an illumination environment in the real space based on illumination information of a virtual object that is superimposed and displayed in the real space on a display device.

2. The information processing system according to claim 1, wherein the illumination information of the virtual object includes at least information on the position of a light source that illuminates the virtual object, the intensity of the light source, and the color temperature of the light source.

3. The information processing system according to claim 1, further comprising means for acquiring information on the virtual object used for displaying the virtual object, wherein the illumination information of the virtual object is included in the information on the virtual object.

4. The information processing system according to claim 1, wherein the illumination information of the virtual object is generated based on an image of the virtual object.

5. The information processing system according to claim 1, wherein the means for performing the control performs control to change the illumination environment in the real space to an illumination environment corresponding to the illumination information of the virtual object.

6. The information processing system according to claim 1, further comprising image acquisition means for photographing the real space, wherein the display device displays a composite image in which an image of the virtual object is superimposed on an image of the real space obtained by photographing the real space.

7. The information processing system according to claim 6, wherein both exposure control of the image acquisition means and control of the illumination environment in the real space are performed.

8. The information processing system according to claim 1, wherein the illumination information of the virtual object is information updated based on the illumination environment around another display device arranged at a location different from the display device.

9. A control method, comprising a step of controlling an illumination environment in the real space based on illumination information of a virtual object that is superimposed and displayed in the real space on a display device.

10. A program for causing a computer to execute control of an illumination environment in the real space based on illumination information of a virtual object that is superimposed and displayed in the real space on a display device.

Images