Information processing apparatus

Abstract

To provide a technique which enables a plurality of users at different places to suitably view a virtual object.SOLUTION: An information processing apparatus of the present invention is a client device connected to a network and includes: receiving means which receives first information being information on a virtual object through the network; acquisition means which acquires second information being information on an ambient environment of the client device; generation means which generates an image of the virtual object on the basis of the first information and the second information; and control means which controls a display unit to display the image of the virtual object generated by the generation means on the display unit.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, and more particularly to a technique for displaying virtual objects for mixed reality or augmented reality. [Background technology]

[0002] Mixed reality (MR) technology and augmented reality (AR) technology are known as technologies that fuse the real world and the virtual world in real time. These technologies seamlessly fuse real space with a virtual space (virtual object) created by a computer. These technologies are expected to be applied to various fields such as assembly support and surgery support. In assembly support, for example, a screen showing the procedure of an assembly work or the state of the work (e.g. wiring) is superimposed on the real space. In surgery support, for example, a graphic showing the state inside the patient's body is superimposed on the surface of the body.

[0003] Display devices for making a user feel that a virtual object actually exists in real space include a video see-through head-mounted display device and an optical see-through head-mounted display device. An example of a head-mounted display device is an HMD (Head Mounted Display). In the case of the video see-through method, for example, an image of the real space captured by an imaging device (for example, a video camera) is displayed in real time. Then, an image of the virtual object is displayed superimposed on the image of the real space. In the case of the optical see-through method, for example, the head-mounted display device has a lens similar to the lens of ordinary glasses, and an image of the virtual object is projected onto the lens.

[0004] There are also systems in which multiple users can share virtual objects over a network. With such systems, an operation (operation) performed by a certain user on a virtual object is reflected in the virtual object viewed by other users, allowing multiple users to work on the virtual object in a cooperative manner.

[0005] Patent Document 1 discloses a client system that downloads content via the Internet, renders the content, and displays it on an HMD. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2018-514017 Summary of the Invention [Problem to be solved by the invention]

[0007] In a virtual reality (VR) system, users view virtual objects in a virtual space that is unrelated to the real world. Therefore, the appearance of virtual objects is the same for multiple users.

[0008] On the other hand, in a system for MR or AR, a user sees a virtual object placed in real space. If the surrounding (real space) environment differs between multiple users, a discrepancy occurs in how the virtual object appears between the multiple users. Specifically, an optical discrepancy occurs in how the virtual object appears between the real space and the virtual object, and the amount of discrepancy varies between multiple users. As a result, a discrepancy occurs in the impression that the virtual object gives between multiple users, making it difficult to perform tasks and sensory evaluations. In addition, if the real space and the virtual object are not optically aligned, If you don't, users will feel uncomfortable.

[0009] An object of the present invention is to provide a technique that enables a plurality of users in different locations to view a virtual object in an optimal manner. [Means for solving the problem]

[0010] A first aspect of the present invention is an information processing device that is a client device connected to a network, comprising: a receiving means for receiving first information, which is information about a virtual object, via the network; an acquiring means for acquiring second information, which is information about the environment surrounding the client device; a generating means for generating an image of the virtual object based on the first information and the second information; and a control means for controlling the image of the virtual object generated by the generating means to be displayed on a display unit.

[0011] A second aspect of the present invention is an information processing device which is a server device connected to a network, comprising: a receiving means for receiving second information, which is information regarding the surrounding environment of a client device connected to the network, via the network; and a control means for controlling the device to provide a predetermined notification to a user of the first client device when a difference between the second information of a first client device and the second information of a second client device is greater than a threshold value.

[0012] A third aspect of the present invention is an information processing device that is a server device connected to a network, and is characterized in having a receiving means for receiving second information via the network, which is information regarding the surrounding environment of a client device connected to the network, a generating means for generating an image of a virtual object based on the second information, and a transmitting means for transmitting the image of the virtual object generated by the generating means to the client device via the network.

[0013] A fourth aspect of the present invention is an information processing device that is a client device connected to a network, comprising: a receiving means for receiving an image of a virtual object via the network; an acquiring means for acquiring second information which is information about the environment surrounding the client device; a processing means for performing image processing on the image of the virtual object based on the second information; and a control means for controlling the image of the virtual object after the image processing to be displayed on a display unit.

[0014] A fifth aspect of the present invention is an information processing device characterized in that it comprises a first client device connected to a network, a receiving means for receiving, via the network, information or an image of a virtual object and second information which is information about the surrounding environment of a second client device connected to the network, an acquiring means for acquiring a real-space image captured around the first client device, a processing means for performing image processing on the real-space image based on the second information, and a control means for controlling a composite image in which an image of the virtual object generated based on the information of the virtual object is superimposed on the real-space image after the image processing to be displayed on a display unit.

[0015] A sixth form of the present invention is a control method for a client device connected to a network, comprising the steps of receiving first information, which is information about a virtual object, via the network, acquiring second information, which is information about the environment surrounding the client device, generating an image of the virtual object based on the first information and the second information, and controlling a display unit to display the generated image of the virtual object.

[0016] A seventh aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device described above. Effect of the Invention

[0017] According to the present invention, a plurality of users in different locations can view a virtual object in an optimal manner. [Brief description of the drawings]

[0018] [Figure 1] 1 is a functional configuration diagram of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a hardware configuration diagram of a client device according to the first embodiment. [Diagram 3] 5 is a flowchart showing a process of the client device according to the first embodiment. [Figure 4] FIG. 11 is a functional configuration diagram of an information processing system according to a second embodiment. [Diagram 5] 10 is a flowchart showing a process of an information processing system according to a second embodiment. [Figure 6] FIG. 11 is a functional configuration diagram of an information processing system according to a third embodiment. [Figure 7] 11 is a flowchart showing a process of a client device according to the third embodiment. [Figure 8] FIG. 13 is a modified example of the functional configuration diagram of the information processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] (Embodiment 1) A first embodiment of the present invention will be described. In the first embodiment, an example in which the present invention is applied to an information processing device externally connected to an HMD (Head Mounted Display) will be described. The information processing device to which the present invention is applied in the first embodiment is a client device, which will be described later, and is, for example, a controller or a PC (Personal Computer).

[0020] The information processing device may be built into the HMD. Therefore, the present invention is also applicable to an HMD, which is a type of head-mounted display device. The present invention is also applicable to other head-mounted display devices. For example, a handheld display device that a user holds in his / her hand and wears (places) on his / her head is a type of head-mounted display device, and the present invention is also applicable to the handheld display device. The handheld display device is, for example, a smartphone or a tablet terminal. Smart glasses (AR (Augmented Reality) glasses) are also a type of head-mounted display device, and the present invention is also applicable to smart glasses. The present invention is also applicable to a head-mounted display device in which a user views an image with both eyes, and a head-mounted display device in which a user views an image with one eye. A smartphone mounted on a head-mounted adapter (for example, VR (Virtual Reality) goggles) is a type of head-mounted display device. The present invention is also applicable to a display device other than a head-mounted display device (for example, a stationary display device). The present invention is also applicable to an information processing system including an information processing device. The present invention is also applicable to a server device described later.

[0021] The present invention is applicable to both the video see-through method and the optical see-through method. In the case of the video see-through method, for example, an image of the real space (outside world) captured by an imaging device (for example, a video camera) is displayed in real time on a display surface (a display surface that does not transmit light from the real space). Then, a graphic (for example, an image of a virtual object) is displayed superimposed (combined) on the image of the real space. In this case, the user cannot directly see the real space, but can indirectly see the real space or see the graphic superimposed on the image of the real space by looking at the displayed image. In the case of the optical see-through method, for example, a graphic (for example, an image of a virtual object) is displayed on a display surface (a display surface that transmits light from the real space (outside world)). In this case, the user can directly see the real space (outside world) through the display surface or see the graphic displayed on the display surface.

[0022] Fig. 1 is a functional configuration diagram of an information processing system according to the first embodiment. The information processing system of Fig. 1 includes a server device 120, a first client device 100, a second client device 110, a first HMD 105, and a second HMD 115. Each of the first HMD 105 and the second HMD 115 is a video see-through type HMD. The first client device 100 and the second client device 110 are connected to the server device 120 via a network, the first HMD 105 is connected to the first client device 100, and the second HMD 115 is connected to the second client device 110. The number of client devices may be more than two. The number of HMDs may also be more than two.

[0023] The first client device 100 can display an image of the real space on the first HMD 105, and can also display an image of a virtual object by superimposing (compositing) it on the image of the real space. A user (first user) of the first client device 100 can perform an operation (operation) on the virtual object while wearing the first HMD 105 and looking at the virtual object. Similarly, the second client device 110 can display an image of the real space on the second HMD 115, and can also display an image of a virtual object by superimposing (compositing) it on the image of the real space. A user (second user) of the second client device 110 can perform an operation (operation) on the virtual object while wearing the second HMD 115 and looking at the virtual object. The first client device 100 and the second client device 110 can share the virtual object via a network.

[0024] The server device 120 has a virtual object holding unit 1070. The virtual object holding unit 1070 holds virtual object information. The virtual object information is information about a virtual object. For example, the virtual object information includes the shape of the virtual object, the arrangement of the virtual object, and the texture of the surface of the virtual object. The virtual object information also includes the position of a virtual light source (a light source that illuminates the virtual object), the intensity of the virtual light source, the color temperature of the virtual light source, and an environment map that indicates the environment around the virtual object. The virtual object information includes exposure information of a virtual camera (a camera that captures the virtual object). In the first client device 100, for example, assuming a case in which the virtual camera captures an image of a virtual object illuminated by a virtual light source, an image corresponding to an image obtained by the virtual camera is generated as an image of the virtual object.

[0025] The first HMD 105 has an imaging unit 1040 and a display unit 1060. The imaging unit 1040 captures an image of the surroundings of the first HMD 105. The surroundings of the first HMD 105 may be regarded as the surroundings of the first client device 100. The imaging unit 1040 may be built into the first HMD 105, or may be provided detachably with respect to the first HMD 105. The imaging unit 1040 is, for example, a video camera or a digital camera. The imaging unit 1040 may be a stereo camera configured to capture images in two directions, a direction close to the visual axis of the left eye and a direction close to the visual axis of the right eye. The display unit 1060 displays an image (video) based on image data (video data) output from the first client device 100. The display unit 1060 is, for example, a small liquid crystal display or a small organic EL display. The image displayed on the display unit 1060 is magnified by a lens provided in the first HMD 105, and the first user wearing the first HMD 105 sees the magnified image.

[0026] The first client device 100 includes a virtual object communication unit 1010 , a real environment information acquisition unit 1020 , a virtual object generation unit 1030 , and a synthesis unit 1050 .

[0027] The virtual object communication unit 1010 communicates with the server device 120 via a network. The communication is, for example, wired LAN (Local Area Network) communication, wireless LAN communication, Wi-fi (registered trademark) communication, Bluetooth (registered trademark) communication, or infrared communication. The first client device 100 receives (acquires) virtual object information from the virtual object communication unit 1010 via the network. When the virtual object information is updated in the first client device 100, the virtual object communication unit 1010 transmits the updated virtual object information to the virtual object holding unit 1070 of the server device 120 via the network. The virtual object information held by the virtual object holding unit 1070 is updated with the virtual object information transmitted from the virtual object communication unit 1010.

[0028] The real environment information acquisition unit 1020 also acquires real environment information relating to the environment around (real space) the first client device 100. The method of acquiring the real environment information will be described later in detail.

[0029] The virtual object generation unit 1030 generates a virtual object image (an image of a virtual object) based on the virtual object information acquired by the virtual object communication unit 1010 and the real environment information acquired by the real environment information acquisition unit 1020. The method of generating a virtual object image will be described later in detail.

[0030] The synthesis unit 1050 acquires a real space image obtained by capturing an image of the surroundings of the first client device 100 from the imaging unit 1040 of the first HMD 105 (image acquisition). The real space image is an image of real space, for example, a moving image. The synthesis unit 1050 also controls the display unit 1060 of the first HMD 105 to display the virtual object image generated by the virtual object generation unit 1030. In the first embodiment, the synthesis unit 1050 controls the display unit 1060 to display a synthesized image obtained by superimposing (overlapping) the virtual object image generated by the virtual object generation unit 1030 on the acquired real space image. For example, the synthesis unit 1050 generates a synthesized image by superimposing the virtual object image on the real space image, and outputs (data of) the synthesized image to the first HMD 105.

[0031] The second HMD 115 has a similar configuration to the first HMD 105. The second HMD 115 has an imaging unit 1140 and a display unit 1160, like the first HMD 105. The second client device 110 has a similar configuration to the first client device 100. The second client device 110 has a virtual object communication unit 1110, a real environment information acquisition unit 1120, a virtual object generation unit 1130, and a synthesis unit 1150, like the first client device 100.

[0032] As described above, a display device other than an HMD may be used. For example, the real space may be captured by a Universal Serial Bus (USB) camera, and a composite image may be displayed by a display separate from the image capture unit. The real space may be captured by a camera built into a laptop PC, and a composite image may be displayed by the display of the laptop PC. The real space may be captured by a camera built into a tablet PC, and a composite image may be displayed by the display of the tablet PC.

[0033] 2 is a diagram showing the hardware configuration of the first client device 100. The first client device 100 has a CPU 201, a RAM 202, a ROM 203, a keyboard 204, a mouse 205, a display unit 206, an external storage device 207, an internal storage device 208, an output I / F 209, and a communication I / F 210. These are connected to each other via a bus 211. As described above, the second client device 110 has a similar configuration to that of the first client device 100.

[0034] A CPU (Central Processing Unit) 201 controls each part of the first client device 100. A ROM (Read Only Memory) 203 stores various information in advance. For example, the ROM 203 stores an operating system (OS) program, a device driver program, and a program for processing according to the first embodiment in advance. A RAM (Random Access Memory) 204 stores a RAM (RAM) 205. The memory 202 temporarily stores various information. The CPU 201 loads a program stored in the ROM 203 into the RAM 202 and executes it.

[0035] Each of the keyboard 204 and the mouse 205 is an operation unit (input I / F (Interface)) that accepts operations (instructions) from the first user. The first user performs various operations (e.g., tasks on virtual objects) using at least one of the keyboard 204 and the mouse 205. Each of the keyboard 204 and the mouse 205 outputs an operation signal according to the operation (instructions) from the first user, and the CPU 201 performs processing according to the operation signal. Each of the keyboard 204 and the mouse 205 may or may not be provided detachably with respect to the first client device 100.

[0036] The display unit 206 displays various images. The display unit 206 may or may not be provided detachably with respect to the first client device 100. The external storage device 207 is a storage device that stores various information, and is provided detachably with respect to the first client device 100. The internal storage device 208 is a storage device that stores various information, and is built into the first client device 100.

[0037] The output I / F 209 is used to output image data to an external device (e.g., the first HMD 105). The output I / F 209 may be used to obtain information from an external device. The communication I / F 210 is used to communicate with an external device (e.g., the server device 120).

[0038] 1 are realized by the CPU 201. For example, the CPU 201 communicates with the server device 120 via the communication I / F 210 (and a network) so as to realize the virtual object communication unit 1010. The CPU 201 acquires a real space image from the first HMD 105 via the output I / F 209 so as to realize the synthesis unit 1050. Then, the CPU 201 generates a synthesis image and outputs the synthesis image to the first HMD 105 via the output I / F 209.

[0039] CPU 201 also updates the virtual object information received from virtual object holding unit 1070 of server device 120. For example, when the first user performs an operation on a virtual object and the state of the virtual object (for example, the shape of the virtual object, the arrangement of the virtual object, or the texture of the surface of the virtual object) changes, CPU 201 updates the virtual object information to reflect that.

[0040] The hardware configuration of the second client device 110 is also similar to that of the first client device 100. Like the first client device 100, the server device also has a CPU, a RAM, and a ROM.

[0041] Fig. 3 is a flowchart showing the processing of the first client device 100. The processing of Fig. 3 is realized by the CPU 201 expanding a program stored in the ROM 203 into the RAM 202 and executing it. For example, when the first user performs an operation (instruction) for displaying a virtual object on the first client device 100, the processing of Fig. 3 starts.

[0042] In step S301, the virtual object communication unit 1010 receives virtual object information from the virtual object storage unit 1070 of the server device 120 via the network. This virtual object information is, for example, virtual object information generated or updated by the second client device 110.

[0043] In step S302, the synthesis unit 1050 captures an image of the surroundings of the first client device 100. The captured real space image is acquired from the imaging unit 1040 of the first HMD 105.

[0044] In step S303, the real-world environment information acquisition unit 1020 acquires real-world environment information related to the environment (real-world space) around the first client device 100. This real-world environment information includes, for example, at least any one of the positions of light sources in the environment around the first client device 100, the intensity of the light sources, the color temperature of the light sources, an environment map showing the environment, the brightness of the environment, the exposure information of the imaging unit 1040, and the real-world space image acquired by the imaging unit 1040.

[0045] The real environment information is acquired, for example, based on the real space image acquired in step S302. The real environment information may or may not reflect the real space environment in real time. The real environment information may be acquired based on a real space image stored in advance in a storage device (for example, the external storage device 207 or the internal storage device 208).

[0046] For example, the position of the light source may be acquired (determined) based on the position of a high-luminance area in the real space image. The intensity of the light source may be acquired (determined) based on the luminance value of a high-luminance area in the real space image. The color temperature of the light source, the brightness of the environment, and the exposure information of the imaging unit 1040 may be acquired from the imaging unit 1040 together with the real space image as incidental information. The color temperature of the light source may be acquired (determined) based on color information of the real space image (e.g., the balance of the R value, the G value, and the B value). The brightness of the environment may be acquired based on the luminance value of the entire real space image. Each of the intensity of the light source and the brightness of the environment may be acquired using a light meter. The environment map is, for example, an omnidirectional HDR (High Dynamic Range) image obtained by capturing 360° around the first client device 100. The environment map may be acquired based on a plurality of real space images captured in different directions, or may not be acquired based on such a case. The environment map may be acquired using the imaging unit 1040, or may be acquired using a camera different from the imaging unit 1040. The environment map may be acquired using a camera with a shooting range (angle of view) narrower than 360°, or may be acquired using a 360° camera. The environment map may or may not be prepared in advance. The environment map may be generated by instructing the first user wearing the first HMD 105 to look in multiple directions and synthesizing multiple real space images (multiple real space images corresponding to the multiple directions) acquired by the imaging unit 1040. The environment map may be generated by synthesizing multiple real space images in the background while the first user is watching the video displayed on the first HMD 105.

[0047] The real environment information acquired in step S303 is not limited to the above information, as long as it is information about the environment around the first client device 100. In addition, the method of acquiring the real environment information is not particularly limited.

[0048] In step S304, the virtual object generation unit 1030 generates a virtual object image based on the virtual object information acquired in step S301 and the real environment information acquired in step S303. For example, the virtual object generation unit 1030 adjusts (updates) the virtual object information based on the real environment information, and generates a virtual object image by performing rendering based on the adjusted virtual object information. In the first embodiment, the virtual object information includes the same type of information as the real environment information. The virtual object generation unit 1030 replaces (overwrites) at least a part of the virtual object information with the real environment information, and performs rendering based on the replaced virtual object information.

[0049] If the real environment information cannot be acquired in step S303, the virtual object generation unit 1030 generates a virtual object image based on the virtual object information acquired in step S301, without using the real environment information. If the real environment information cannot be acquired in step S303, the virtual object generation unit 1030 may use real environment information acquired in the past (for example, last time).

[0050] The virtual object generating unit 1030 may paste the environment map included in the real environment information as a reflection image inside a virtual sphere or cube that encloses the virtual object, and simulate the surrounding environment being reflected on the surface of the virtual object. This method allows the reflection to be expressed (calculated) in a shorter time (less processing amount) than the ray tracing method. The virtual object generating unit 1030 may use the light source information (e.g., the position of the light source, the intensity of the light source, and the color temperature of the light source) included in the real environment information and the environment map to express (calculate) the reflection by the ray tracing method. In this case, for example, the virtual object generating unit 1030 uses the environment map to generate a three-dimensional model that imitates the real space (a three-dimensional model on which the virtual space, for example, the environment map (image of the real space) is pasted as a texture). Then, the reflection of the three-dimensional model on the virtual object is expressed (calculated) by the ray tracing method using the light source information. The ray tracing method calculates various phenomena that can occur in real space, resulting in more realistic reflections. For example, the ray tracing method calculates the refraction and reflection of light that occurs on the surface of water or an object based on the amount and direction of light.

[0051] The virtual object generation unit 1030 may acquire the position and orientation of the image capture unit 1040 and set them as the position and orientation of the virtual camera. In this way, a virtual object image can be rendered so that the real space and the virtual object are geometrically consistent. The position and orientation of the image capture unit 1040 may be acquired (estimated) by any method. For example, SLAM (Simultaneous Localization and Modeling) may be used, which detects feature points from a captured image (an image obtained by the image capture unit 1040) and simultaneously estimates the self-position and a map of the feature points. A method of detecting an index whose position in real space is known from a captured image may be used. Assuming that camera-specific parameters such as focal length and frustum are known, distortion of the captured image may be corrected using the camera-specific parameters when detecting feature points or indices from the captured image. An optical or magnetic sensor or an IMU (Inertial Measurement Unit) may be used. When a sensor is used, information on the positional relationship between the sensor and the imaging unit 1040 is prepared in advance.

[0052] In step S305, the synthesis unit 1050 generates a synthetic image by synthesizing (superimposing) the virtual object image generated in step S304 on the real space image acquired in step S302. Then, the synthesis unit 1050 outputs (data of) the generated synthetic image to the first HMD 105. The display unit 1060 of the first HMD 105 displays the synthetic image based on the data of the synthetic image output from the synthesis unit 1050.

[0053] The synthesis unit 1050 may acquire depth information of the real space (e.g., distance from the imaging unit 1040). The method of acquiring the depth information is not particularly limited. The synthesis unit 1050 may compare the depth information of the virtual object (e.g., distance from the virtual camera) with the depth information of the real space to generate a synthetic image such that the real object (object in the real space) and the virtual object that is in the foreground are displayed. In this way, the front-to-back relationship between the real object and the virtual object can be correctly expressed.

[0054] In step S306, CPU 201 determines whether or not to end the processing in Fig. 3. For example, if the first user performs an operation (instruction) on first client device 100 to hide the virtual object (to end the display of the virtual object), CPU 201 determines to end the processing in Fig. 3. If such an operation is not performed, CPU 201 determines not to end the processing in Fig. 3. If CPU 201 determines to end the processing in Fig. 3, it ends the processing in Fig. 3, and if it determines not to end the processing in Fig. 3, it returns to step S301.

[0055] As described above, according to the first embodiment, when the client device generates a virtual object image, the surrounding environment of the client device is taken into consideration. This allows multiple users in different locations to view the virtual object conveniently (without feeling uncomfortable). For example, even if the surrounding (real space) environment is different between multiple users, there is no large optical difference in the appearance between the real space and the virtual object, and there is no large difference in the appearance of the virtual object between multiple users. As a result, there is no large difference in the impression that the virtual object gives between multiple users, and each user can perform the work and sensory evaluation conveniently.

[0056] (Variation 1-1) In the first embodiment, an example has been described in which the virtual object generation unit 1030 adjusts the virtual object information based on the real environment information, and generates a virtual object image based on the adjusted virtual object information. However, the method of generating a virtual object image is not limited to this. For example, the virtual object generation unit 1030 may generate a virtual object image based on the virtual object information received from the server device 120 without adjusting the virtual object information, and perform image processing on the virtual object image based on the real environment information. Then, the synthesis unit 1050 may control the display unit 1060 of the first HMD 105 to display the virtual object image after the image processing.

[0057] Image processing is not particularly limited. For example, the virtual object generation unit 1030 may change the brightness (luminance value) of the virtual object image according to the brightness of the environment included in the real environment information. To brighten the virtual object image, the luminance value of the virtual object image may be increased, and to darken the virtual object image, the luminance value of the virtual object image may be decreased. The virtual object generation unit 1030 may change the luminance value of a part of the virtual object image according to the position and intensity of the light source included in the real environment information. The virtual object generation unit 1030 may change the color tone of the virtual object image (for example, the balance of the R value, the G value, and the B value) according to the color temperature of the light source included in the real environment information.

[0058] In the modified example 1-1, the surrounding environment of the client device is also taken into consideration when the client device generates a virtual object image, thereby making it possible to obtain the same effects as in the first embodiment.

[0059] (Variation 1-2) In the first embodiment, for example, acquisition of real environment information (and generation of virtual object images and generation of composite images) is performed at the imaging rate of the real space image (frame rate of a video that is a real space image). However, if the acquisition of real environment information is performed at a high frequency, a high-spec computing device is required, and manufacturing costs increase. Therefore, the real environment information acquisition unit 1020 may acquire the real environment information at a frequency lower than the imaging rate of the real space image. For example, the real environment information acquisition unit 1020 may acquire the real environment information at a frequency or timing according to an instruction from the first user. In the case of a room where the light source is constant, the change in the environment of the real space is considered to be sufficiently small, so the real environment information acquisition unit 1020 may acquire the real environment information only once. For example, the real environment information acquisition unit 1020 may acquire the real environment information only when the first client device 100 is started. The real environment information acquisition unit 1020 may acquire the real environment information at a timing when the environment around the first client device 100 (real space) changes. The real environment information acquisition section 1020 may acquire real environment information during a period in which the surrounding environment of the first client device 100 is determined to be different from the surrounding environment of the second client device 110.

[0060] According to the modification 1-2, the number of times that the real environment information is acquired is reduced, which has the effect of reducing the processing load, shortening the processing time, and reducing the manufacturing cost.

[0061] (Variation 1-3) In the first embodiment, the virtual object generation unit 1030 generates a virtual object based on real environment information and virtual object information. An example of generating a virtual object image has been described above, but the virtual object generation unit 1030 may generate a virtual object image based on virtual object information received from the server device 120 without using real environment information.

[0062] The server device 120 may have a receiving unit that receives real-world environment information from the client device via a network. The server device 120 may have a storage unit that stores the real-world environment information of the client device in association with the ID (identification information) of the client device. The server device 120 may have a control unit that controls the server device 120 to give a predetermined notification to the first user when a difference between the real-world environment information of the first client device 100 and the real-world environment information of the second client device 110 is greater than a threshold (predetermined value). For example, the control unit transmits an instruction (control signal) to the first client device 100 to give a predetermined notification to the first user. The CPU 201 of the first client device 100 controls the server device 120 to give a predetermined notification to the first user according to the instruction (control signal) output from the server device 120. At this time, the control unit of the server device 120 may control the server device 120 to give a predetermined notification to both the first user and the second user, or may control the server device 120 to give a predetermined notification only to the first user. The user to whom the predetermined notification has been given can understand that the environment of the real space is different from that of other users. The notification method is not particularly limited. For example, the notification may be a display of text or an icon, or may be an audio output.

[0063] According to the modification 1-3, the predetermined notification allows the user to understand that the environment of the real space is different from that of other users. This is expected to have the effect of allowing the user to perform the work and the sensory evaluation appropriately while taking into account the difference in the environment of the real space. The predetermined notification may be a notification that only indicates that the environment of the real space is different from that of other users, or may be a notification of the details of the difference in the environment of the real space. For example, the user may be notified that the user is viewing a virtual object under white lighting, while other users are viewing the virtual object under yellow lighting. By being notified of the details of the difference in the environment of the real space, the user can imagine how the virtual object looks to other users, and can more reliably perform the work and the sensory evaluation appropriately.

[0064] Note that the control method when there are three or more users (three or more client devices) is not particularly limited. For example, the control unit of the server device 120 may compare a representative value (e.g., maximum value, average value, median value, mode, or minimum value) of the difference between two pieces of real environment information with a threshold value. Then, the control unit may control to give a predetermined notification to all users when a difference larger than the threshold value exists. The control unit may control to give a predetermined notification only to at least one of two users (two client devices) corresponding to two pieces of real environment information between which a difference larger than the threshold value is obtained. The control unit may compare the variance (e.g., standard deviation) of all the real environment information with a threshold value. Then, the control unit may control to give a predetermined notification to all users when the variance is larger than the threshold value.

[0065] (Variation 1-4) In the modification 1-3, an example in which the virtual object generation unit 1030 generates a virtual object image without using real environment information has been described, but the virtual object generation unit 1030 may use real environment information as necessary when generating a virtual object image. For example, the server device 120 may instruct the first client device 100 to use the real environment information when a difference between the real environment information of the first client device 100 and the real environment information of the second client device 110 is greater than a threshold value (a predetermined value). Then, the first client device 100 may use the real environment information only when instructed by the server device 120. The predetermined notification of the modification 1-4 may or may not be performed.

[0066] According to the modification 1-4, when generating a virtual object image, the real environment information is used as necessary, which can reduce the processing load compared to a configuration in which the real environment information is always used.

[0067] (Variation 1-5) In the first embodiment, an example in which one server device 120 is used has been described. However, a plurality of server devices may be used, and the number of server devices is not particularly limited. For example, as shown in FIG. 8, the information processing system may have two server devices 821 and 822. The server device 821 has a virtual object holding unit 8071 similar to the virtual object holding unit 1070, and the server device 822 has a virtual object holding unit 8072 similar to the virtual object holding unit 1070. The virtual object holding unit 8071 and the virtual object holding unit 8072 communicate with each other so as to share virtual object information (store the same virtual object information). The other configurations are the same as those of the first embodiment.

[0068] In Modifications 1-5, it is possible to obtain the same effects as in embodiment 1. Moreover, the processing of server device 120 in embodiment 1 can be shared among a plurality of server devices so as to reduce the processing load on one server device.

[0069] (Embodiment 2) A description will be given of a second embodiment of the present invention. Note that in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0070] In the first embodiment, an example has been described in which a client device renders a virtual object image. However, the calculation load of rendering is large, and each client device needs to be provided with an expensive GPU (Graphics Processing Unit) with high computing power. Therefore, in the second embodiment, the server device renders the virtual object image. This can reduce the processing load of the client device, shorten the processing time of the client device, and reduce the manufacturing cost of the client device. Such a configuration is called remote rendering or cloud rendering. The client device obtains (the data of) the virtual object generated by the server device from the server device.

[0071] Fig. 4 is a functional configuration diagram of an information processing system according to embodiment 2. The information processing system in Fig. 4 includes a server device 420, a first client device 400, a second client device 410, a first HMD 105, and a second HMD 115.

[0072] The server device 420 has a virtual object holding unit 4070 and a virtual object generation unit 4030. The virtual object holding unit 4070 has the same function as the virtual object holding unit 1070 of the first embodiment. The virtual object generation unit 4030 receives real environment information from a client device via a network. When receiving real environment information from the client device, the virtual object generation unit 4030 also receives an ID (identification information) of the client device, and holds the real environment information and the ID in association with each other. The virtual object generation unit 4030 generates a virtual object image based on the virtual object information held by the virtual object holding unit 4070 and the real environment information received from the client device. Then, the virtual object generation unit 4030 transmits the generated virtual object image to the client device via a network. The client device to which the virtual object image is transmitted is a client device having an ID associated with the real environment information used to generate the virtual object image.

[0073] The first client device 400 includes a communication unit 4010, a real environment information acquisition unit 4020, and a synthesis unit 4050. The real environment information acquisition unit 4020 has a function similar to that of the real environment information acquisition unit 1020 of the first embodiment, and the synthesis unit 4050 has a function similar to that of the synthesis unit 4050 of the first embodiment. The communication unit 4010 communicates with the server device 420 via a network, similar to the virtual object communication unit 1010 of the first embodiment. The communication unit 4010 transmits the real environment information acquired by the real environment information acquisition unit 4020 and the ID of the first client device 400 to the virtual object generation unit 4030 of the server device 420 via the network. In addition, when the first user performs an operation on a virtual object and the state of the virtual object changes, the communication unit 4010 transmits information indicating the state after the change (at least a part of the virtual object information) to the virtual object holding unit 4070 of the server device 420 via the network. The virtual object information held by the virtual object holding unit 4070 is updated based on the information transmitted from the communication unit 4010 .

[0074] The second client device 410 has a similar configuration to the first client device 400. Like the first client device 400, the second client device 410 has a communication unit 4110, a real environment information acquisition unit 4120, and a synthesis unit 4150.

[0075] Fig. 5 is a flowchart showing the processing of the information processing system according to the second embodiment. The processing of steps S501 to S505 is the processing of the first client device 400, and is realized by the CPU 201 of the first client device 400 expanding a program stored in the ROM 203 into the RAM 202 and executing it. The processing of steps S511 and S512 is the processing of the server device 420, and is realized by the CPU of the server device 420 expanding a program stored in the ROM into the RAM and executing it. For example, when the first user performs an operation (instruction) for displaying a virtual object on the first client device 400, the processing of Fig. 5 starts.

[0076] In step S501, the real environment information acquisition unit 4020 acquires real environment information relating to the environment around the first client device 400 (real space).

[0077] In step S502, the communication unit 4010 transmits the real environment information acquired in step S501 and the ID of the first client device 400 to the virtual object generation unit 4030 of the server device 420 via the network.

[0078] In step S511, the virtual object generation unit 4030 generates a virtual object image based on the virtual object information held by the virtual object holding unit 4070 and the real environment information transmitted from the first client device 400 in step S502. The virtual object information held by the virtual object holding unit 4070 is, for example, virtual object information generated or updated by the second client device 410. The method of generating a virtual object image is the same as in the first embodiment. For example, the virtual object generation unit 4030 adjusts (updates) the virtual object information based on the real environment information, and generates a virtual object image by performing rendering based on the adjusted virtual object information.

[0079] In step S512, the virtual object generation unit 4030 transmits the virtual object image generated in step S511 to the communication unit 4010 of the first client device 400 via the network, in accordance with the ID transmitted from the first client device 400 in step S502.

[0080] In step S503 , the synthesis unit 4050 acquires a real space image obtained by capturing an image of the surroundings of the first client device 400 from the imaging unit 1040 of the first HMD 105 .

[0081] In step S504, the synthesis unit 4050 receives the The synthesis unit 4050 generates a synthetic image by synthesizing (superimposing) the virtual object image transmitted from the synthesis unit 4050 on the real space image acquired in step S503. Then, the synthesis unit 4050 outputs (data of) the generated synthetic image to the first HMD 105. The display unit 1060 of the first HMD 105 displays the synthetic image based on the data of the synthetic image output from the synthesis unit 4050.

[0082] In step S505, the CPU 201 of the first client device 400 determines whether or not to end the process in Fig. 5. If the CPU 201 determines that the process in Fig. 5 should be ended, the process in Fig. 5 ends, and if the CPU 201 determines that the process in Fig. 5 should not be ended, the process returns to step S501.

[0083] As described above, in the second embodiment as well, when the client device generates a virtual object image, the surrounding environment of the client device is taken into consideration. This makes it possible to obtain the same effects as in the first embodiment. Furthermore, according to the second embodiment, the virtual object image is generated (rendered) by the server device. This makes it possible to reduce the processing load on the client device, shorten the processing time of the client device, and reduce the manufacturing cost of the client device.

[0084] (Variation 2-1) As in Modification 1-1, the virtual object generation unit 4030 may not adjust the virtual object information, but may generate a virtual object image based on the virtual object information held by the virtual object holding unit 4070. Then, the virtual object generation unit 4030 may perform image processing on the virtual object image based on the real environment information received from the client device, and transmit the virtual object image after image processing to the client device.

[0085] The virtual object generating unit 4030 may transmit a virtual object image not reflecting the real environment information to the client device without receiving the real environment information from the client device. The virtual object image not reflecting the real environment information is a virtual object image generated based on the virtual object information held by the virtual object holding unit 4070, and is a virtual object image before the above-mentioned image processing. In the first client device 400, the communication unit 4010 may not transmit the real environment information to the server device 420, and may receive a virtual object image not reflecting the real environment information from the server device 420. The first client device 400 may have a processing unit that performs image processing on the virtual object image (the virtual object image not reflecting the real environment information) based on the real environment information acquired by the real environment information acquisition unit 4020. Then, the synthesis unit 4050 may control the display unit 1060 of the first HMD 105 to display the virtual object image after the image processing (the virtual object image reflecting the real environment information).

[0086] In Modification 2-1, when a client device generates a virtual object image, the surrounding environment of the client device is also taken into consideration. This makes it possible to obtain the same effects as in the first embodiment. Furthermore, the server device generates (renders) the virtual object image. This makes it possible to obtain the same effects as in the second embodiment (reducing the processing load on the client device, shortening the processing time of the client device, and reducing the manufacturing cost of the client device).

[0087] (Variation 2-2) The server device may have one GPU or may have multiple GPUs. Multiple GPUs can perform rendering simultaneously (in parallel). Therefore, when the server device has multiple GPUs, the server device can use the multiple GPUs to simultaneously (in parallel) perform rendering of multiple pieces of virtual object information corresponding to multiple client devices. This can reduce processing time. Note that, in a certain client device, when a user performs an operation on a virtual object, When the state of a virtual object changes, information (at least a part of the virtual object information) indicating the changed state is transmitted from the client device to the server device. The server device then reflects the received information in all virtual object information corresponding to the same virtual object (the virtual object corresponding to the received information). In this way, multiple client devices can share the virtual object.

[0088] (Embodiment 3) A description will be given of a third embodiment of the present invention. In the following, a description of the same points as in the first and second embodiments (for example, the same configurations and processes as in the first and second embodiments) will be omitted, and only points different from the first and second embodiments will be described.

[0089] In the first and second embodiments, examples have been described in which a virtual object image is generated based on real-world environment information in order to optically match the real space with the virtual object. In the third embodiment, in order to optically match the real space with the virtual object, a client device performs image processing on a real-world environment image captured around itself based on real-world environment information of another client device, rather than its own real-world environment information.

[0090] Fig. 6 is a functional configuration diagram of an information processing system according to embodiment 3. The information processing system in Fig. 6 includes a server device 620, a first client device 600, a second client device 610, a first HMD 105, and a second HMD 115.

[0091] The server device 620 has an information storage unit 6070. The information storage unit 6070 stores virtual object information, similar to the virtual object storage unit 1070 of the first embodiment. The information storage unit 6070 receives real environment information from a client device via a network. When receiving real environment information from a client device, the information storage unit 6070 also receives an ID (identification information) of the client device, and stores the real environment information and the ID in association with each other.

[0092] The first client device 600 has a communication unit 6010, a real environment information acquisition unit 6020, a virtual object generation unit 6030, a real image processing unit 6035, and a synthesis unit 6050. The real environment information acquisition unit 6020 has a function similar to that of the real environment information acquisition unit 1020 of the first embodiment, and the synthesis unit 6050 has a function similar to that of the synthesis unit 6050 of the first embodiment.

[0093] The communication unit 6010 communicates with the server device 420 via the network, similar to the virtual object communication unit 1010 of the first embodiment. The communication unit 6010 receives virtual object information from the information storage unit 6070 of the server device 620 via the network, similar to the virtual object communication unit 1010 of the first embodiment. Furthermore, when the first user performs an operation on a virtual object and the state of the virtual object changes, the communication unit 6010 transmits information indicating the state after the change (at least a part of the virtual object information) to the information storage unit 6070 of the server device 620 via the network. The virtual object information stored in the information storage unit 6070 is updated based on the information transmitted from the communication unit 6010.

[0094] Furthermore, the communication unit 6010 transmits the real environment information acquired by the real environment information acquisition unit 6020 and the ID of the first client device 600 to the information storage unit 6070 of the server device 620 via the network. Also, the communication unit 6010 receives the real environment information of the second client device 610 from the information storage unit 6070 of the server device 620 via the network.

[0095] The virtual object generation unit 6030 generates a virtual object image based on the virtual object information received by the communication unit 6010. Unlike the virtual object generation unit 1030 of the first embodiment, the virtual object generation unit 6030 uses the real environment information acquired by the real environment information acquisition unit 6020. Not for use.

[0096] The real image processing unit 6035 acquires a real space image obtained by capturing an image of the surroundings of the first client device 600 from the imaging unit 1040 of the first HMD 105. Then, the real image processing unit 6035 performs image processing on the acquired real space image based on the real environment information of the second client device 610 received by the communication unit 6010.

[0097] The second client device 610 has a similar configuration to the first client device 600. The second client device 610 has a communication unit 6110, a real environment information acquisition unit 6120, a virtual object generation unit 6130, a real image processing unit 6135, and a synthesis unit 6150, similar to the first client device 600. However, in the third embodiment, the virtual object generation unit 6130 of the second client device 610 generates a virtual object image based on the virtual object information acquired by the communication unit 6110 and the real environment information acquired by the real environment information acquisition unit 6120. The real image processing unit 6135 of the second client device does not perform image processing on the real space image.

[0098] Fig. 7 is a flowchart showing the processing of the first client device 600. The processing of Fig. 7 is realized by the CPU 201 of the first client device 600 expanding a program stored in the ROM 203 into the RAM 202 and executing it. For example, when the first user performs an operation (instruction) on the first client device 600 to display a virtual object, the processing of Fig. 7 starts.

[0099] In step S701, the real environment information acquisition unit 6020 acquires real environment information relating to the environment around the first client device 600 (real space).

[0100] In step S702, the communication unit 6010 transmits the real-world environment information acquired in step S701 and the ID of the first client device 600 to the information storage unit 6070 of the server device 620 via the network. Similarly, the communication unit 6110 of the second client device 610 transmits the real-world environment information acquired by the real-world environment information acquisition unit 6120 of the second client device 610 and the ID of the second client device 610 to the information storage unit 6070 of the server device 620.

[0101] In step S703, the communication unit 6010 receives the real environment information of the second client device 610 from the information storage unit 6070 of the server device 620 via the network.

[0102] In step S704, the communication unit 6010 receives virtual object information from the information storage unit 6070 of the server device 620 via the network. This virtual object information is, for example, virtual object information generated or updated by the second client device 610.

[0103] In step S705, the virtual object generation unit 6030 generates a virtual object image based on the virtual object information received in step S705 and the real environment information of the second client device 610 received in step S703. The method of generating a virtual object image is the same as in the first embodiment. Note that, if the real environment information of the second client device 610 is reflected in the virtual object information received in step S705, the virtual object generation unit 6030 does not need to use the real environment information of the second client device 610. Also, in step S704, the communication unit 6010 may obtain a virtual object image (a virtual object image reflecting the real environment information of the second client device 610) from the server device 620 as in the second embodiment. In that case, the process of step S705 is not required.

[0104] In step S706, the real image processing unit 6035 detects the surroundings of the first client device 600. A real space image of the surrounding area is acquired from the imaging unit 1040 of the first HMD 105.

[0105] In step S707, the real image processing unit 6035 performs image processing on the real space image acquired in step S706 based on the real environment information of the second client device 610 received in step S703. The image processing is not particularly limited. For example, the real image processing unit 6035 may change the brightness (luminance value) of the real space image according to the brightness of the environment included in the real environment information. To brighten the real space image, the luminance value of the real space image may be increased, and to darken the real space image, the luminance value of the real space image may be decreased. The real image processing unit 6035 may change the luminance value of a part of the real space image according to the position and intensity of the light source included in the real environment information. The real image processing unit 6035 may change the color tone of the real space image (for example, the balance of the R value, the G value, and the B value) according to the color temperature of the light source included in the real environment information.

[0106] In step S708, the synthesis unit 6050 synthesizes (superimposes) the virtual object image generated in step S705 on the real space image after the image processing in step S707 to generate a synthetic image. Then, the synthesis unit 6050 outputs (data of) the generated synthetic image to the first HMD 105. The display unit 1060 of the first HMD 105 displays the synthetic image based on the data of the synthetic image output from the synthesis unit 6050.

[0107] In step S709, the CPU 201 of the first client device 600 determines whether or not to end the process in Fig. 7. If the CPU 201 determines that the process in Fig. 7 is to be ended, the process in Fig. 7 is ended, and if the CPU 201 determines that the process in Fig. 7 is not to be ended, the process returns to step S701.

[0108] As described above, according to the third embodiment, image processing is performed on a real space image captured around a client device based on real environment information of other client devices. This allows multiple users in different locations to view a virtual object conveniently (in an environment equivalent to the environment around a specific client device). If a virtual object image is generated in which the virtual object is optically consistent with the environment around a specific client device, multiple users can view the virtual object without feeling uncomfortable.

[0109] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the above-described configurations.

[0110] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0111] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A client device connected to a network, A receiving means for receiving first information, which is information on a virtual object, via the network; An acquisition means for acquiring second information which is information regarding the surrounding environment of the client device; a generating means for generating an image of the virtual object based on the first information and the second information; a control means for controlling the image of the virtual object generated by the generation means to be displayed on a display unit; 13. An information processing device comprising: (Configuration 2) The client device further includes an image acquisition unit for acquiring a real space image obtained by capturing an image of the surroundings of the client device, The control means controls the display unit to display a composite image in which the image of the virtual object is superimposed on the real space image. 2. The information processing device according to configuration 1. (Configuration 3) The second information includes exposure information of an imaging unit that captures an image of the surroundings of the client device, or the real space image. 3. The information processing device according to configuration 2. (Configuration 4) The acquisition means acquires the second information at a frequency lower than a frame rate of the real space image. 4. The information processing device according to configuration 2 or 3. (Configuration 5) The second information includes at least one of a position of a light source in the environment, an intensity of the light source, a color temperature of the light source, an environment map indicating the environment, and a brightness of the environment. 5. The information processing device according to any one of configurations 1 to 4. (Configuration 6) The first information includes the same type of information as the second information. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) The generating means adjusts the first information based on the second information, and generates an image of the virtual object based on the adjusted first information. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) the generation means generates an image of the virtual object based on the first information, and performs image processing on the image of the virtual object based on the second information; The control means controls the display unit to display the image of the virtual object after the image processing. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) A server device connected to a network, a receiving means for receiving, via the network, second information which is information relating to the surrounding environment of a client device connected to the network; a control means for controlling the first client device to give a predetermined notification to a user of the first client device when a difference between the second information of the first client device and the second information of the second client device is greater than a threshold value; 13. An information processing device comprising: (Configuration 10) A server device connected to a network, a receiving means for receiving, via the network, second information which is information relating to the surrounding environment of a client device connected to the network; A generating means for generating an image of a virtual object based on the second information; a transmission means for transmitting an image of the virtual object generated by the generation means to the client device via the network; 13. An information processing device comprising: (Configuration 11) A client device connected to a network, A receiving means for receiving an image of a virtual object via the network; An acquisition means for acquiring second information which is information regarding the surrounding environment of the client device; A processing means for performing image processing on an image of the virtual object based on the second information; a control means for controlling the image of the virtual object after the image processing to be displayed on a display unit; 13. An information processing device comprising: (Configuration 12) a first client device connected to a network, A receiving means for receiving, via the network, information or an image of a virtual object and second information which is information relating to the surrounding environment of a second client device connected to the network; An acquisition means for acquiring a real space image obtained by capturing an image of the surroundings of the first client device; a processing means for performing image processing on the real space image based on the second information; a control means for controlling the display unit to display a composite image obtained by superimposing an image of the virtual object generated based on information of the virtual object on the real space image after the image processing; 13. An information processing device comprising: (method) A method for controlling a client device connected to a network, comprising: receiving first information, which is information about a virtual object, via the network; obtaining second information, the second information being information regarding an environment surrounding the client device; generating an image of the virtual object based on the first information and the second information; A step of controlling the display unit to display an image of the generated virtual object; A control method comprising the steps of: (program) 13. A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0112] 100, 400, 600: First client device 105: First HMD 110, 410, 610: Second client device 115: Second HMD 120,420,620,821,822:Server equipment 201: CPU 1010, 1110: Virtual object communication unit 1020, 1120: Real environment information acquisition unit 1030, 1130: Virtual object generation unit 1040, 1140: Imaging unit 1050,1150:Composition section 1060,1160:Display section 1070: Virtual object holding unit 4010, 4110: Communication unit 4020, 4120: Real-world environment information acquisition unit 4030: Virtual object generation unit 4050, 4150: Synthesis unit 4070: Virtual object holding unit 6010, 6110: Communication unit 6020, 6120: Real-world environment information acquisition unit 6030, 6130: Virtual object generation unit 6035, 6135: Real image processing unit 6050,6150:Composition section 6070:Information holding section 8071: Virtual object holding unit 8072: Virtual object holding unit

Claims

1. A first client device connected to a network, comprising: receiving means for receiving first information, which is information on a virtual object, via the network; acquiring means for acquiring second information, which is information on the environment around the first client device; generating means for generating a first image of the virtual object based on the first information and the second information; control means for controlling to display the first image of the virtual object generated by the generating means on a first display unit; and for a second client device whose surrounding environment is different from that of the first client device, a second image of the virtual object is generated based on the first information and information on the environment around the second client device, and the second image is displayed on a second display unit by the second client device. An information processing apparatus characterized by the above.

2. The first client device further includes image acquisition means for acquiring a real space image obtained by imaging the surroundings of the first client device, and the control means controls to display on the first display unit a composite image in which the first image of the virtual object is superimposed on the real space image. The information processing apparatus according to claim 1, characterized by the above.

3. The second information includes exposure information of an imaging unit that images the surroundings of the first client device or the real space image. The information processing apparatus according to claim 2, characterized by the above.

4. The acquiring means acquires the second information at a frequency lower than the frame rate of the real space image. The information processing apparatus according to claim 2, characterized by the above.

5. The second information includes at least any one of the position of a light source in the environment, the intensity of the light source, the color temperature of the light source, an environment map indicating the environment, and the brightness of the environment. The information processing apparatus according to claim 1, characterized by the above.

6. The first information includes information of the same type as the second information. The information processing apparatus according to claim 1, characterized by the above.

7. The generating means adjusts the first information based on the second information, and generates the first image of the virtual object based on the adjusted first information. The information processing apparatus according to claim 1, characterized by the above.

8. The generating means generates the first image of the virtual object based on the first information, and performs image processing on the first image of the virtual object based on the second information. The control means controls to display the first image of the virtual object after the image processing on the first display unit. The information processing apparatus according to claim 1, characterized in that.

9. The receiving means is a communication means for receiving the first information from a server device via the network, The generating means changes the state of the virtual object according to an instruction from the user, The communication means transmits, via the network, third information which is information about the virtual object after the state is changed, to the server device. The information processing apparatus according to claim 1, characterized in that.

10. When the difference between the environment around the first client device and the environment around the second client device is greater than a threshold value, the control means controls to give a predetermined notification to the user of the first client device. The information processing apparatus according to claim 1, characterized in that.

11. When the difference between the environment around the first client device and the environment around the second client device is equal to or less than the threshold value, the generating means generates the first image of the virtual object based on the first information without using the second information, and when the difference between the environment around the first client device and the environment around the second client device is greater than the threshold value, the generating means generates the first image of the virtual object based on the first information and the second information. The information processing apparatus according to claim 1, characterized in that.

12. A first client device connected to a network, Receiving means for receiving, via the network, first information which is information about a virtual object and second information which is information about the environment around a second client device different from the first client device, Acquiring means for acquiring a real-space image obtained by imaging the surroundings of the first client device, Processing means for performing image processing on the real-space image based on the second information, Generating means for generating an image of the virtual object based on the first information, Control means for controlling to display, on a display unit, a composite image in which the image of the virtual object is superimposed on the real-space image after the image processing. An information processing apparatus, characterized by comprising.

13. A server device connected to a network, Receiving means for receiving, via the network, information about the environment around a first client device connected to the network. Generating means for generating a first image of a virtual object based on the information; Transmitting means for transmitting the first image of the virtual object generated by the generating means to the first client device via the network; and; For a second client device whose surrounding environment is different from that of the first client device, a second image of the virtual object is generated based on information regarding the surrounding environment of the second client device, and the second image is displayed on a second display unit by the second client device. An information processing apparatus characterized by this.

14. A control method for a first client device connected to a network, comprising: Receiving, via the network, first information which is information on a virtual object; Obtaining second information which is information regarding the surrounding environment of the first client device; Generating a first image of the virtual object based on the first information and the second information; Controlling to display the generated first image of the virtual object on a first display unit; and; For a second client device whose surrounding environment is different from that of the first client device, a second image of the virtual object is generated based on the first information and information regarding the surrounding environment of the second client device, and the second image is displayed on a second display unit by the second client device. A control method characterized by this.

15. A program for causing a computer to function as each means of the information processing apparatus according to any one of Claims 1 to 13.

Images