Information processing system, method for controlling information processing system, and program

Abstract

To enable an image obtained by synthesizing a plurality of images to be provided without allowing at least one device to have information on all three-dimensional objects that appear in the image obtained by synthesizing the plurality of images.SOLUTION: An information processing system includes a first information processing device for storing information on a first object, and a second information processing device for storing information on a second object different from the first object. The first information processing device has first generation means for generating a first image that draws the first object on the basis of information on a position attitude of the first object and the information on the first object, the second information processing device has second generation means for generating a second image that draws the second object on the basis of the information on the position attitude of the first object and the information on the second object, and the first information processing device further has first synchronization means for generating an image obtained by synchronizing the first image with the second image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, a control method for the information processing system, and a program.

Background Art

[0002] In recent years, the development of Virtual Reality (VR) and Mixed Reality (MR) systems aimed at seamless connection between the real space and the virtual space has been actively carried out. Also, the development of a system in which a plurality of users share the state of virtual objects via a network has been carried out. According to such a system, the work (operation) performed by a certain user on a virtual object is also reflected in the virtual object viewed by other users, and a plurality of users can cooperate to perform work on the virtual object.

[0003] Such a system draws a virtual space based on information such as the information on the position and orientation of the user and the information on the virtual object. The computing resources of the information processing device used for drawing the virtual space are finite. Therefore, when performing high-quality drawing and drawing a large number of virtual objects, it may be difficult to perform drawing using a single information processing device.

[0004] As a solution to such problems, in Patent Document 1, a plurality of three-dimensional objects are classified, and different drawing processing units perform drawing for each classified three-dimensional object. At that time, each pixel has depth information in pixel units, and the three-dimensional objects drawn by the plurality of drawing processing units are integrated based on the depth information. Patent Document 2 discloses a system in which, based on an instruction according to a user's operation, a plurality of objects are classified into two categories, and the objects belonging to each category are drawn by a terminal device and a server device and then synthesized by the terminal device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] In the above-described technology, synthesis can be achieved by having all the three-dimensional objects used in the synthesis held in one place or having the same three-dimensional object held in multiple places. That is, at least one device needs to have the information of all the three-dimensional objects appearing in the image obtained by synthesizing a plurality of images.

[0007] An object of the present invention is to provide an image obtained by synthesizing a plurality of images without at least one device having the information of all the three-dimensional objects appearing in the image. [Means for Solving the Problems]

[0008] One aspect of the present invention is an information processing system including a first information processing device that stores information of a first object and a second information processing device that stores information of a second object different from the first object, wherein the first information processing device has first generation means for generating a first image depicting the first object based on information on the position and orientation of a first object and the information of the first object and the second information processing device has second generation means for generating a second image depicting the second object based on information on the position and orientation of the first object and the information of the second object, and the first information processing device further has first synthesis means for generating an image obtained by synthesizing the first image and the second image. The information processing system is characterized by the above.

[0009] One aspect of the present invention is A control method for an information processing system having a first information processing device that stores information of a first object and a second information processing device that stores information of a second object different from the first object, In the first information processing device, a first generation step of generating a first image depicting the first object based on information on the position and orientation of a first object and information on the first object; In the second information processing device, a second generation step of generating a second image depicting the second object based on information on the position and orientation of the first object and information on the second object; In the first information processing device, a first synthesis step of generating an image obtained by synthesizing the first image and the second image; A control method for an information processing system, characterized by comprising the above.

Advantages of the Invention

[0010] According to the present invention, it is possible to provide an image in which a plurality of images are synthesized without at least one device having information on all three-dimensional objects appearing in the synthesized image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Best Mode for Carrying Out the Invention

[0012] Hereinafter, with reference to the accompanying drawings, embodiments for carrying out the present invention will be described in detail. It should be noted that the embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. The present invention is not limited to the following embodiments.

[0013] <Embodiment 1> As shown in FIG. 1, the information processing system according to Embodiment 1 includes a display device 100, an information processing device 110, and an information processing device 120. In Embodiment 1, the information processing device 110 and the information processing device 120 may each be a server device, a PC (Personal Computer), or may have a cloud form.

[0014] Note that the display device 100 is, for example, an HMD (Head Mounted Display), which is a type of head-mounted display device. In Embodiment 1, the display device 100 may be another type of head-mounted display device. For example, the display device 100 may be a handheld display device. A handheld display device is a type of head-mounted display device that a user holds in their hand and mounts (applies) on their head. A handheld display device is, for example, a smartphone or a tablet terminal.

[0015] ​The display device 100 may be smart glasses (AR (Augmented Reality) glasses). Smart glasses are a type of head-mounted display device. The display device 100 may be a head-mounted display device that allows the user to view an image with both eyes, or it may be a head-mounted display device that allows the user to view an image with one eye. The display device 100 may be a smartphone mounted on a head-mounted adapter (e.g., VR (Virtual Reality) goggles). The display device 100 may be a display device other than a head-mounted display device (e.g., a stationary display device).

[0016] The display device 100 may be a device in either the video see-through format or the optical see-through format. In the video see-through format, for example, an image of the real space (the outside world) captured by an imaging device (e.g., a video camera) is displayed in real time on a display surface (a display surface that does not transmit light from the real space). Then, graphics (e.g., an image of a virtual object) are superimposed (composited) on the image of the real space and displayed. In this case, the user cannot directly view the real space, but can indirectly view the real space and view the graphics superimposed on the image of the real space by viewing the displayed image.

[0017] In the optical see-through format, for example, graphics (e.g., an image of a virtual object) are displayed on a display surface (a display surface that transmits light from the real space (the outside world)). In this case, the user can directly view the real space (the outside world) through the display surface and view the graphics displayed on the display surface.

[0018] The information processing device 110, the display device 100, and the information processing device 120 are connected, for example, so as to be able to communicate data with each other. The connection between the information processing device 110, the display device 100, and the information processing device 120 may be realized by either wired or wireless means. Also, all or part of the information processing device 110 may be arranged inside the housing of the display device 100.

[0019] The display device 100 includes an imaging unit 1010 and a display unit 1020 .

[0020] The imaging unit 1010 captures images of the real space continuously in time series, and outputs the captured images of the real space to the information processing device 110. The imaging unit 1010 may include a stereo camera. The stereo camera has, for example, two cameras fixed so as to be able to capture images of the real space in the user's line of sight from positions close to the user's eyes.

[0021] The display unit 1020 displays the MR image output from the information processing device 110. The display unit 1020 may include a display arranged corresponding to the left eye of the user and a display arranged corresponding to the right eye of the user. In this case, the MR image for the left eye is displayed on the display corresponding to the left eye of the user, and the MR image for the right eye is displayed on the display corresponding to the right eye of the user.

[0022] The information processing device 110 includes a position and orientation acquisition unit 1110 , an image generation unit 1120 , an image synthesis unit 1130 , a communication unit 1140 , and a data storage unit 1150 .

[0023] The position and orientation acquisition unit 1110 acquires the position and orientation of the image capture unit 1010 (display device 100) in the world coordinate system. The position and orientation (position and orientation) are calculated. Specifically, the position and orientation acquisition unit 1110 extracts markers assigned to the world coordinate system from an image of the real space captured by the imaging unit 1010. Then, the position and orientation acquisition unit 1110 calculates the position and orientation of the imaging unit 1010 in the world coordinate system based on the position and orientation of the markers. The position and orientation acquisition unit 1110 outputs information on the calculated position and orientation of the imaging unit 1010 to the data storage unit 1150.

[0024] Note that there are various methods for calculating the position and orientation of the imaging unit 1010, and the method is not limited to the above-described method. For example, the position and orientation acquisition unit 1110 may perform Simultaneous Localization and Mapping (SLAM) processing based on feature points appearing in the image to obtain the position and orientation of the imaging unit 1010 or the position and orientation of the individual coordinate system. Further, when the display device 100 is provided with a sensor whose relative position and orientation with respect to the imaging unit 1010 are known, the position and orientation acquisition unit 1110 may obtain the position and orientation of the imaging unit 1010 by converting the measurement value by the sensor based on the relative position and orientation. Further, the position and orientation acquisition unit 1110 may obtain the position and orientation of the imaging unit 1010 using a motion capture system.

[0025] The image generation unit 1120 generates a virtual image (hereinafter referred to as the "first virtual image") obtained by rendering the virtual space based on the data of the virtual space stored in the data storage unit 1150. The data of the virtual space includes data related to one or more virtual objects (virtual objects) constituting the virtual space. Further, the data of the virtual space includes data related to a real approximation object in which the three-dimensional shape information of a real object in the real space is incorporated into the virtual space, and data related to a light source that irradiates the virtual space. Then, the image generation unit 1120 generates an image of the virtual space viewed from a viewpoint corresponding to the position and orientation calculated by the position and orientation acquisition unit 1110 as the first virtual image. For this reason, the image generation unit 1120 generates, for example, a first virtual image in which the first virtual object is drawn in the virtual space based on the data of the virtual space including at least the data of the first virtual object and the position and orientation of the imaging unit 1010. The first virtual object may be a three-dimensional object representing an object actually arranged in the real space. Note that since the technology for generating an image of the virtual space viewed from a viewpoint corresponding to a predetermined position and orientation is a well-known technology, a detailed description of this technology is omitted.

[0026] In addition, the image generation unit 1120 generates a depth image corresponding to the generated first virtual image (hereinafter referred to as the "first depth image"). The depth image is an image in which depth information indicating the distance to the object shown (distance in a specific direction) is set for each pixel. Therefore, the depth information indicating the distance to the object shown in the pixel at the coordinates (x, y) of the first depth image (distance from the reference position to the object) is set for the pixel at the coordinates (x, y) of the first virtual image. Then, the image generation unit 1120 outputs the first virtual image and the first depth image to the image composition unit 1130.

[0027] The image composition unit 1130 composes the first virtual image and the virtual image generated by the information processing device 120 (hereinafter referred to as the "second virtual image"). When composing, the image composition unit 1130 compares the first depth image with the depth image corresponding to the second virtual image (hereinafter referred to as the "second depth image") and determines which of the first virtual image and the second virtual image is located in the front in terms of pixels. Then, the image composition unit 1130 composes the two virtual images so as to display the pixels of the virtual image located in the front. After that, the image composition unit 1130 composes the image obtained by composing the two virtual images and the image of the real space captured by the imaging unit 1010 to generate an MR image. The image composition unit 1130 outputs the generated MR image to the display unit 1020.

[0028] The communication unit 1140 transmits the position and orientation information of the imaging unit 1010 calculated by the position and orientation acquisition unit 1110 to the information processing device 120. In addition, the communication unit 1140 receives the second virtual image and the second depth image from the information processing device 120.

[0029] As described above, the data storage unit 1150 stores various types of information. The data storage unit 1150 includes a RAM (Random Access Memory) or a hard disk drive device, etc. Note that the data storage unit 1150 also stores information described as known information.

[0030] The information processing apparatus 120 includes a communication unit 1210, an image generation unit 1220, and a data storage unit 1230.

[0031] The communication unit 1210 receives information on the position and orientation of the imaging unit 1010 from the information processing apparatus 110. Then, the communication unit 1210 outputs the information on the position and orientation of the imaging unit 1010 to the data storage unit 1230. Also, the communication unit 1210 transmits the second virtual image and the second depth image generated by the image generation unit 1220 to the information processing apparatus 110.

[0032] The image generation unit 1220 generates a second virtual image based on the data of the virtual space stored in the data storage unit 1230. The data of the virtual space includes data related to a light source that irradiates the virtual space and data related to one or more virtual objects (one or more virtual objects different from the virtual objects stored in the information processing apparatus 110) that constitute the virtual space. At this time, the data of the virtual space does not include the data of the virtual objects used to generate the first virtual image. The data of the virtual objects stored in the information processing apparatus 110 is not stored in the information processing apparatus 120. On the other hand, the data of the virtual objects stored in the information processing apparatus 120 is not stored in the information processing apparatus 110. Then, the image generation unit 1220 generates a virtual image (virtual image) viewed from a viewpoint corresponding to the position and orientation of the imaging unit 1010 as the second virtual image. For this reason, the image generation unit 1220 generates, for example, a second virtual image in which a second virtual object is drawn in the virtual space based on the data of the virtual space including at least the data of the second virtual object different from the first virtual object described above and the position and orientation of the imaging unit 1010. The second virtual object may be a three-dimensional object that represents an object actually arranged in the real space.

[0033] Also, the image generation unit 1220 generates a second depth image corresponding to the second virtual image. Then, the image generation unit 1220 outputs the second virtual image and the second depth image to the data storage unit 1230.

[0034] The data storage unit 1230 stores various information. The data storage unit 1230 is composed of a RAM or a hard disk drive device, etc. Note that the data storage unit 1230 also stores information described as known information.

[0035] Referring to the flowchart of FIG. 2, the information processing apparatus 110 and the information processing apparatus 120 will be described in terms of the process of generating an MR image and outputting it to the display device 100.

[0036] In step S2010, the position and orientation acquisition unit 1110 calculates the position and orientation of the imaging unit 1010.

[0037] In step S2020, the communication unit 1140 transmits the information on the position and orientation of the imaging unit 1010 to the information processing apparatus 120.

[0038] In step S2030, the image generation unit 1120 generates a first virtual image and a first depth image based on the position and orientation of the imaging unit 1010.

[0039] In step S2040, simultaneously with the process of step S2030, the image generation unit 1220 generates a second virtual image and a second depth image based on the position and orientation of the imaging unit 1010.

[0040] In step S2050, the communication unit 1210 transmits the second virtual image and the second depth image to the information processing apparatus 110.

[0041] In step S2060, the image synthesis unit 1130 synthesizes the first virtual image, the second virtual image, and the real image acquired by the imaging unit 1010. Thereby, the image synthesis unit 1130 generates an MR image.

[0042] FIGS. 3A to 3G are diagrams showing an example of a virtual image, a depth image, and an MR image.

[0043] The real image 3010 is an image representing the real space. The real image 3010 is an image captured by the imaging unit 1010 of the real space.

[0044] The virtual objects 3020 and 3030 as shown in FIG. 3A are 3D objects stored in the data storage unit 1150. The area 3040 as shown in FIG. 3B is an area in the first depth image generated by the image generation unit 1120 where the virtual object 3020 is drawn. The area 3050 is an area in the first depth image where the virtual object 3030 is drawn.

[0045] The virtual objects 3060 and 3070 as shown in FIG. 3C are 3D objects stored in the data storage unit 1230. The area 3080 as shown in FIG. 3D is an area in the second depth image where the virtual object 3060 is drawn. The area 3090 is an area in the second depth image where the virtual object 3070 is drawn.

[0046] FIG. 3A shows the first virtual image generated by the image generation unit 1120. FIG. 3B shows the first depth image generated by the image generation unit 1120. The depth image is a monochrome image. In the depth image, a value corresponding to the distance between the reference position (the viewpoint of the display device 100) and the virtual object is set for each pixel. In the depth image, pixels closer to the reference position have values closer to black, and pixels farther from the reference position have values closer to white. The distance between the virtual object 3030 and the reference position is shorter than the distance between the virtual object 3020 and the reference position. And there are no objects to be drawn outside the areas of the virtual objects 3030 and 3020. For this reason, in FIG. 3B, the area 3050 is an area with a stronger black tone compared to the area 3040, and the other areas are white areas.

[0047] FIG. 3C shows the second virtual image generated by the image generation unit 1220. FIG. 3D shows the second depth image generated by the image generation unit 1220. The distance between the virtual object 3070 and the reference position is shorter than the distance between the virtual object 3060 and the reference position. There is no object to be drawn outside the area between the virtual object 3070 and the virtual object 3060. Therefore, in FIG. 3D, the area 3090 is an area with a stronger black color compared to the area 3080, and the other areas are white areas.

[0048] FIG. 3E is an image obtained by the image synthesis unit 1130 synthesizing the first virtual image shown in FIG. 3A and the second virtual image shown in FIG. 3C based on the front - rear relationship of the virtual objects determined according to the first depth image shown in FIG. 3B and the second depth image shown in FIG. 3D. The image synthesis unit 1130 compares the first depth image and the second depth image on a pixel - by - pixel basis and determines that the pixel closer to the reference position (that is, the pixel with a value closer to black) exists in the front. The image synthesis unit 1130 uses each pixel of the virtual image corresponding to the depth information of the pixel determined to exist in the front for the synthesis.

[0049] FIG. 3F shows the real image 3010 acquired by the imaging unit 1010. And, as shown in FIG. 3G The MR image is an image obtained by the image synthesis unit 1130 synthesizing the real image 3010 and the synthesized image shown in FIG. 3E.

[0050] According to Embodiment 1, each of the two information processing devices generates a virtual image according to the data of different virtual objects. Then, one information processing device generates a synthesized image by synthesizing the two virtual images. Therefore, it is not necessary for one information processing device to store the data of all the virtual objects appearing in the synthesized image. Thus, for example, it is also possible to hold the data of virtual objects that are not desired to be shared with other information processing devices only in one information processing device. According to this, it is also possible to prevent a company's technical secrets from being known by other companies.

[0051] (Modification Example 1) In Embodiment 1, it was explained that the data related to the light source that irradiates the virtual space is data held by both the information processing device 110 and the information processing device 120. And the data related to the light source that irradiates the virtual space is common data in the information processing device 110 and the information processing device 120. However, the data related to the light source may be held by only one of the information processing device 110 and the information processing device 120. The data related to the light source held by the information processing device 110 may be transmitted to the information processing device 120, and the data transmitted from the information processing device 110 may be used for image generation performed by the information processing device 120. The data related to the light source held by the information processing device 120 may be transmitted to the information processing device 110, and the data transmitted from the information processing device 120 may be used for image generation performed by the information processing device 110. Also, the information processing device 110 and the information processing device 120 may share the data related to the light source with each other, and all or part of the data may be used for image generation of the information processing device 110 or the information processing device 120.

[0052] <Embodiment 2> In Embodiment 2, the information processing system has a plurality of display devices.

[0053] FIG. 4 is a diagram showing a configuration example of the information processing system according to Embodiment 2. As shown in FIG. 4, the information processing system according to Embodiment 2 includes a display device 100, a display device 400, an information processing device 410, and an information processing device 420. In FIG. 4, the same parts as those in FIG. 1 are denoted by the same reference numerals, and the description thereof is omitted.

[0054] The display device 400 includes an imaging unit 4010 and a display unit 4020.

[0055] The imaging unit 4010 sequentially and continuously captures the real space, and outputs an image of the captured real space to the information processing device 420. The imaging unit 4010 may include a stereo camera. The stereo camera has, for example, two cameras fixed so as to be able to capture the real space in the viewing direction of the user from the position of the user's eyes (viewpoint position).

[0056] The display unit 4020 displays the MR image output from the information processing device 420. The display unit 4020 may include a display arranged corresponding to the user's left eye and a display arranged corresponding to the user's right eye. In this case, the left-eye MR image is displayed on the display corresponding to the user's left eye, and the right-eye MR image is displayed on the display corresponding to the user's right eye.

[0057] Here, the information processing device 410 is connected so as to be capable of data communication with the display device 100 and the information processing device 420. The display device 400 and the information processing device 420 are connected so as to be capable of data communication with each other. Also, all or part of the information processing device 410 may be arranged inside the housing of the display device 100. All or part of the information processing device 420 may be arranged inside the housing of the display device 400. In addition, the display device 100, the display device 400, the information processing device 410, and the information processing device 420 may exist in the same space or in separate spaces.

[0058] The information processing device 410 includes a position and orientation acquisition unit 1110, an image generation unit 1120, an image composition unit 1130, a data storage unit 1150, a second image generation unit 4110, and a communication unit 4120. Note that the second image generation unit 4110 may be included in the image generation unit 1120.

[0059] The second image generation unit 4110 generates a virtual image (hereinafter referred to as the "third virtual image") based on the data of the virtual space stored in the data storage unit 1150. The data of the virtual space includes data related to each virtual object constituting the virtual space, data related to a realistic approximation object obtained by incorporating the three-dimensional shape information of a real object acquired from the real space into the virtual space, and data related to a light source that irradiates the virtual space. Here, the data of the virtual space (virtual object) used to generate the third virtual image is the same as the data of the virtual space used to generate the first virtual image. That is, the same virtual object (for example, the first virtual object) is drawn in the first virtual image and the third virtual image. Then, the second image generation unit 4110 generates, as the third virtual image, an image of the virtual space (virtual object) viewed from a viewpoint corresponding to the position and orientation of the imaging unit 4010 calculated by the position and orientation acquisition unit 4210. Further, the second image generation unit 4110 generates a depth image corresponding to the third virtual image (hereinafter referred to as the "third depth image").

[0060] The communication unit 4120 transmits the information on the position and orientation of the imaging unit 1010 to the information processing device 420. The communication unit 4120 further transmits the third virtual image and the third depth image generated by the second image generation unit 4110 to the information processing device 420. Also, the communication unit 4120 receives the second virtual image and the second depth image generated by the image generation unit 1220, and the information on the position and orientation of the imaging unit 4010 calculated by the position and orientation acquisition unit 4210.

[0061] The information processing device 420 includes an image generation unit 1220, a data storage unit 1230, a position and orientation acquisition unit 4210, a second image generation unit 4220, an image synthesis unit 4230, and a communication unit 4240. Note that the second image generation unit 4220 may be included in the image generation unit 1220.

[0062] The position and orientation acquisition unit 4210 calculates the position and orientation of the imaging unit 4010 in the world coordinate system. The position and orientation acquisition unit 4210 can calculate the position and orientation of the imaging unit 4010 by the same method as the position and orientation acquisition unit 1110.

[0063] The second image generation unit 4220 generates a virtual image (hereinafter referred to as the "fourth virtual image") for display on the display device 400 based on the data of the virtual space stored in the data storage unit 1230. Here, the data of the virtual space (virtual object) used to generate the fourth virtual image is the same as the data of the virtual space used to generate the second virtual image. That is, the same virtual object (for example, the second virtual object) is drawn in the second virtual image and the fourth virtual image. The second image generation unit 4220 generates an image of the virtual space (virtual object) viewed from the viewpoint corresponding to the position and orientation calculated by the position and orientation acquisition unit 4210 as the fourth virtual image. Similar to Embodiment 1, the data of the virtual object stored in the information processing device 410 is not stored in the information processing device 420. On the other hand, the data of the virtual object stored in the information processing device 420 is not stored in the information processing device 410.

[0064] Also, the second image generation unit 4220 generates a depth image corresponding to the fourth virtual image (hereinafter referred to as the "fourth depth image"). Then, the second image generation unit 4220 outputs the fourth virtual image and the fourth depth image to the image synthesis unit 4230.

[0065] The image synthesis unit 4230 synthesizes the third virtual image and the fourth virtual image. The image synthesis unit 4 230 compares the third depth image and the fourth depth image during synthesis to determine which of the third virtual image and the fourth virtual image is located in front at each pixel. Then, the image synthesis unit 4230 synthesizes the third virtual image and the fourth virtual image so that the pixels of the virtual image determined to be located in front are displayed in front. Then, the image synthesis unit 4230 synthesizes the image obtained by synthesizing the two virtual images and the image of the real space captured by the imaging unit 4010 to generate an MR image. After that, the image synthesis unit 4230 outputs the generated MR image to the display unit 4020.

[0066] The communication unit 4240 transmits information on the position and orientation of the imaging unit 4010, the second virtual image, and the second depth image to the information processing device 410. Further, the communication unit 4240 receives information on the position and orientation of the imaging unit 1010, the third virtual image, and the third depth information from the information processing device 410.

[0067] Referring to the flowchart of FIG. 5, the processes performed by the information processing device 410 and the information processing device 420 to generate an MR image will be described. In FIG. 5, the same parts as those in FIG. 2 are denoted by the same reference numerals, and the description thereof will be omitted. Note that, as long as there is no contradiction, the order of the processes in the flowchart can be arbitrarily changed.

[0068] In step S5010, simultaneously with the process of step S2010, the position and orientation acquisition unit 4210 calculates the position and orientation of the imaging unit 4010.

[0069] In step S5020, the communication unit 4240 transmits information on the position and orientation of the imaging unit 4010 to the information processing device 410. Note that the process of step S5020 may be executed before the process of step S2020.

[0070] In step S5030, the second image generation unit 4110 generates a third virtual image and a third depth image based on the information on the position and orientation of the imaging unit 4010.

[0071] In step S5040, simultaneously with the process of step S5030, the second image generation unit 4220 generates a fourth virtual image and a fourth depth image based on the information on the position and orientation of the imaging unit 4010.

[0072] In step S5050, the communication unit 4120 transmits the third virtual image and the third depth image to the information processing device 420. Note that the process of step S5050 may be executed before the process of step S2050.

[0073] In step S5060, the image synthesis unit 4230 synthesizes the third virtual image, the fourth virtual image, and the real image acquired by the imaging unit 4010. In this way, the image synthesis unit 4230 generates an MR image.

[0074] 6A to 6G are diagrams showing examples of a virtual image, a depth image, and an MR image. In Fig. 6A to 6G, the same parts as those in Fig. 3A to 3G are given the same numbers, and their description will be omitted. In this example, the display device 400 is located on the rear side (opposite side) of the display device 100.

[0075] The real image 6010 is an image representing the real space. The real image 6010 is an image of the real space captured by the imaging unit 4010.

[0076] The area 6020 is an area in which the virtual object 3020 is drawn in the third depth image generated by the second image generation unit 4110. The area 6030 is an area in which the virtual object 3020 is drawn in the third depth image generated by the second image generation unit 4110. An area 6040 is an area in the fourth depth image generated by the second image generation unit 4220 where a virtual object 3030 is drawn. An area 6040 is an area in the fourth depth image generated by the second image generation unit 4220 where a virtual object 3060 is drawn. An area 6050 is an area in the fourth depth image generated by the second image generation unit 4220 where a virtual object 3070 is drawn.

[0077] Fig. 6A shows a third virtual image generated by the second image generation section 4110. Fig. 6B shows a third depth image generated by the second image generation section 4110. The distance between the virtual object 3020 and the reference position (the viewpoint of the display device 400) is shorter than the distance between the virtual object 3030 and the reference position. No objects to be drawn exist in other regions. For this reason, in Fig. 6B, the region 6020 is a darker region than the region 6030, and the other regions are white.

[0078] FIG. 6C shows a fourth virtual image generated by the second image generation unit 4220. FIG. 6D shows a fourth depth image generated by the second image generation unit 4220. The distance between the virtual object 3060 and the reference position (the viewpoint of the display device 400) is shorter than the distance between the virtual object 3070 and the reference position. In other regions, there are no objects to be drawn. Therefore, in FIG. 6D, the region 6040 is a region with a stronger black color than the region 6050, and the other regions are white regions.

[0079] FIG. 6E is an image in which the image synthesis unit 4230 determines the front-rear relationship of virtual objects by comparing the third depth image and the fourth depth information, and synthesizes the third virtual image and the fourth virtual image. The image synthesis unit 4230 compares the third depth image and the fourth depth information on a pixel-by-pixel basis, and determines that the pixels on the side closer to the reference position (that is, the pixels having a value closer to black) are in the front. The image synthesis unit 4230 uses each pixel of the virtual image corresponding to the depth information determined to be in the front for generating the synthesized image.

[0080] FIG. 6F is a real image 6010 obtained by the imaging unit 4010 imaging the real space. The MR image shown in FIG. 6G is an image obtained by the image synthesis unit 4230 synthesizing the real image 6010 and the synthesized image shown in FIG. 6E.

[0081] Therefore, according to the second embodiment, even when a plurality of users view the same MR space from different viewpoints, a plurality of information processing apparatuses can generate different virtual images. That is, even in such a case, there is no need to store the data of all the virtual objects appearing in the synthesized image in one information processing apparatus. Therefore, for example, the data of virtual objects that are not desired to be shared with other information processing apparatuses can be held only in one information processing apparatus.

[0082] (Modification 2) In Embodiment 2, an example in which the information processing system has two display devices has been described, but the information processing system may have three or more display devices. In this case, after the information processing device 410 and the information processing device 420 each generate a virtual image according to the information on the position and orientation of each display device, the MR image can be displayed on the display device by combining the two virtual images.

[0083] (Modification Example 3) The existence of a three-dimensional reconstruction method is known as a known technique. By using the three-dimensional reconstruction method, it is possible to restore (estimate) the shape (information) of a virtual object based on an image depicting the virtual object that is a three-dimensional object. That is, the virtual object managed in the information processing device 410 can be restored in the information processing device 420. For this reason, there may be cases where an administrator or organization wants to prevent the disclosure of all or part of the information on the shape of the virtual object to different information processing devices, but the prevention of disclosure cannot be achieved in the above-described Embodiments 1 and 2.

[0084] Therefore, in Modification Example 3, specific conditions for determining whether or not to generate a virtual image may be provided. Hereinafter, the case where the image generation unit 1120 determines whether or not to generate a virtual image will be described. Note that the second image generation unit 4110, the image generation unit 1220, and the second image generation unit 4220 also determine whether or not to generate a virtual image in the same manner as the image generation unit 1120.

[0085] The specific condition is a condition for determining whether it is appropriate to provide a virtual image (information represented in the virtual image) to the information processing apparatus that generates the MR image. The specific condition is, for example, a condition that the position and orientation of the imaging unit 1010 in the world coordinate system are a specific position and orientation (for example, a position and orientation facing downward). Alternatively, the specific condition may be a condition that the distance or angle formed between each virtual object and the display device 100 (reference position and orientation) is a specific value. In other words, the specific condition may be a condition that there is a specific relationship between the position and orientation of each virtual object and the position and orientation of the imaging unit 1010. The image generation unit 1120 determines whether the specific condition is satisfied based on the information on the position and orientation of the imaging unit 1010 in the world coordinate system, or the distance or angle formed between each virtual object and the display device 100. Then, the image generation unit 1120 controls whether to generate the first virtual image according to whether the specific condition is satisfied. Specifically, when it is determined that the specific condition is not satisfied, the image generation unit 1120 generates the first virtual image in the same manner as in the above embodiment. On the other hand, when it is determined that the specific condition is satisfied, the generation of the first virtual image is not performed. In this case, the display device 100 may display an image of the real space.

[0086] Note that when the information processing apparatus 410 directly connected to the display device 100 generates a virtual image for the display device 100 to display, the virtual image may be generated regardless of whether the specific condition is satisfied. That is, the image generation unit 1120 may generate the first virtual image regardless of whether the specific condition is satisfied. In this case, since the image composition unit 1130 does not acquire the second virtual image, an image obtained by synthesizing the first virtual image and the real image may be generated and output to the display device 100.

[0087] Further, the image generation unit 1120 does not control whether to generate the first virtual image, but may process the first virtual image or data of the virtual space when a specific condition is satisfied. For example, when a specific condition is satisfied, the image generation unit 1120 may change the data related to the light source irradiating the virtual space to a virtual light source that does not cast a shadow on the virtual object, and generate the first virtual image. Further, when a specific condition is satisfied, the image generation unit 1120 may change the drawing method to a method that does not cast a shadow on the virtual object, and generate the first virtual image. Further, the image generation unit 1120 may project a three-dimensional virtual object onto a predetermined virtual plane and process it so that it looks the same as the normal first virtual image only when viewed from a specific area. In addition, when a specific condition is satisfied, the image generation unit 1120 may lower the resolution (or reduce the total number of pixels) for drawing the first virtual image.

[0088] That is, when a specific condition is satisfied, the image generation unit 1120 generates a first virtual image (a first virtual image with low ease of estimating the shape of the virtual object) in which it is more difficult to estimate the shape of the virtual object from the first virtual image than when the specific condition is not satisfied. More specifically, when a specific condition is satisfied, the image generation unit 1120 reduces the amount of information or the accuracy of the information of the virtual object appearing in the first virtual image compared to when the specific condition is not satisfied.

[0089] By providing the conditions as shown in Modification 3, it is possible to provide a composite image without the need for data of all three-dimensional objects to be held in multiple locations while maintaining the security of the user.

[0090] <Embodiment 3> In Embodiments 1 and 2, all the components of each information processing device are configured by hardware. However, a part of each information processing device may be configured by software. In this case By causing a computer that implements the remaining part as hardware to execute this software, this computer realizes the operations of the information processing apparatus described in the above embodiment.

[0091] FIG. 7 is a block diagram showing a hardware configuration example of a computer applicable to each of the information processing apparatuses 110, 120, 410, and 420. The computer includes a CPU 701, a RAM 702, a ROM 703, a keyboard 704, a mouse 705, a display unit 706, an external storage device 707, a storage medium drive 708, and an interface 709.

[0092] The CPU 701 is a control unit that controls the entire computer using programs and data stored in the RAM 702 or the ROM 703. The CPU 701 executes each process performed by the information processing apparatuses 110, 120, 410, and 420.

[0093] The RAM 702 has an area for temporarily storing programs and data loaded from the external storage device 707 or the storage medium drive 708. Further, the RAM 702 has an area for temporarily storing data received from the outside via the interface 709. Also, the data received from the outside can be, for example, a real image. Further, the RAM 702 also has a work area used when the CPU 701 executes each process. That is, the RAM 702 can appropriately provide various areas. For example, the RAM 702 also functions as the data storage unit 1150 and the data storage unit 1230.

[0094] The ROM 703 stores computer setting data, a boot program, and the like.

[0095] The keyboard 704 and the mouse 705 are input devices (operating members) that receive user operations. A computer user can input various instructions to the CPU 701 by operating at least one of the keyboard 704 and the mouse 705.

[0096] The display unit 706 has a CRT or a liquid crystal screen, etc. The display unit 706 can display the processing results executed by the CPU 701 in the form of images or characters, etc.

[0097] The external storage device 707 is a large-capacity information storage device typified by a hard disk drive device. Programs (such as an OS (operating system)) and data for causing the CPU 701 to execute each of the above-described processes performed by each information processing device are stored in the external storage device 707. Such programs include programs executed by each component of each information processing device of the information processing system. Also, such data includes data in the virtual space and what was described as known information in the above description.

[0098] The programs and data stored in the external storage device 707 are appropriately loaded into the RAM 702 according to the control by the CPU 701. The CPU 701 executes processing using this loaded program and data, thereby executing each of the above-described processes performed by each information processing device. Note that the external storage device 707 may be used as the data storage unit 1150 and the data storage unit 1230.

[0099] The storage medium drive 708 reads programs and data recorded on a storage medium (such as a CD-ROM or a DVD-ROM). Also, the storage medium drive 708 writes programs and data to such a storage medium. Note that what is stored in the external storage device 707 Part or all of the program and data described above may be recorded on this storage medium. The program and data read by the storage medium drive 708 are output to the external storage device 707 or the RAM 702.

[0100] The interface 709 is an interface for connecting to the imaging unit 1010 and the imaging unit 4010. The interface 709 has an analog video port or a digital input / output port (such as IEEE1394). Also, the interface 709 has an Ethernet (registered trademark) port or the like for outputting to the display unit 1020 and the display unit 4020. The data input via the interface 709 is output to the RAM 702 or the external storage device 707. Also, when a sensor system is used to acquire the position and orientation information, the sensor system is connected to the interface 709.

[0101] The bus 710 is a bus connecting the above-described respective units.

[0102] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Some of the above-described embodiments may be appropriately combined.

[0103] Also, in the above, "If A is greater than or equal to B, proceed to step S1; if A is less than B, proceed to step S2" may be read as "If A is greater than B, proceed to step S1; if A is less than or equal to B, proceed to step S2". Conversely, "If A is greater than B, proceed to step S1; if A is less than or equal to B, proceed to step S2" may be read as "If A is greater than or equal to B, proceed to step S1; if A is less than B, proceed to step S2". Therefore, as long as there is no contradiction, "A or more" may be read as "greater than (higher; longer; more) than A", and "A or less" may be read as "less than (lower; shorter; less) than A". And "greater than (higher; longer; more) than A" may be read as "A or more", and "less than (lower; shorter; less) than A" may be read as "A or less".

[0104] Note that each functional unit of each of the above embodiments (each modification) may be individual hardware, or not. The functions of two or more functional units may be realized by common hardware. Each of the multiple functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Also, each functional unit may be realized by hardware such as an ASIC, FPGA, DSP, etc., or not. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. And the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0105] (Other Embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

[0106] The disclosure of the above embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing system having a first information processing device that stores information of a first object and a second information processing device that stores information of a second object different from the first object, wherein the first information processing device has first generation means for generating a first image depicting the first object based on information on the position and orientation of a first object and the information of the first object, the second information processing device has second generation means for generating a second image depicting the second object based on information on the position and orientation of the first object and the information of the second object, the first information processing device further has first composition means for generating an image obtained by combining the first image and the second image, characterized in that it is an information processing system. (Configuration 2) The first generation means further generates a first depth image indicating the depth information of each pixel of the first image, the second generation means further generates a second depth image indicating the depth information of each pixel of the second image, the first composition means generates an image obtained by combining the first image and the second image based on the first depth image and the second depth image, characterized in that it is the information processing system according to Configuration 1. (Configuration 3) The first generation means generates a third image depicting the first object based on information on the position and orientation of a second object and the information of the first object, the second generation means generates a fourth image depicting the second object based on the information on the position and orientation of the second object and the information of the second object, the second information processing device further has second composition means for generating an image obtained by combining the third image and the fourth image, The information processing system according to Configuration 1 or 2, characterized by the following. (Configuration 4) In the first case where a preset specific condition is satisfied, the second generation means does not generate the second image, and in the second case where the specific condition is not satisfied, the second generation means generates the second image. The information processing system according to any one of Configurations 1 to 3, characterized by the following. (Configuration 5) The second object is a three-dimensional object. In the first case where a preset specific condition is satisfied, the second generation means generates the second image in which the ease of estimating the shape of the second object is lower than in the second case where the specific condition is not satisfied. The information processing system according to any one of Configurations 1 to 3, characterized by the following. (Configuration 6) The second image generated in the first case has a smaller total number of pixels than the second image generated in the second case. The information processing system according to Configuration 5, characterized by the following. (Configuration 7) In the second image generated in the first case, the second object has no shadow. The information processing system according to Configuration 5 or 6, characterized by the following. (Configuration 8) In the first case, the second generation means generates the second image based on information obtained by processing the information of the second object. The information processing system according to any one of Configurations 5 to 7, characterized by the following. (Configuration 9) The specific condition is a condition that the position and orientation of the first object are in a specific position and orientation. The information processing system according to any one of Configurations 4 to 8, characterized by the following. (Configuration 10) The specific condition is a condition that the relationship between the position and orientation of the first object and the position and orientation of the second object is a specific relationship. The information processing system according to any one of Configurations 4 to 8, characterized in that... (Configuration 11) The first object is a three-dimensional object, In the first case, the first generation means generates the first image in which the easiness of estimating the shape of the first object is lower than that in the second case. The information processing system according to any one of Configurations 4 to 10, characterized in that... (Configuration 12) At least one of the first object and the second object is a three-dimensional object representing an object arranged in the real space. The information processing system according to any one of Configurations 1 to 11, characterized in that... (Configuration 13) The first information processing device does not store the information of the second object, The second information processing device does not store the information of the first object. The information processing system according to any one of Configurations 1 to 12, characterized in that... (Method) A control method for an information processing system having a first information processing device that stores information of a first object and a second information processing device that stores information of a second object different from the first object, In the first information processing device, a first generation step of generating a first image in which the first object is drawn based on the information of the position and orientation of the first object and the information of the first object; In the second information processing device, a second generation step of generating a second image in which the second object is drawn based on the information of the position and orientation of the first object and the information of the second object; In the first information processing device, a first synthesis step of generating an image obtained by synthesizing the first image and the second image; A control method for an information processing system, characterized by including the above steps. (Program) A program for causing a computer to function as each means of the information processing system described in any one of Configurations 1 to 13.

Explanation of Signs

[0107] 100: Display device, 110: Information processing device, 120: Information processing device, 1120: Image generation unit, 1130: Image composition unit, 1220: Image generation unit

Claims

1. An information processing system having a first information processing device for storing information of a first object and a second information processing device for storing information of a second object different from the first object, wherein the first information processing device has first generation means for generating a first image depicting the first object based on information on the position and orientation of a first object and information on the first object, the second information processing device has second generation means for generating a second image depicting the second object based on information on the position and orientation of the first object and information on the second object, and the first information processing device further has first composition means for generating an image obtained by compositing the first image and the second image. An information processing system characterized by the above.

2. The first generation means further generates a first depth image indicating depth information of each pixel of the first image, the second generation means further generates a second depth image indicating depth information of each pixel of the second image, and the first composition means generates an image obtained by compositing the first image and the second image based on the first depth image and the second depth image. The information processing system according to claim 1, characterized by the above.

3. The first generation means generates a third image depicting the first object based on information on the position and orientation of a second object and information on the first object, the second generation means generates a fourth image depicting the second object based on the information on the position and orientation of the second object and the information on the second object, and the second information processing device further has second composition means for generating an image obtained by compositing the third image and the fourth image. The information processing system according to claim 1 or 2, characterized by the above.

4. The second generation means does not generate the second image in a first case where a preset specific condition is satisfied, and generates the second image in a second case where the specific condition is not satisfied. The information processing system according to claim 1 or 2, characterized by the above.

5. The second object is a three-dimensional object. When the first case where a preset specific condition is satisfied occurs, the second generation means generates the second image in which the easiness of estimating the shape of the second object is lower than that in the second case where the specific condition is not satisfied. The information processing system according to claim 1 or 2, characterized in that.

6. The second image generated in the first case has a smaller total number of pixels than the second image generated in the second case. The information processing system according to claim 5, characterized in that.

7. In the second image generated in the first case, the second object has no shadow. The information processing system according to claim 5, characterized in that.

8. In the first case, the second generation means generates the second image based on the information obtained by processing the information of the second object. The information processing system according to claim 5, characterized in that.

9. The specific condition is the condition that the position and orientation of the first object are in a specific position and orientation. The information processing system according to claim 4, characterized in that.

10. The specific condition is the condition that the relationship between the position and orientation of the first object and the position and orientation of the second object is a specific relationship. The information processing system according to claim 4, characterized in that.

11. The first object is a three-dimensional object. In the first case, the first generation means generates the first image in which the easiness of estimating the shape of the first object is lower than that in the second case. The information processing system according to claim 4, characterized in that.

12. At least one of the first object and the second object is a three-dimensional object representing an object arranged in the real space. The information processing system according to claim 1 or 2, characterized in that.

13. The first information processing device does not store the information of the second object. The second information processing device does not store the information of the first object. The information processing system according to claim 1 or 2, characterized in that.

14. A control method for an information processing system having a first information processing device that stores information of a first object and a second information processing device that stores information of a second object different from the first object. In the first information processing apparatus, a first generation step of generating a first image in which the first object is drawn based on information on the position and orientation of the first object and information on the first object; In the second information processing apparatus, a second generation step of generating a second image in which the second object is drawn based on information on the position and orientation of the first object and information on the second object; In the first information processing apparatus, a first synthesis step of generating an image obtained by synthesizing the first image and the second image; A control method for an information processing system, comprising the above.

15. A program for causing a computer to function as each means of the information processing system according to claim 1 or 2.

Images