Information processing system, information processing device, and information processing method

Abstract

To improve secrecy in image transmission.SOLUTION: In an information processing system having a first information processing device and a second information processing device, the first information processing device has: acquisition means for acquiring a picked-up image obtained by imaging a real space; processing means for processing the picked-up image in the case of satisfying a specific condition on the necessity of a secret of the picked-up image to the second information processing device; and first transmission means for transmitting the processed picked-up image to the second information processing device, and the second information processing device has: first generation means for generating an object image of a virtual object on the basis of the picked-up image received from the first transmission means; and second transmission means for transmitting the generated object image to the first information processing device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing apparatus, and an information processing method for performing image processing related to virtual reality, augmented reality, and mixed reality.

Background Art

[0002] In recent years, the development of virtual reality (VR) and mixed reality (MR) systems aimed at seamless connection between the real space and the virtual space has been actively carried out. Also, the development of a system in which a plurality of users share an image of a virtual object via a network has been carried out. According to such a system, an operation performed by a certain user on a virtual object is reflected in the virtual object viewed by other users, and a plurality of users can cooperate to perform operations on the virtual object.

[0003] For example, Patent Document 1 discloses a client system that downloads content via the Internet, renders the content, and displays it on an HMD. Also, as a method of rendering an image of a virtual reality system or a mixed reality system, there is a server-side rendering method performed on a server with abundant computing resources as in Patent Document 2, for example. Also, as another method of rendering an image, there are a cloud rendering method that performs the same processing in the cloud and a method that renders an image by a device that presents the image.

[0004] Also, Patent Document 3 proposes a technique of performing authentication using biometric information of a user and displaying confidential data on a display when the authentication is successfully completed. Also, Patent Document 4 proposes a technique of controlling the display of confidential data according to the attributes of the user and the type of the display device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] According to the above prior art, the display of the virtual space is controlled using the user's identification information. On the other hand, when presenting an MR video including an object in the virtual space having, for example, reflection characteristics to the user, the captured image of the real space is sent to the server, and the server reflects the information of the real space on the object in the virtual space. In this case, in the above prior art, the information of the real space is sent to the server regardless of the user's identification information, and an MR video in which the information of the real space is reflected on the object in the virtual space is displayed. At this time, since the information of the real space is sent to the server, problems related to information security may occur when the server exists in an environment outside the user's management.

[0007] The present invention has been made in view of the above, and an object thereof is to improve the confidentiality in the transmission of images. [Means for Solving the Problems

[0008] The information processing system according to the present disclosure includes a first information processing apparatus and a second information processing apparatus. An information processing system, wherein the first information processing device includes an acquisition unit configured to acquire a captured image obtained by capturing a real space, a processing unit configured to process the captured image when a specific condition regarding the necessity of concealing the captured image with respect to the second information processing device is satisfied, and a first transmission unit configured to transmit the processed captured image to the second information processing device; and the second information processing device includes a first generation unit configured to generate an object image of a virtual object based on the captured image received from the first transmission unit, and a second transmission unit configured to transmit the generated object image to the first information processing device.

[0009] An information processing device according to the present disclosure includes an acquisition unit configured to acquire a captured image obtained by capturing a real space, a processing unit configured to process the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific device is satisfied based on the captured image, and a transmission unit configured to transmit the processed captured image to the specific device.

[0010] Further, an information processing method according to the present disclosure causes an information processing device to execute an acquisition step of acquiring a captured image obtained by capturing a real space, a processing step of processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific device is satisfied based on the captured image, and a transmission step of transmitting the processed captured image to the specific device.

Advantages of the Invention

[0011] According to the technology of the present disclosure, it is possible to improve the security in image transmission.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Note that the embodiments described below are merely examples of means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0014] [First Embodiment] The information processing system according to the first embodiment will be described. The information processing system according to the present embodiment includes an image display device and an information processing device connected to the image display device. The information processing device constituting the information processing system in the present embodiment may be a server device, a PC (Personal Computer), or may have a cloud form.

[0015] In addition, the image display device constituting the information processing system according to the present embodiment is applicable to an HMD (Head Mounted Display), which is a type of head-mounted display device. The image display device according to the present embodiment may be other head-mounted display devices. For example, the image display device according to the present embodiment is also applicable to a handheld type display device, which is a type of head-mounted display device and is held by the user's hand and mounted (applied) to the head. The handheld type display device is, for example, a smartphone or a tablet It is a head-mounted terminal. Further, the image display device according to the present embodiment is also applicable to smart glasses (AR (Augmented Reality) glasses), which are a type of head-mounted display device.

[0016] Furthermore, the image display device according to the present embodiment is applicable to both a head-mounted display device in which the user views an image with both eyes and a head-mounted display device in which the user views an image with one eye. Also, a smartphone mounted on a head-mounted adapter (for example, VR (Virtual Reality) goggles) is a type of head-mounted display device. Further, the image display device according to the present embodiment is also applicable to display devices other than head-mounted display devices (for example, stationary display devices).

[0017] It is assumed that the video see-through method is adopted in the image display device according to the present embodiment. When the image display device adopts the video see-through method, for example, an image of the real space (the outside world) captured by an imaging device (for example, a video camera) is displayed in real time on the display surface of the image display device (a display surface that does not transmit light from the real space). Also, graphics (for example, images of objects in the virtual space) are superimposed (composed) on the image of the real space and displayed on the display surface. In this case, although the user cannot directly view the real space, by viewing the displayed image, the user can indirectly view the real space or view the graphics superimposed on the image of the real space.

[0018] FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 according to the present embodiment includes an information processing device 100 and an image display device 110.

[0019] The image display device 110 is assumed to be an HMD using the video see-through method as an example. The image display device 110 is a first information processing device and is connected to an information processing device 100, which is a second information processing device, via a network. Although one image display device is shown in the figure, a plurality of image display devices having the same configuration as the image display device 110 may be connected to the information processing device 100 to form the information processing system 1.

[0020] The information processing device 100 includes an image generation unit 1010, an imaging image application unit 1020, and a data storage unit 1030. In this embodiment, as an example, the information processing device 100 is assumed to be a server device.

[0021] The image generation unit 1010 constructs a virtual space based on the data of the virtual space stored in the data storage unit 1030. The data of the virtual space includes data of virtual objects related to each virtual object constituting the virtual space, data related to a realistic approximation object in which the three-dimensional shape information of a real object acquired from the real space is incorporated into the virtual space, and data related to a light source that irradiates the virtual space. Then, the image generation unit 1010 acquires the position and orientation information of the virtual viewpoint stored in the data storage unit 1030 and sets a virtual viewpoint (reference position) in the virtual space. Then, the image generation unit 1010 generates an image of the virtual space (virtual space image) seen from the set virtual viewpoint. Since the technology for generating an image of the virtual space seen from a viewpoint having a predetermined position and orientation is a well-known technology, a detailed description of image generation is omitted here.

[0022] The imaging image application unit 1020 is a generation means for generating an object image of a virtual object based on the imaging image. The imaging image application unit 1020 applies the information of the imaging image to the data of the virtual space using the image of the real space received from the image display device 110. The imaging image application unit 1020 may apply the information of the imaging image to an environment map or to the texture of a virtual object existing in the virtual space. Also, the imaging image application unit 1020 may apply the information of the imaging image to the position, intensity, and color temperature of the light source in the virtual space.

[0023] As described above, the data storage unit 1030 stores various information and is configured by a RAM, a hard disk drive device, etc. Note that in addition to the information described above to be stored in the data storage unit 1030, information described as known information in this embodiment is also stored in the data storage unit 1030.

[0024] Next, the image display device 110 includes an imaging unit 1110, a position and orientation calculation unit 1120, a control information acquisition unit 1130, a control unit 1140, an image synthesis unit 1150, and a display unit 1160.

[0025] The imaging unit 1110 is an acquisition means for acquiring an imaging image of the real space, continuously captures images of the real space over time, and outputs the captured images of the real space to the control unit 1140. The imaging unit 1110 may include a stereo camera composed of two cameras fixed to each other so as to be able to image the real space in the line-of-sight direction from the viewpoint position of the user wearing the image display device. Further, the imaging unit 1110 may include a camera that captures an image of the current space around including the user.

[0026] The position and orientation calculation unit 1120 is a calculation means for calculating position and orientation information regarding the position and orientation of the imaging unit 1110. The position and orientation calculation unit 1120 calculates the position and orientation of the imaging unit 1110 in the world coordinate system. Specifically, markers are arranged in advance at positions where the image display device 110 can image. Then, the imaging unit 1110 images the real space including the markers. The position and orientation calculation unit 1120 extracts the markers based on the image of the real space captured by the imaging unit 1110. Then, the position and orientation calculation unit 1120 calculates the position and orientation of the imaging unit 1110 in the world coordinate system based on the position and orientation of the markers in the image. Then, the position and orientation calculation unit 1120 outputs the calculated position and orientation information of the imaging unit 1110 to the control unit 1140.

[0027] Note that the method for calculating the position and orientation of the imaging unit 1110 is not limited to the above method, and various methods may be adopted. For example, the position and orientation calculation unit 1120 may perform Simultaneous Localization and Mapping (SLAM) processing based on the feature points shown in the image to calculate the position and orientation of the imaging unit 1110 and the position and orientation of the individual coordinate system. When a sensor whose relative position and orientation with respect to the imaging unit 1110 is known is attached to the image display device 110, the position and orientation calculation unit 1120 may convert the measurement value by the sensor based on the relative position and orientation to calculate the position and orientation of the imaging unit 1110. Further, the position and orientation calculation unit 1120 may calculate the position and orientation of the imaging unit 1110 based on the information obtained by using a motion capture system.

[0028] The control information acquisition unit 1130 acquires information for the control unit 1140 to control the information transmitted to the information processing apparatus 100, and outputs the acquired information to the control unit 1140. The information acquired by the control information acquisition unit 1130 is information related to the transmission conditions regarding the necessity of concealing the captured image with respect to the information processing apparatus 100. Specifically, the control information acquisition unit 1130 acquires the destination information of the user of the image display device 110. Further, the control information acquisition unit 1130 may acquire the position information of the image display device 110 or the position information of the user of the image display device 110. Furthermore, the control information acquisition unit 1130 may acquire the device information of the image display device 110 or the device information of the information processing apparatus 100. Also, the control information acquisition unit 1130 may acquire information regarding the communication path between the image display device 110 and the information processing apparatus 100. Then, the control information acquisition unit 1130 transmits the acquired information to the control unit 1140.

[0029] The control unit 1140 is a processing means for processing the captured image when the transmission conditions regarding the necessity of concealing the captured image are satisfied. Specifically, the control unit 1140 is the control information acquisition unit 11 Based on the information obtained by 30, it is determined whether it is necessary to process the information to be transmitted to the information processing apparatus 100, and the information to be transmitted to the information processing apparatus 100 is processed based on the determination result. A further specific example of the information processing by the control unit 1140 will be described later.

[0030] The image synthesis unit 1150 synthesizes the image of the virtual space received from the information processing apparatus 100 and the image of the real space captured by the imaging unit 1110 to generate an MR image. Then, the image synthesis unit 1150 outputs the generated MR image to the display unit 1160.

[0031] The display unit 1160 displays the MR image generated by the image synthesis unit 1150. The display unit 1160 may include two displays arranged corresponding to the left and right eyes of the user, respectively. In this case, the MR image for the left eye is displayed on the display corresponding to the left eye of the user, and the MR image for the right eye is displayed on the display corresponding to the right eye of the user.

[0032] Next, the processing executed in the information processing system 1 will be described with reference to FIG. 2. As shown in FIG. 2, in the present embodiment, the image display apparatus 110 processes and transmits the information to be transmitted to the information processing apparatus 100. Further, the information processing apparatus 100 generates an image of the virtual space based on the information received from the image display apparatus 110, and transmits the generated image to the image display apparatus 110. Further, the image display apparatus 110 performs a synthesis process on the image of the real space based on the image received from the information processing apparatus 100, and displays the synthesized image.

[0033] First, in step S2010, the control information acquisition unit 1130 acquires control information related to the transmission of information to the information processing apparatus 100. This control information includes information related to the transmission conditions regarding the necessity of concealing an image with respect to the information processing apparatus 100. As a specific example, the control information includes the destination information of the user of the image display apparatus 110, the position information of the image display apparatus 110, and the position information of the user of the image display apparatus 110. Further, the control information includes the device information of the image display apparatus 110, the device information of the information processing apparatus 100, and information regarding the communication path between the image display apparatus 110 and the information processing apparatus 100.

[0034] Next, in step S2020, the imaging unit 1110 captures the surroundings of the real space where the image display apparatus 110 is located. The image captured by the imaging unit 1110 is transmitted to the control unit 1140.

[0035] Then, in step S2030, the control unit 1140 determines whether the transmission conditions regarding the necessity of concealing an image with respect to the information processing apparatus 100 are satisfied based on the control information acquired in step S2010. The control unit 1140 determines whether it is necessary to process the information to be transmitted to the information processing apparatus 100 according to whether the transmission conditions are satisfied. Here, the transmission conditions are determined using user information related to the user who uses the image display apparatus 110, information indicating the existence of the image display apparatus 110 or the user who uses the image display apparatus 110, information of the image display apparatus 110, or information of the communication path. For example, as a transmission condition, there is a condition that the destination of the user who uses the image display apparatus 110 is different from the destination of the information processing apparatus 100. Further, as another example of the transmission condition, there is a condition that there are a plurality of destinations of the user who uses the image display apparatus 110. In addition to this, as the transmission conditions, any conditions regarding the necessity of concealing an image to be transmitted to the information processing apparatus 100, which can be determined using the various pieces of information described above, may be included.

[0036] Then, when the transmission condition is satisfied, the control unit 1140 performs processing for processing the information to be transmitted to the information processing apparatus 100. For example, when the control unit 1140 determines that the transmission condition that the destination of the user who uses the image display device 110 is different from the destination of the information processing apparatus 100 is satisfied, the control unit 1140 processes the image to be transmitted to the information processing apparatus 100.

[0037] For example, the control unit 1140 two-dimensionally reduces at least a part of the image area of the captured image. Further, the control unit 1140 may detect a light source, the user's body, or a real object held by the user in the captured image, and leave the image area related to the detected object and reduce the other areas. In addition to this, the control unit 1140 can execute various processes for changing at least a part of the captured image as processing for processing.

[0038] For example, the control unit 1140 may reduce the resolution of at least a part of the captured image. Further, the control unit 1140 may apply at least a part of the area of the captured image to another area in the captured image. Further, the control unit 1140 may change the color of the captured image by changing the color table applied to at least a part of the captured image. When changing the color of the captured image, the control unit 1140 may binarize the captured image or set colors according to the image area. Further, the control unit 1140 may detect a real object in the captured image and make the colors different for each object, or make the colors different according to the distance between the image display device 110 and the real object. Further, the control unit 1140 may replace at least a part of the captured image with another image specified in advance among the images stored in a storage unit (not shown) of the image display device 110. In this case, the control unit 1140 may select, as the image to be replaced, an image similar to the part to be replaced in the captured image. Note that the control unit 1140 may acquire the image to be replaced from an external device (not shown) that holds the image.

[0039] Also, for example, the information regarding the user's affiliation of the image display device 110 may include information indicating the user's position or information indicating that the user is an external user (e.g., an external contractor, etc.) outside the affiliation destination. In this case, for example, when a detailed shape with relatively high confidentiality among the shapes of an object is depicted in the captured image and the user has no position, the control unit 1140 may perform processing to delete the detailed shape from the captured image. Further, when the user is an external user outside the affiliation destination, the control unit 1140 may perform processing to delete the object itself from the captured image. Instead of or in addition to this, processing to replace the detailed shape with a simple shape may be performed according to the type of the image display device 110 and the displayable resolution. In this way, by processing the shape in the image into a detailed shape or a simple shape according to the user's position and type and the performance of the image display device 110, it becomes possible to transmit and display an image more suitable for the usage environment of the information processing system.

[0040] Note that the information transmitted to the information processing device 100 processed in step S2030 may include all or part of the captured image by the imaging unit 1110, or may include the captured image applied to the environment map. Also, the information transmitted to the information processing device 100 processed in step S2030 may include an image of the environment map created based on the captured image, or may include information regarding the light source in the image calculated based on the captured image.

[0041] Next, in step S2040, the position and orientation calculation unit 1120 calculates position and orientation information indicating the position and orientation at the time of acquisition of the captured image by the imaging unit 1110. Next, in step S2050, the control unit 1140 transmits the captured image processed in step S2030 to the information processing apparatus 100. Further, in step S2060, the control unit 1140 transmits, as transmission means, the position and orientation information calculated in step S2040 to the information processing apparatus 100. Note that the control unit 1140 does not have to transmit the processed captured image to the information processing apparatus 100 when the transmission condition is satisfied in step S2030. By doing so, it is also possible to enhance the confidentiality of the image transmitted from the image display apparatus 110 to the information processing apparatus 100.

[0042] The processes from step S2070 to step S2090 are processes executed by the information processing apparatus 100. In step S2070, the captured image application unit 1020 of the information processing apparatus 100 , generates an image of a virtual object that constitutes a virtual space using the captured image received from the image display apparatus 110 in step S2050. For example, the captured image application unit 1020 generates an image of a virtual object by applying the captured image received from the image display apparatus 110 to the texture of the virtual object. Note that in step S2060, when the processed captured image is not transmitted from the image display apparatus 110 to the information processing apparatus 100, the captured image application unit 1020 may generate an image of a virtual object without using the image to be applied to the texture. Alternatively, the captured image application unit 1020 may generate an image of a virtual object by using an image stored in advance in the data storage unit 1030 as the image to be applied to the texture.

[0043] Next, in step S2080, the image generation unit 1010 generates an image of the virtual space using the image of the virtual object generated in step S2070 and the position and orientation information of the virtual viewpoint calculated in step S2040. Next, in step S2090, the image generation unit 1010 transmits, as transmission means, the image of the virtual space generated in step S2080 to the image display apparatus 110.

[0044] Next, the processes of step S2100 and step S2110 are processes executed by the image display device 110. In step S2100, the image composition unit 1150 of the image display device 110 composes the image captured by the imaging unit 1110 in step S2020 and the image of the virtual space received from the information processing device 100 in step S2090. By doing this, the image composition unit 1150 generates a composite image. Then, in step S2110, the display unit 1160 displays the composite image generated in step S2100.

[0045] With reference to FIGS. 3A to 3D, an example of a composite image using the image transmitted from the image display device 110 to the information processing device 100 and the object image of the virtual space generated by the information processing device 100 will be described.

[0046] FIG. 3A shows a captured image 3010 that constitutes a series of real-space images captured by the image display device 110 in step S2020 above. In the captured image 3010 of FIG. 3A, for example, an object 3011 that needs to be concealed outside the destination of the user of the image display device 110 is depicted. Here, it is assumed that the destination of the information processing device 100 is the same as the destination of the user of the image display device 110, and the transmission condition used for the processing determination of the transmission information is that the destination of the user of the image display device 110 and the destination of the information processing device 100 are different. In this case, in step S2030, the control unit 1140 determines that the transmission condition of the captured image 3010 is not satisfied, and transmits the captured image 3010 to the information processing device 100 without performing the process of deleting the object 3011 from the captured image 3010.

[0047] FIG. 3B shows an image 3020 of a virtual space generated by the information processing apparatus 100 using the captured image 3010 received from the image display apparatus 110 when the control unit 1140 determines that the transmission condition is not satisfied and does not perform the processing on the captured image 3010. In this case, in step S2070, the captured image application unit 1020 of the information processing apparatus 100 generates an image of a virtual object 3030 to which the captured image 3010 is applied as a texture. Then, in step S2080, the image generation unit 1010 generates an image 3020 of a virtual space using the generated image of the virtual object 3030 and the position and orientation information received in step S2060. As shown in FIG. 3B, in the generated image 3020 of the virtual space, the object 3011 depicted in the captured image 3010 is also applied to the texture as the object 3031 in the virtual object 3030.

[0048] Next, as an example, when the destination of the information processing apparatus 100 is different from the destination of the user Assume that the transmission condition used for the determination of the processing of the transmission information is that the destination of the user of the image display apparatus 110 is different from the destination of the information processing apparatus 100. In this case, in step S2030, the control unit 1140 determines that the transmission condition of the captured image 3010 is satisfied, and performs a process of deleting the object 3011 from the captured image 3010. Note that the process of specifying the object to be processed among the objects depicted in the image can be realized using a well-known technique, and thus a detailed description thereof is omitted here. FIG. 3C shows the captured image 3040 after the processing is performed on the captured image 3010. As shown in FIG. 3C, the object 3011 depicted in the captured image 3010 is deleted from the captured image 3040. Thereby, when it is necessary to ensure the confidentiality of the information transmitted from the image display apparatus 110 to the information processing apparatus 100, the image display apparatus 110 can ensure the confidentiality of the transmitted information by performing the above-described processing on the image to be transmitted.

[0049] FIG. 3D shows an image 3050 of a virtual space generated by the information processing apparatus 100 using the captured image 3040 received from the image display apparatus 110 when the control unit 1140 determines that the transmission condition is satisfied and performs a processing operation on the captured image 3010. In this case, in step S2070, the captured image application unit 1020 of the information processing apparatus 100 generates an image of a virtual object 3060 to which the captured image 3040 is applied as a texture. Then, in step S2080, the image generation unit 1010 generates the image 3050 of the virtual space using the image of the generated virtual object 3060 and the position and orientation information received in step S2060. As shown in FIG. 3D, in the generated image 3050 of the virtual space, it can be seen that the object 3011 depicted in the captured image 3010 is not applied to the texture of the object in the virtual object 3060.

[0050] Therefore, according to the information processing system 1 of the present embodiment, image processing is performed based on the condition related to the confidentiality of the information transmitted and received between the image display apparatus 110 and the information processing apparatus 100. By this, an improvement in the security at the time of transmitting the information used for generating the image of the virtual space can be expected.

[0051] Next, a modification of the information processing system 1 of the above embodiment will be described. In the following description, the same components and processes as those of the above embodiment will be denoted by the same reference numerals, and detailed descriptions thereof will be omitted.

[0052] [Modification 1] In the above embodiment, the information processing apparatus 100 generates the image of the virtual space using the virtual object image, but the image display apparatus 110 may generate the image of the virtual space.

[0053] The processing executed in the information processing system 1 of this modification example will be described with reference to FIG. 4. In this modification example, the image display device 110 and the information processing device 100 of the information processing system 1 execute the above-described processing of steps S2010 to S2050 and S2070, respectively. Then, in step S2090, the information processing device 100 transmits the object image generated in step S2070 to the image display device 110.

[0054] In step S2200, the control unit 1140 of the image display device 110 generates an image of the virtual space based on the object image received in step S2090 and the position and orientation information of the virtual viewpoint calculated in step S2040. Then, in step S2100, the image composition unit 1150 of the image display device 110 composes the image captured by the imaging unit 1110 in step S2020 and the image of the virtual space generated in step S2200. Then, in step S2110, the display unit 1160 displays the composite image composed in step S2100.

[0055] According to this modification example, the amount of information transmitted and received between the image display device 110 and the information processing device 100 can be further reduced, such as omitting the transmission of the position and orientation information from the image display device 110 to the information processing device 100.

[0056] [Modification Example 2] In the above-described embodiment, it is assumed that the image display device 110 is an image display device adopting the video see-through method, but the image display device 110 may be an image display device adopting the optical see-through method. When the image display device 110 adopts the optical see-through method, for example, a graphic (for example, an image of an object in the virtual space) is displayed on the display surface (the display surface that transmits light from the real space (the outside world)). In this case, the user of the image display device can directly view the real space (the outside world) through the display surface or view the graphic displayed on the display surface.

[0057] The processing executed in the information processing system 1 of this modification example will be described with reference to FIG. 5. In this modification example, the image display device 110 and the information processing device 100 of the information processing system 1 execute the processing of steps S2010 to S2090 described above, respectively.

[0058] Then, in step S2300, the image composition unit 1150 of the image display device 110 superimposes and displays the image of the virtual space received from the information processing device 100 in step S2090 on the display surface of the display unit 1160.

[0059] Therefore, according to this modification example, regardless of whether the image display method adopted by the image display device 110 is a video see-through method or an optical see-through method, the confidentiality of the information transmitted and received between the image display device 110 and the information processing device 100 can be enhanced.

[0060] [Second Embodiment] Next, an information processing system according to the second embodiment will be described. In the following description, the same components and processes as those in the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are omitted.

[0061] The information processing system 1 according to this embodiment has, as shown in FIG. 1, an image display device 110 and an information processing device 100. In the first embodiment, it is assumed that each part constituting the image display device 110 shown in FIG. 1 is configured by hardware. However, a part of each part of the image display device 110 may be configured by software. In this case, by causing software to be executed on the part implemented as hardware, the processing of the image display device 110 described in the first embodiment can be realized.

[0062] FIG. 6 is a block diagram showing an example of the hardware configuration of a computer applicable to the image display device 110 of the information processing system 1 according to this embodiment.

[0063] The CPU 4001 controls the entire image display device 110 using the programs and data stored in the RAM 4002 and the ROM 4003, and executes each of the above-described processes explained in the first embodiment.

[0064] The RAM 4002 has an area for temporarily storing programs and data loaded from the external storage device 4007 and the storage medium drive 4008. Further, the RAM 4002 has an area for temporarily storing data received from the outside via the I / F (interface) 4009. Also, the data received from the outside refers to an image in the real space and input values generated by operation input devices such as the keyboard 4004 and the mouse 4005. Furthermore, the RAM 4002 also has a work area used when the CPU 4001 executes each process. That is, the RAM 4002 can appropriately provide various areas necessary for the processes executed by the image display device 110. The ROM 4003 stores computer setting data, a boot program, and the like.

[0065] The keyboard 4004 and the mouse 4005 are examples of operation input devices, and various instructions can be input to the CPU 4001 by a user of the computer operating them.

[0066] The display unit 4006 is composed of a CRT, a liquid crystal screen, or the like, and can display the processing results by the CPU 4001 as images, characters, or the like.

[0067] The external storage device 4007 is a large-capacity information storage device typified by a hard disk drive device. The external storage device 4007 stores an OS (Operating System), programs, and data for causing the CPU 4001 to execute each of the above-described processes performed by the image display device 110. Such programs include, for example, programs corresponding to the position and orientation calculation unit 1120, the control information acquisition unit 1130, the control unit 1140, and the image synthesis unit 1150. Further, such data includes data of the virtual space and those described as known information in the above description. The programs and data stored in the external storage device 4007 are appropriately loaded into the RAM 4002 according to the control by the CPU 4001. The CPU 4001 executes processes using the loaded programs and data, thereby executing each of the above-described processes performed by the image display device 110. Note that the external storage device 4007 may be used as a storage unit for data used by the image display device 110 in the above processes.

[0068] The storage medium drive 4008 reads programs and data recorded on a storage medium such as a CD-ROM or a DVD-ROM, and writes programs and data to such a storage medium. Note that part or all of the programs and data described as being stored in the external storage device 4007 may be recorded on this storage medium. The programs and data read by the storage medium drive 4008 from the storage medium are output to the external storage device 4007 or the RAM 4002.

[0069] I / F4009 is composed of an analog video port for connecting the imaging unit 1110 of the image display device 110, or a digital input / output port such as IEEE1394. Also, I / F4009 is composed of an Ethernet (registered trademark) port for outputting the composite image to the display unit 1160 of the image display device 110. The data received via I / F4009 is input to the RAM4002 and the external storage device 4007. Also, when a sensor system is used to acquire the position and orientation information, the sensor system will be connected to this I / F4009. 4010 is a bus connecting the above-mentioned respective parts.

[0070] Also, although the present invention has been described in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of this invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine each embodiment.

[0071] [Other Embodiments] The present invention includes the case where a software program is supplied directly or remotely to a system or device, and the computer of the system or device reads and executes the supplied program code to achieve the functions of the above-described embodiments. In this case, the supplied program is a computer-readable program corresponding to the flowchart illustrated in the embodiment.

[0072] Also, when the computer executes the read program, in addition to realizing the functions of the above-described embodiments, based on the instructions of the program, the functions of the embodiments may also be realized in cooperation with the OS or the like running on the computer. In this case, the OS or the like performs part or all of the actual processing, and the functions of the above-described embodiments are realized by this processing.

[0073] The present invention supplies a program that causes a computer to execute each step of the information processing method according to the above-described embodiment to a system or device via a network or a storage medium. Such a program is configured to be read and executed by one or more processors in a computer of the system or device. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0074] The disclosure of this embodiment includes the following configurations, methods, and programs. (Configuration 1) An information processing system having a first information processing device and a second information processing device, wherein the first information processing device has an acquisition means for acquiring a captured image obtained by imaging the real space, a processing means for processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to the second information processing device is satisfied, a first transmission means for transmitting the processed captured image to the second information processing device, and the second information processing device has a first generation means for generating an object image of a virtual object based on the captured image received from the first transmission means, a second transmission means for transmitting the generated object image to the first information processing device, and an information processing system characterized by the above. (Configuration 2) The first information processing device further has a calculation means for calculating position and orientation information regarding the position and orientation of the first information processing device, the first transmission means transmits the position and orientation information to the second information processing device, the first generation means generates an image of a virtual space including the object image based on the position and orientation information received from the first transmission means, and the second transmission means transmits the generated image of the virtual space to the first information processing device The information processing system according to Configuration 1, characterized by the above. (Configuration 3) The first information processing device Calculation means for calculating position and orientation information regarding the position and orientation of the first information processing device, Second generation means for generating an image of a virtual space including the object image based on the object image and the position and orientation information, The information processing system according to Configuration 1, further characterized by including the above. (Configuration 4) The specific condition includes a condition related to at least one of the user information of the first information processing device, the position information of the first information processing device, the device information of the first information processing device, the device information of the second information processing device, and the information regarding the communication path between the first information processing device and the second information processing device. The information processing system according to any one of Configurations 1 to 3 system. (Configuration 5) A destination is set for the user of the first information processing device and the second information processing device, The specific condition is that the destination of the user of the first information processing device is different from the destination of the second information processing device The information processing system according to any one of Configurations 1 to 3, characterized by the above. (Configuration 6) A destination is set for the user of the first information processing device and the second information processing device, The specific condition is that there are a plurality of destinations for the user of the first information processing device The information processing system according to any one of Configurations 1 to 3, characterized by the above. (Configuration 7) The first transmission means does not transmit the captured image to the second information processing device when the specific condition is satisfied. The information processing system according to any one of Configurations 1 to 6 (Configuration 8) When the specific condition is satisfied, the processing means processes the captured image so as to leave at least one of the area related to the user of the first information processing apparatus and the area related to the light source included in the captured image. The information processing system according to any one of Configurations 1 to 6. (Configuration 9) The area related to the user of the first information processing apparatus includes an area related to an object held by the user. The information processing system according to Configuration 8. (Configuration 10) When the specific condition is satisfied, the processing means changes at least a part of the captured image. The information processing system according to any one of Configurations 1 to 6. (Configuration 11) The processing means reduces the resolution of at least a part of the captured image. The information processing system according to Configuration 10. (Configuration 12) The processing means replaces at least a part of the captured image with another image. The information processing system according to Configuration 10. (Configuration 13) The other image is an image similar to at least a part of the captured image. The information processing system according to Configuration 12. (Configuration 14) The processing means changes the color of at least a part of the captured image. The information processing system according to Configuration 10. (Configuration 15) The processing means changes the color table applied to at least a part of the captured image. The information processing system according to Configuration 14. (Configuration 16) The processing means changes the color of at least a part of the captured image according to the object included in at least a part of the captured image. The information processing system according to Configuration 14. (Configuration 17) The information processing system according to Configuration 16, wherein the processing means changes at least a part of the colors of the captured image according to the distance between the image display device and an object included in at least a part of the captured image. (Configuration 18) The information processing system according to Configuration 14, wherein the processing means binarizes at least a part of the captured image. (Configuration 19) The information processing system according to any one of Configurations 1 to 18, wherein the second information processing device is a server device. (Configuration 20) An acquisition means for acquiring a captured image obtained by imaging a real space; A processing means for processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific device that generates an object image of a virtual object based on the captured image is satisfied; A transmission means for transmitting the processed captured image to the specific device; An information processing device, characterized by comprising the above. (Method) In an information processing device, An acquisition step of acquiring a captured image obtained by imaging a real space; A processing step of processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific device that generates an object image of a virtual object based on the captured image is satisfied; A transmission step of transmitting the processed captured image to the specific device; An information processing method, characterized by executing the above. (Program) A program for causing a computer to realize each means of the information processing device according to Configuration 20.

Explanation of Reference Numerals

[0075] 1 Information processing system, 100 Information processing device, 110 Image display device, 1110 Imaging unit, 1140 Control unit, 1020 Captured image application unit, 1010 Image generation unit

Claims

1. An information processing system having a first information processing device and a second information processing device, wherein the first information processing device has an acquisition means for acquiring a captured image obtained by imaging a real space, a processing means for processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to the second information processing device is satisfied, and a first transmission means for transmitting the processed captured image to the second information processing device, and the second information processing device has a first generation means for generating an object image of a virtual object based on the captured image received from the first transmission means, and a second transmission means for transmitting the generated object image to the first information processing device An information processing system characterized by the above.

2. The first information processing device further has a calculation means for calculating position and orientation information regarding the position and orientation of the first information processing device, the first transmission means transmits the position and orientation information to the second information processing device, the first generation means generates an image of a virtual space including the object image based on the position and orientation information received from the first transmission means, and the second transmission means transmits the generated image of the virtual space to the first information processing device The information processing system according to claim 1, characterized by the above.

3. The first information processing device has a calculation means for calculating position and orientation information regarding the position and orientation of the first information processing device, and a second generation means for generating an image of a virtual space including the object image based on the object image and the position and orientation information, The information processing system according to claim 1, further characterized by the above.

4. The specific condition includes a condition regarding at least one of user information of the first information processing device, position information of the first information processing device, device information of the first information processing device, device information of the second information processing device, and information regarding a communication path between the first information processing device and the second information processing device. The information processing system according to any one of claims 1 to 3, characterized by the above.

5. A destination is set for the user of the first information processing device and the second information processing device, and the specific condition is that the destination of the user of the first information processing device is different from the destination of the second information processing device ​ The information processing system according to any one of claims 1 to 3, characterized in that it is so.

6. The destination is set for the user of the first information processing device and the second information processing device, The specific condition is that there are a plurality of destinations for the user of the first information processing device The information processing system according to any one of claims 1 to 3, characterized in that it is so.

7. The information processing system according to any one of claims 1 to 3, characterized in that the first transmission means does not transmit the captured image to the second information processing device when the specific condition is satisfied.

8. The information processing system according to any one of claims 1 to 3, characterized in that when the specific condition is satisfied, the processing means processes the captured image so as to leave at least one of the area related to the user of the first information processing device and the area related to the light source included in the captured image.

9. The information processing system according to claim 8, characterized in that the area related to the user of the first information processing device includes an area related to an object held by the user.

10. The information processing system according to any one of claims 1 to 3, characterized in that the processing means changes at least a part of the captured image when the specific condition is satisfied.

11. The information processing system according to claim 10, characterized in that the processing means reduces the resolution of at least a part of the captured image.

12. The information processing system according to claim 10, characterized in that the processing means replaces at least a part of the captured image with another image.

13. The information processing system according to claim 12, characterized in that the other image is an image similar to at least a part of the captured image.

14. The information processing system according to claim 10, characterized in that the processing means changes the color of at least a part of the captured image.

15. The information processing system according to claim 14, characterized in that the processing means changes the color table applied to at least a part of the captured image.

16. The information processing system according to claim 14, characterized in that the processing means changes the color of at least a part of the captured image according to the object included in at least a part of the captured image.

17. The information processing system according to claim 16, wherein the processing means changes at least a part of the color of the captured image according to the distance between the first information processing apparatus and an object included in at least a part of the captured image.

18. The information processing system according to claim 14, wherein the processing means binarizes at least a part of the captured image.

19. The information processing system according to any one of claims 1 to 3, wherein the second information processing apparatus is a server apparatus.

20. An acquisition means for acquiring a captured image obtained by imaging a real space; A processing means for processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific apparatus that generates an object image of a virtual object based on the captured image is satisfied; A transmission means for transmitting the processed captured image to the specific apparatus; An information processing apparatus, characterized by comprising the above.

21. An information processing apparatus, An acquisition step of acquiring a captured image obtained by imaging a real space; A processing step of processing the captured image when a specific condition regarding the necessity of concealing the captured image with respect to a specific apparatus that generates an object image of a virtual object based on the captured image is satisfied; A transmission step of transmitting the processed captured image to the specific apparatus; An information processing method, characterized by causing the above to be executed.

22. A program for causing a computer to realize each means of the information processing apparatus according to claim 20.

Images