Electronic apparatus and method for controlling electronic apparatus

Abstract

To provide an electronic apparatus capable of reducing a communication amount when receiving information for displaying a CG image from an external device.SOLUTION: The electronic apparatus includes: acquisition means for acquiring, from an external device, a first image which is an image obtained when a virtual object is viewed from a first viewpoint and first depth information which is information concerning a distance of a depth direction when the virtual object is viewed from the first viewpoint; information generation means for, based on the first depth information, generating second depth information which is information concerning a distance of a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint; and image generation means for, based on the first image and the first depth information, generating a second image which is an image obtained when the virtual object is viewed from the second viewpoint and based on the second depth information and the second image, generating a composite image of the second viewpoint in which a depth of the virtual object in a space viewed from the second viewpoint is expressed.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an electronic device and a method for controlling an electronic device. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly blends the real world and the virtual world in real time. MR technology is used, for example, in video see-through head-mounted displays (HMDs). HMDs capture images of a subject observed from the pupil position of a user wearing the HMD using a video camera or the like, and display an image in which computer graphics (CG) are superimposed on the captured image.

[0003] A video see-through HMD captures an image of a subject using a charge-coupled device such as a CCD, acquires digital image data of the subject, and displays MR images (mixed reality images) to the user via a display device such as an LCD or organic EL display.

[0004] The HMD can receive and display superimposed images, in which a CG image is superimposed on a captured image, from an external device. The HMD transmits the captured image captured by the HMD to the external device. The external device calculates the position and orientation of the HMD using the captured image received from the HMD. Based on the calculated position and orientation of the HMD, the external device superimposes a CG image on the captured image and transmits it to the HMD. The HMD displays the superimposed image received from the external device. By wearing the HMD, a user wearing the HMD can observe an MR space.

[0005] Patent Document 1 proposes a method in which an image generating device transmits a computer graphics image together with depth information to an HMD, and the HMD generates an augmented reality image by superimposing the computer graphics image on a captured image of real space. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-95916 Summary of the Invention [Problem to be solved by the invention]

[0007] However, when CG images corresponding to the left and right eyes and depth information are transmitted from an external device such as an image generation device to an HMD, the amount of communication between the image generation device and the HMD increases.

[0008] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an electronic device that can reduce the amount of communication traffic when receiving information for displaying CG images from an external device. [Means for solving the problem]

[0009] The electronic device according to the present invention includes an acquisition means for acquiring, from an external device, a first image, which is an image of a virtual object viewed from a first viewpoint, and first depth information, which is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint; an information generation means for generating, based on the first depth information, second depth information, which is information about a distance in a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint; and an information generation means for generating, based on the first image and the first depth information, a second image, which is an image of the virtual object viewed from the second viewpoint, and and an image generation means for generating a composite image of the second viewpoint that expresses the depth of the virtual object in the space viewed from the second viewpoint, based on depth information and the second image. [Effects of the Invention]

[0010] According to the present invention, it is possible to reduce the amount of communication traffic when receiving information for displaying a CG image from an external device. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a system according to a first embodiment. [Figure 2] 1 is a block diagram of an HMD and an image processing device according to a first embodiment. [Figure 3] 4 is another example of a block diagram of the HMD and the image processing device according to the first embodiment. [Figure 4] 10A and 10B are diagrams illustrating a synthesis process performed by a synthesis unit. [Figure 5] 4 is a flowchart illustrating an image synthesis process according to the first embodiment. [Figure 6] FIG. 10 is a block diagram of an HMD and an image processing device according to a second embodiment. [Figure 7] 10 is a flowchart illustrating an image synthesis process according to the second embodiment. [Figure 8] FIG. 10 is a block diagram of an HMD and an image processing device according to a third embodiment. [Figure 9] FIG. 10 is a block diagram of an HMD and an image processing device according to a fourth embodiment. [Figure 10] FIG. 10 is a diagram illustrating differential image information. [Figure 11] FIG. 10 is a flowchart illustrating an image synthesis process according to the fourth embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a combination of transmission capacity and transmission information. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] <Embodiment 1> Fig. 1 is a diagram illustrating the configuration of an image display system according to embodiment 1. The image display system shown in Fig. 1 includes a head mounted display (HMD) 100, which is an example of an electronic device according to the present invention, and an image processing device 200. The image processing device 200 is connected to a display unit 211 and an operation unit 212.

[0014] The HMD 100 is worn on the head of a user. The HMD 100 can receive and display images generated by the image processing device 200. The display unit of the HMD 100 includes optical systems arranged in front of each of the user's left and right eyes.

[0015] The HMD 100 can communicate with the image processing device 200 via a small-scale network such as a wireless local area network (WLAN) or a wireless personal area network (WPAN). Communication between the HMD 100 and the image processing device 200 is not limited to a wireless communication method, and a wired communication method may also be used.

[0016] The image processing device 200 has a storage unit that stores content for allowing the user to experience MR. The content includes information for rendering CG (virtual objects) placed in the space viewed by the user. The image processing device 200 can communicate with the HMD 100 via wired or wireless communication.

[0017] The operation unit 212 connected to the image processing device 200 is an input device such as a keyboard. A user can input data, commands, etc. using the operation unit 212. The display unit 211 displays data input by the user and displays the results of processing based on commands from the user.

[0018] 2 is a block diagram of the HMD 100 and image processing device 200 according to embodiment 1. The HMD 100 includes a left imaging unit 102L, a right imaging unit 102R, a position and orientation acquisition unit 103, a communication unit 104, a left display unit 105L, a right display unit 105R, a depth calculation unit 106, a stereo depth information generation unit 107, a stereo image generation unit 108, a depth determination unit 109, and a synthesis unit 110.

[0019] The left imaging unit 102L and the right imaging unit 102R capture images of the outside world from approximately the same positions as the user's eyes. The left imaging unit 102L and the right imaging unit 102R are also collectively referred to as imaging units 102. The position and orientation acquisition unit 103 acquires information about the position and orientation of the HMD 100 from images of the outside world captured by the imaging units 102. The communication unit 104 is a communication interface for communicating with external devices. The communication unit 104 transmits and receives various data, such as images and depth information, as well as various control signals, to and from external devices such as the image processing device 200.

[0020] The left display unit 105L and the right display unit 105R display images (stereo images) to a user wearing the HMD 100. The left display unit 105L displays an image for the user's left eye, and the right display unit 105R displays an image for the user's right eye. The left display unit 105L and the right display unit 105R are also collectively referred to as display unit 105.

[0021] The depth calculation unit 106 calculates the depth of a real object (subject) in an image captured by the imaging unit 102, and generates stereo depth information of the real space. The stereo depth information generation unit 107 receives, from the image processing device 200, depth information (first depth information) corresponding to an image of a CG (virtual object) viewed with one eye of the user (first image, an image of the virtual object viewed from the first viewpoint). The first depth information is information about the distance in the depth direction when the CG is viewed from the first viewpoint. Based on the received first depth information, the stereo depth information generation unit 107 generates second depth information, which is information about the distance in the depth direction when the virtual object is viewed from a second viewpoint having parallax with respect to the first viewpoint. The stereo depth information generation unit 107 can acquire stereo depth information of the CG by generating the second depth information. The first depth information is depth information of the first image viewed with one eye of the user, and the second depth information is depth information of the second image viewed with the other eye of the user (an image having parallax with respect to the first image, an image of the CG viewed from the second viewpoint).

[0022] The stereo image generation unit 108 receives a CG first image from the image processing device 200. The stereo image generation unit 108 generates a second image having parallax with respect to the first image, based on the received first image and the first depth information received by the stereo depth information generation unit 107. The stereo image generation unit 108 can generate a CG stereo image by generating the second image.

[0023] The depth determination unit 109 compares the depth of the real object in the space in which the CG is placed with the depth of the CG, and determines which is placed further back. The synthesis unit 110 synthesizes the images captured by the imaging unit 102 with the CG stereo images generated by the stereo image generation unit 108. The stereo images synthesized by the synthesis unit 110 are displayed on the display unit 105.

[0024] The image processing device 200 is an external device such as a personal computer (PC), a workstation (WS), or a cloud server connected via a public network, which is different from the HMD 100. The image processing device 200 includes a communication unit 201, a content DB 202, and a CG drawing unit 203.

[0025] The communication unit 201 transmits and receives various data such as images and depth information, as well as various control signals, to and from the HMD 100. The content DB 202 stores the contents of virtual images. The CG drawing unit 203 uses the CG information stored in the content DB 202 to This is used to render CG according to the position and posture of the HMD 100.

[0026] The image processing device 200 can acquire information about the position and orientation of the HMD 100 from the HMD 100 and render CG of a virtual image. The position and orientation of the HMD 100 may be measured by an external sensor. The image processing device 200 can acquire information about the position and orientation of the HMD 100 from an external sensor and render CG. The image processing device 200 may also acquire information about the position and orientation of the HMD 100 from, for example, a captured image captured by the imaging unit 102 of the HMD 100.

[0027] When transmitting information about the position and orientation of the HMD 100 from the HMD 100 to the image processing device 200, the position and orientation acquisition unit 103 can acquire the position and orientation of the HMD 100 based on the captured image of the real space received from the imaging unit 102. The information about the position and orientation of the HMD 100 acquired by the position and orientation acquisition unit 103 is transmitted to the image processing device 200 via the communication unit 104.

[0028] The CG drawing unit 203 of the image processing device 200 draws a CG image for the left eye and an image for the right eye based on the received position and orientation information of the HMD 100. In the first embodiment, the image processing device 200 transmits the CG image for the left eye and the CG image for the right eye (CG stereo images) to the HMD 100 via the communication unit 201.

[0029] The composition unit 110 of the HMD 100 composes the CG stereo images received from the image processing device 200 with the captured images (stereo images of real space) captured by the imaging unit 102. For example, in the CG stereo images, the composition unit 110 uses the captured images captured by the imaging unit 102 in areas of a pre-specified chromakey color, and uses the CG stereo images in areas other than the chromakey color.

[0030] In addition, the synthesis unit 110 may receive an alpha value, which is information indicating the transparency of the CG, from the image processing device 200 along with the CG stereo image, and synthesize the captured image captured by the imaging unit 102 with the CG stereo image based on the alpha value.

[0031] The image synthesized by the synthesis unit 110 is displayed on the display unit 105. By wearing the HMD 100, the user can view a synthesized image of the CG image drawn by the image processing device 200 and the captured image captured by the imaging unit 102 in a state that corresponds to the position and posture of the user.

[0032] 2, the depth calculation unit 106 acquires depth information of a physical object in physical space (stereo depth information of the physical space) based on the captured image captured by the imaging unit 102. However, the depth calculation unit 106 may acquire the depth information of a physical object in physical space by measuring the distance to the physical object using a distance sensor 301 shown in FIG.

[0033] FIG. 3 shows another example of a block diagram of the HMD 100 and the image processing device according to the first embodiment. In the example of FIG. 3, the HMD 100 includes a distance sensor 301 in addition to the configuration shown in FIG. 2. The distance sensor 301 measures the distance to a subject of a real object. The depth calculation unit 106 acquires depth information of the real object captured by the imaging unit 102 based on the measurement result of the distance sensor 301. The distance sensor 301 is a sensor other than the imaging unit 102, and includes a LiDAR sensor (Light Detection and Ranging sensor), a ToF sensor (Time of Flight sensor), a millimeter-wave radar, etc. In the following description, it is assumed that the HMD 100 has the configuration shown in FIG. 2, and the depth calculation unit 106 acquires depth information of the real object based on the captured image captured by the imaging unit 102.

[0034] 4 is a diagram illustrating the synthesis process performed by the synthesis unit 110. In this example, the image for one eye (for example, the left eye) is synthesized instead of the images for both eyes. The synthesis unit 110 can also synthesize the image for the right eye in the same way as the image for the left eye.

[0035] Fig. 4(A) shows a CG image 400 received from the image processing device 200. The CG image 400 includes a CG cylinder 401. Fig. 4(B) shows a captured image 410 of real space captured by the imaging unit 102. The captured image 410 includes a table 411, which is a real object.

[0036] FIG. 4(C) shows a composite image 420 in which the cylinder 401 is located closer to the viewer than the table 411 (closer to the HMD 100). In the composite image 420, part of the table 411 is hidden by the cylinder 401. On the other hand, FIG. 4(D) shows a composite image 430 in which the cylinder 401 is located further from the viewer than the table 411 (farther from the HMD 100). In the composite image 430, part of the cylinder 401 is hidden by the table 411. By using depth information from the HMD 100 to the cylinder 401 and depth information from the HMD 100 to the table 411, the synthesis unit 110 can generate a composite image in which the depth of the cylinder 401 in real space is appropriately represented.

[0037] A method for calculating depth information of a real object and CG will be described. First, the calculation of depth information of a real object will be described. Depth information of a real object can be acquired using stereo images of a real space with parallax. The left imaging unit 102L and right imaging unit 102R shown in FIG. 2 are disposed at positions similar to the user's eyes and can capture stereo images of a real space with parallax. The depth calculation unit 106 calculates the depth of a real object using the stereo images of the real space captured by the imaging unit 102. The depth calculation unit 106 can calculate the depth of a real object from two captured images with parallax using, for example, the Sum of Absolute Difference (SAD) method or the Semi-Global Matching (SGM) method. Depth information is calculated for each pixel of an image.

[0038] Next, calculation of CG depth information will be described. CG stereo images are images to be synthesized with captured images, and are images with parallax, like the captured images. The stereo depth information generation unit 107 receives first depth information corresponding to a CG image (first image) seen by one of the user's eyes from the image processing device 200, and performs perspective projection transformation on the first depth information. The stereo depth information generation unit 107 can generate second depth information with parallax for the first depth information by transforming the first depth information using a known perspective projection transformation. The second depth information is depth information corresponding to a CG image (second image) seen by the other of the user's eyes. The stereo depth information generation unit 107 acquires CG stereo depth information by generating the second depth information from the first depth information.

[0039] The depth determination unit 109 compares the depth information of the real object in real space calculated by the depth calculation unit 106 with the stereo depth information of the CG generated by the stereo depth information generation unit 107, and determines for each pixel which is closer to the HMD 100. The synthesis unit 110 uses the image of the real object or the CG that is closer to the HMD 100 and performs synthesis processing.

[0040] Fig. 5 is a flowchart illustrating image synthesis processing according to embodiment 1. The processing shown in Fig. 5 is processing for synthesizing images of real space captured by the image capturing unit 102 with CG stereo images placed in real space.

[0041] In step S501, the depth calculation unit 106 acquires depth information of a real object in a captured image. Specifically, first, the left imaging unit 102L and the right imaging unit 102R acquire two captured images (stereo images of real space). Next, the depth calculation unit 106 calculates the depth of a real object (subject) in real space from the stereo images of real space, thereby acquiring stereo depth information of the real space.

[0042] In step S502, the stereo image generation unit 108 generates a CG image (second image) to be viewed by one eye of the user, based on the CG image (first image) received from the image processing device 200 and the first depth information corresponding to the first image. The second image is an image having parallax with respect to the first image. The stereo image generation unit 108 can obtain a CG stereo image by generating the second image from the first image.

[0043] The stereo image generation unit 108 can generate the second image from the first image by using a known method such as perspective projection transformation. The stereo image generation unit 108 may generate the second image from the first image by using other known methods other than perspective projection transformation.

[0044] In step S503, the stereo depth information generation unit 107 generates second depth information corresponding to a CG image (second image) seen with one eye of the user from the first depth information corresponding to a CG image (first image) seen with the other eye received from the image processing device 200. The second depth information is generated based on the first depth information, which is information about the distance in the depth direction when the CG is seen from the first viewpoint, and is information about the distance in the depth direction when the CG is seen from a second viewpoint having parallax with respect to the first viewpoint. The stereo depth information generation unit 107 can acquire stereo depth information of the CG by generating the second depth information from the first depth information.

[0045] In step S504, the depth determination unit 109 compares the depth of the real object in real space with the depth of the CG. The depth determination unit 109 compares the depth for each pixel of the image based on the depth information of the real object in real space (stereo depth information of real space) acquired by the depth calculation unit 106 and the stereo depth information of the CG generated by the stereo depth information generation unit 107.

[0046] In step S505, the synthesis unit 110 synthesizes the captured images of the real space with the CG stereo images based on the comparison result between the depth of the real object and the depth of the CG in step S504. The synthesis unit 110 synthesizes the captured images and the CG stereo images into a single image by using the image of the real object or the CG that is closer to the HMD 100. In step S505, the synthesis unit 110 generates a first viewpoint synthetic image (a synthetic image of an image of the real space and an image of the CG) that expresses the depth of the CG in the real space seen from the first viewpoint, based on the first depth information and the first image. Furthermore, the synthesis unit 110 generates a second viewpoint synthetic image (a synthetic image of an image of the real space and an image of the CG) that expresses the depth of the CG in the real space seen from the second viewpoint, based on the second depth information and the second image. The synthesis unit 110 generates a composite image of a first viewpoint and a composite image of a second viewpoint based on captured images of the real space, and can generate a stereo image in which the CG depth in the real space (mixed reality space) is appropriately expressed.

[0047] The second image generated in step S502 and the second depth information generated in step S503 may contain a loss of information about an occlusion region that is not visible in the received first image and first depth information.

[0048] The stereo image generation unit 108 may interpolate missing areas in the second image based on information about surrounding pixels. For example, the stereo image generation unit 108 may interpolate missing areas using an inpainting technique. Furthermore, the stereo depth information generation unit 107 may interpolate missing portions of the second depth information based on depth information about surrounding pixels.

[0049] In the first embodiment, the HMD 100 generates a CG image (first image) viewed by one eye of the user. The HMD 100 receives corresponding first depth information from the image processing device 200 and generates second depth information corresponding to the image (second image) seen by the other eye by perspective projection transformation using the first depth information. However, the present invention is not limited to this, and the HMD 100 may receive depth information at a viewpoint between the left eye and the right eye and perform perspective projection transformation using the received depth information to obtain depth information for the images seen by each of the left and right eyes.

[0050] In the first embodiment, the HMD 100 receives a CG image (first image) viewed by one of the user's eyes and first depth information corresponding to the first image from the image processing device 200. Based on the received first image and first depth information, the HMD 100 generates a second image viewed by the user's other eye and second depth information corresponding to the second image. The HMD 100 can generate an image in which the CG depth in real space is appropriately represented using a CG stereo image based on the first image and the second image and CG stereo depth information based on the first depth information and the second depth information. In this way, the HMD 100 can generate an image in which the CG depth is appropriately represented without receiving a second image and second depth information corresponding to one of the eyes from the image processing device 200, thereby reducing the amount of communication with the image processing device 200.

[0051] <Embodiment 2> In the first embodiment, the HMD 100 can generate an image that correctly represents the depth relationship between the real space and the CG in an MR system by receiving only a CG image and depth information corresponding to one eye from the image processing device 200. The second embodiment is an embodiment that generates an image that correctly represents the depth relationship between the real space and the CG in a VR (Virtual Reality) system while suppressing the amount of communication with the image processing device 200.

[0052] 6 is a block diagram of the HMD 100 and the image processing device 200 according to the second embodiment. The following describes configurations that differ from the configurations of the HMD 100 and the image processing device 200 according to the first embodiment (FIG. 2). The HMD 100 according to the second embodiment has a CG rendering unit 601, a controller position and orientation acquisition unit 602, and a reconstruction unit 603 instead of the depth calculation unit 106.

[0053] The CG rendering unit 601 receives the information on the position and orientation of the HMD 100 acquired by the position and orientation acquisition unit 103 , and can render CG different from the CG received from the image processing device 200 .

[0054] The position and orientation acquisition unit 103 can calculate the position and orientation of the HMD 100 from the captured image captured by the imaging unit 102, but may also acquire the position and orientation of the HMD 100 by other methods. For example, the position and orientation acquisition unit 103 may receive information measured by an external sensor or the like to acquire information about the position and orientation of the HMD 100. Furthermore, the position and orientation acquisition unit 103 may measure the position and orientation of the HMD 100 using a 6DoF (Degree of Freedom) sensor or a 3DoF sensor provided separately from the imaging unit 102.

[0055] A controller position and orientation acquisition unit 602 acquires information about the position and orientation of a controller for operating the HMD 100. The controller is communicably connected to the HMD 100, and is held in the hand of a user wearing the HMD 100.

[0056] The controller position and orientation acquisition unit 602 can acquire information about the position and orientation of the controller from an image of the controller captured by the imaging unit 102. The controller position and orientation acquisition unit 602 may acquire information about the position and orientation of the controller by receiving information measured by an external sensor or the like. The controller position and orientation acquisition unit 602 may also acquire information about the position and orientation of the controller by receiving from the controller the position and orientation calculated by the controller.

[0057] The reconstruction unit 603 performs 3D reconstruction based on the captured image captured by the imaging unit 102, and generates a reconstructed model. The reconstruction unit 603 can perform 3D reconstruction using a known method, but the reconstruction method is not particularly limited. As post-processing, the reconstruction unit 603 may extract a 3D reconstructed model of the hand of the user wearing the HMD 100 using a known method. The reconstruction unit 603 can generate the reconstructed model in any format, for example, polygon, voxel, or point cloud.

[0058] The CG drawing unit 601 can draw CG using the position and orientation of the HMD 100 acquired by the position and orientation acquisition unit 103, the position and orientation of the controller acquired by the controller position and orientation acquisition unit 602, and information on the reconstructed model generated by the reconstruction unit 603.

[0059] 6, the HMD 100 and the image processing device 200 each have a CG drawing unit 601 and a CG drawing unit 203. For example, it is assumed that the CG drawing unit 203 of the image processing device 200 draws CG for the entire screen, and the CG drawing unit 601 of the HMD 100 draws simple CG such as a controller held in the user's hand.

[0060] The CG rendering unit 601 transmits the rendered CG and the depth information of the CG to the depth determination unit 109. The depth determination unit 109 compares the stereo depth information of the CG generated by the stereo depth information generation unit 107 (CG received from the image processing device 200) with the stereo depth information of the CG rendered by the CG rendering unit 601, and determines which CG is closer to the HMD 100.

[0061] The synthesis unit 110 synthesizes a stereo image of the first CG with a stereo image of the second CG based on a comparison result between the depth of the CG (first CG) drawn by the CG drawing unit 601 and the depth of the CG (second CG) received from the image processing device 200. The synthesis unit 110 uses the image of the first CG or the second CG that is closer to the HMD 100 to synthesize the stereo image of the first CG with the stereo image of the second CG. In this way, the synthesis unit 110 can generate an image that expresses the depth relationship between the two CGs.

[0062] Fig. 7 is a flowchart illustrating an example of image synthesis processing according to embodiment 2. The same steps as those in the image synthesis processing according to embodiment 1 in Fig. 5 are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0063] In step S701, the CG rendering unit 601 renders CG based on the position and orientation of the HMD 100, the position and orientation of the controller, the reconstructed model, and the like.

[0064] In step S502, the stereo image generation unit 108 acquires a CG stereo image by generating a second image from the first image, as described in Fig. 5. In step S503, the stereo depth information generation unit 107 acquires CG stereo depth information by generating second depth information from the first depth information.

[0065] In step S702, the depth determination unit 109 compares the stereo depth information of the CG (first CG) drawn by the CG drawing unit 601 with the stereo depth information of the CG (second CG) received from the image processing device 200 for each pixel of the image.

[0066] In step S703, the synthesis unit 110 synthesizes the stereo images of the first CG and the stereo images of the second CG based on the result of comparing the depth of the first CG and the depth of the second CG in step S702. The synthesis unit 110 uses the image of the first CG or the second CG that is closer to the HMD 100, thereby synthesizing the stereo images of the first CG and the stereo images of the second CG into a single image.

[0067] In the above-described second embodiment, similarly to the first embodiment, the HMD 100 receives a CG image (first image) seen by one eye of the user and first depth information corresponding to the first image from the image processing device 200. Based on the received first image and first depth information, the HMD 100 generates a second image seen by the other eye of the user and second depth information corresponding to the second image. That is, the HMD 100 can receive information on a portion of the CG (second CG) from the image processing device 200 and acquire a stereo image and stereo depth information of the CG (second CG).

[0068] The HMD 100 synthesizes a stereo image of the second CG with a stereo image of the first CG based on the stereo depth information of the second CG and the stereo depth information of the CG (first CG) rendered by the HMD 100. This allows the HMD 100 to generate an image in which the depth relationship between the second CG (part of which has been received) from the image processing device 200 and the first CG rendered by the HMD 100 is appropriately expressed.

[0069] The HMD 100 can generate an image in which the depth of the CG (second CG) is appropriately expressed without receiving a second image and second depth information of the CG (second CG) from the image processing device 200, thereby reducing the amount of communication with the image processing device 200.

[0070] The CG (first CG) rendered by HMD 100 may be CG representing a virtual space provided by a VR system. In this case, in step S703, based on the first depth information and the first image, the synthesis unit 110 generates a first viewpoint composite image (a composite image of an image of the virtual space and an image of the second CG) in which the depth of the second CG in the virtual space seen from the first viewpoint is expressed. Furthermore, based on the second depth information and the second image, the synthesis unit 110 generates a second viewpoint composite image (a composite image of an image of the virtual space and an image of the second CG) in which the depth of the second CG in the virtual space seen from the second viewpoint is expressed. The synthesis unit 110 generates the first viewpoint composite image and the second viewpoint composite image based on the image of the virtual space, and can generate a stereo image in which the depth of the CG in the virtual space is appropriately expressed.

[0071] <Embodiment 3> In the first and second embodiments, the HMD 100 generates second depth information corresponding to a CG image (second image) viewed by one of the user's eyes from first depth information corresponding to a CG image (first image) viewed by the other eye. In the third embodiment, the HMD 100 generates second depth information corresponding to the second image from a CG stereo image (first image and second image) generated from the first CG image and the first depth information. That is, the HMD 100 acquires CG stereo depth information by generating second depth information corresponding to the second image based on the CG stereo image.

[0072] 8 is a block diagram of an HMD 100 and an image processing device 200 according to a third embodiment. The configuration of the HMD 100 and the image processing device 200 according to the third embodiment differs from the configuration of the first embodiment shown in FIG. 2 in that the stereo image generation unit 108 transmits CG stereo images generated to the stereo depth information generation unit 107. The stereo depth information generation unit 107 generates second depth information corresponding to a CG image (second image) seen by the other eye from the received CG stereo images (two images with parallax), similar to the case of captured images. The method of generating the depth information may be SAD, SGM, or the like, similar to the case of stereo depth information of real space obtained from two captured images, and is not particularly limited.

[0073] In the above-described third embodiment, the HMD 100 receives, similarly to the first embodiment, a CG image (first image) seen by one eye of the user and first depth information corresponding to the first image from the image processing device 200. Also, unlike the first embodiment, the HMD 100 according to the third embodiment receives the first image of the CG Second depth information corresponding to the second image is generated from a CG stereo image (first image and second image) generated from the image and the first depth information.

[0074] The HMD 100 can generate an image in which the depth of the CG is appropriately expressed, even when the second depth information is generated based on a CG stereo image without receiving the second image and the second depth information from the image processing device 200. Furthermore, the HMD 100 can reduce the amount of communication with the image processing device 200.

[0075] <Embodiment 4> In the first to third embodiments, the HMD 100 receives a CG image (first image) viewed by one eye of the user and first depth information corresponding to the first image from the image processing device 200, and generates a CG image (second image) viewed by the other eye and second depth information corresponding to the second image. That is, the HMD 100 acquires, from the image processing device 200, the first image, which is an image of a virtual object viewed from a first viewpoint, and the first depth information, which is information regarding the distance in the depth direction when the CG is viewed from the first viewpoint. Then, the HMD 100 generates, based on the first depth information, second depth information, which is information regarding the distance in the depth direction when the CG is viewed from a second viewpoint.

[0076] In contrast, in the fourth embodiment, the HMD 100 receives, in addition to the first image, differential image information, which is the difference between the first image and the second image, from the image processing device 200. The differential image information is information on a differential image relating to an image area that is not visible when viewed from the first viewpoint, of the CG viewed from the second viewpoint that has parallax with respect to the first viewpoint. The HMD 100 generates a second image (an image of the CG viewed from the second viewpoint) based on the first image and the differential image information, thereby generating a CG stereo image.

[0077] Furthermore, in addition to the first depth information, the HMD 100 may receive differential depth information, which is the difference between the first depth information and the second depth information, from the image processing device 200. The HMD 100 can generate CG stereo depth information by generating second depth information based on the first depth information and the differential depth information.

[0078] The HMD 100 according to the fourth embodiment can acquire CG stereo images and CG stereo depth information by receiving the difference between the first image and the first depth information without receiving the second image and the second depth information from the image processing device 200. Therefore, the HMD 100 can generate an image in which the CG depth is expressed.

[0079] 9 is a block diagram of the HMD 100 and the image processing device 200 according to the fourth embodiment. The following describes configurations that differ from the configurations of the HMD 100 and the image processing device 200 according to the first embodiment (FIG. 2). The HMD 100 according to the fourth embodiment has a stereo depth information restoration unit 901 and a stereo image restoration unit 902 instead of the stereo depth information generation unit 107 and the stereo image generation unit 108.

[0080] In the following description, the reference image (first image) transmitted by the image processing device 200 to the HMD 100 is a CG image viewed by the user's left eye. The image processing device 200 transmits the CG reference image viewed by the left eye and differential image information between the CG image (second image) viewed by the right eye and the reference image to the HMD 100. Note that the reference image may be a CG image viewed by the user's right eye.

[0081] The image processing device 200 transmits a CG image for the left eye and differential image information that is the difference between the CG image for the left eye and the CG image for the right eye to the HMD 100. The image processing device 200 also transmits depth information corresponding to the CG image for the left eye and differential depth information that is the difference between the depth information corresponding to the CG image for the right eye and the depth information corresponding to the CG image for the left eye to the HMD 100.

[0082] The stereo depth information restoration unit 901 restores (generates) depth information corresponding to the CG image for the right eye based on the depth information and differential depth information corresponding to the CG image for the left eye, thereby generating CG stereo depth information. The stereo image restoration unit 902 restores (generates) a CG image for the right eye based on the CG image for the left eye and differential image information, thereby generating a CG stereo image.

[0083] 10(A) to 10(C) are diagrams illustrating differential image information. 10(A) to 10(C) illustrate left and right CG images and differential image information when a rectangular parallelepiped CG image is rendered. 10(A) and 10(B) respectively show a CG image for the left eye and a CG image for the right eye generated by the image processing device 200. 10(C) shows differential image information that is the difference between the CG image for the left eye and the CG image for the right eye.

[0084] Generally, there is a high correlation between the left-eye image and the right-eye image of a stereo image, and therefore the image processing device 200 can reduce the amount of transmission by transmitting differential image information between the left-eye image and the right-eye image without transmitting the right-eye image to the HMD 100. The HMD 100 can generate a CG stereo image by receiving the CG image for the left eye, which is the reference image, and the differential image information from the image processing device 200.

[0085] Similarly, there is a high correlation between depth information corresponding to an image for the left eye (left-eye depth information) and depth information corresponding to an image for the right eye (right-eye depth information) of the stereo depth information. Therefore, the image processing device 200 can reduce the amount of transmission by transmitting differential depth information between the left-eye depth information and the right-eye depth information without transmitting right-eye depth information to the HMD 100. The HMD 100 can generate CG stereo depth information by receiving left-eye depth information, which is reference depth information, and differential depth information from the image processing device 200. The HMD 100 can generate an image in which the depth of the CG is expressed, based on the generated stereo image and stereo depth information.

[0086] Fig. 11 is a flowchart illustrating image synthesis processing according to the fourth embodiment. In the image synthesis processing shown in Fig. 11, the HMD 100 generates a CG stereo image using the left-eye image and difference image information received from the image processing device 200. The HMD 100 also generates CG stereo depth information using the left-eye depth information and difference depth information received from the image processing device 200. The same processes as those in the image synthesis processing according to the first embodiment in Fig. 5 are assigned the same reference numerals, and detailed descriptions thereof will be omitted.

[0087] In step S501, as described in FIG. 5, the depth calculation unit 106 acquires depth information of a real object in a captured image (stereo image in real space). In step S1101, the stereo image restoration unit 902 restores (generates) a CG image for the right eye based on a CG image for the left eye and differential image information that is the difference between the CG image for the right eye and the CG image for the left eye. In step S1102, the stereo depth information restoration unit 901 restores (generates) right-eye depth information corresponding to the CG image for the right eye based on left-eye depth information corresponding to the CG image for the left eye and differential depth information that is the difference between the right-eye depth information and the left-eye depth information.

[0088] 5 instead of step S1101, and then executes step S1102. That is, the HMD 100 may generate a CG stereo image by generating a CG image for the right eye based on a CG image for the left eye and left-eye depth information, without receiving differential image information from the image processing device 200.

[0089] After executing the process of step S1101, the HMD 100 executes the process of step S1102. 5 may be executed instead of the process in step S503 in Fig. 5. That is, the HMD 100 may generate right-eye depth information from left-eye depth information without receiving differential depth information from the image processing device 200, thereby generating CG stereo depth information.

[0090] The information to be used to generate or restore the right-eye CG image and right-eye depth information may be determined when the image processing device 200 is connected to the HMD 100. For example, the HMD 100 may determine the method of generating / restoring the right-eye CG image and right-eye depth information based on an operation from the user or based on the communication status with the image processing device 200.

[0091] Furthermore, the image processing device 200 may transmit data indicating the type of information to be transmitted along with the information to be transmitted to the HMD 100. The HMD 100 can determine a method for generating / restoring a CG image for the right eye and depth information for the right eye according to the content of the information received from the image processing device 200. If the HMD 100 has not received differential image information, it may execute the process of step S502 instead of the process of step S1101, and if the HMD 100 has not received differential depth information, it may execute the process of step S503 instead of the process of step S1102.

[0092] Furthermore, when transmitting the differential image information and differential depth information to the HMD 100, the image processing device 200 can further reduce the amount of transmission by compressing the differential information using a technique such as encoding.

[0093] In the above-described fourth embodiment, the HMD 100 receives, from the image processing device 200, a CG image for the left eye, which is a reference image, as well as differential image information, which is the difference between the CG image for the left eye and the CG image for the right eye. The HMD 100 also receives, from the image processing device 200, depth information for the left eye as well as differential image information, which is the difference between the depth information for the left eye and the depth information for the right eye. The image processing device 200 can also reduce the amount of communication traffic by transmitting differential image information instead of a CG image for the right eye and differential depth information instead of right eye depth information. The HMD 100 can generate a composite image in which the depth of the CG is appropriately expressed by generating a CG stereo image using the CG image for the left eye, differential image information, left eye depth information, and differential depth information, and synthesizing the generated image with a captured image in real space.

[0094] <Embodiment 5> In the first to third embodiments, the HMD 100 generates an image in which the depth of the CG is expressed by receiving a CG image (first image) seen by one eye of the user and first depth information corresponding to the first image from the image processing device 200. In the fourth embodiment, the HMD 100 further receives differential image information and differential depth information to generate an image in which the depth of the CG is expressed.

[0095] In the fifth embodiment, the HMD 100 generates CG stereo depth information and CG stereo images, and changes information acquired from the image processing device 200 to generate an image expressing the CG depth, according to predetermined conditions. The predetermined conditions include, for example, a condition regarding the transmission capacity between the HMD 100 and the image processing device 200, and a condition regarding the distance to the CG.

[0096] An example in which the predetermined condition is a condition regarding the transmission capacity between the HMD 100 and the image processing device 200 will be described with reference to Fig. 12. Fig. 12 is a diagram showing examples of combinations of the transmission capacity between the image processing device 200 and the HMD 100 and the transmission data transmitted from the image processing device 200 to the HMD 100. Fig. 12 illustrates three combinations depending on the transmission capacity, but the combinations of transmission capacity and transmission information are not limited to these.

[0097] In the case of a wireless connection, the transmission capacity is expected to change due to, for example, radio interference and congestion. In the case of a wired connection, the transmission capacity is expected to change due to, for example, the cable length and cable quality. For the transmission capacity "large," "medium," and "small," corresponding transmission capacity ranges are set in advance according to various expected conditions.

[0098] When the transmission capacity is "large," priority is given to the image quality of the CG, and therefore the image processing device 200 transmits left and right CG images (CG stereo images) and left and right depth information (CG stereo depth information) to the HMD 100. The HMD 100 uses the received left and right CG images and left and right depth information to generate a composite image of the captured images (stereo images of real space) and (CG stereo images).

[0099] When the transmission capacity is "medium," the image processing device 200 transmits a CG image for the left eye, differential image information, left-eye depth information, and differential depth information to the HMD 100, as described in the fourth embodiment. The HMD 100 generates a CG image for the right eye and right-eye depth information using the received information, thereby generating a CG stereo image and CG stereo depth information.

[0100] When the transmission capacity is "small," the image processing device 200 transmits only the left-eye CG image and left-eye depth information to the HMD 100. As described in the first embodiment, the HMD 100 generates the right-eye CG image and right-eye depth information using the received information, thereby generating the CG stereo image and CG stereo depth information.

[0101] The timing for determining the transmission capacity includes, for example, when the image processing device 200 is started, when the HMD 100 is started, and when an application for rendering CG is started. The timing for determining the transmission capacity may be when the communication state changes by monitoring the communication state. The determination of the transmission capacity may be performed by the image processing device 200 or the HMD 100.

[0102] For example, when the HMD 100 determines the transmission capacity, the HMD 100 changes the information to be acquired from the image processing device 200 according to the transmission capacity. The HMD 100 notifies the image processing device 200 of the type of information to be acquired from the image processing device 200. The image processing device 200 changes the information to be transmitted to the HMD 100 based on the notified type of information.

[0103] When the image processing device 200 determines the transmission capacity, the image processing device 200 changes the information to be transmitted to the HMD 100 in accordance with the transmission capacity. The image processing device 200 notifies the HMD 100 of the type of information to be transmitted to the HMD 100.

[0104] The type of information transmitted and received between the image processing device 200 and the HMD 100 may be changed for each frame. In this case, the image processing device 200 adds data indicating the type of information included in each frame to each frame of an image to be transmitted.

[0105] The predetermined condition for determining whether to change the information acquired from the image processing device 200 is not limited to the transmission capacity between the image processing device 200 and the HMD 100. For example, the predetermined condition may be a condition regarding the distance to the CG. A CG that is located far from the HMD 100 has less parallax between the image for the left eye and the image for the right eye. Therefore, when a CG is located far from the HMD 100, the HMD 100 can generate an image in which the depth of the CG is appropriately expressed even when it receives only the CG image seen by one of the user's eyes and the corresponding depth information.

[0106] In the fifth embodiment, the image processing device 200 sends the image to the HMD 100 in accordance with a predetermined condition. By appropriately changing the type of information to be transmitted, the transmission capacity between the image processing device 200 and the HMD 100 is reduced. Therefore, the HMD 100 can reduce the transmission capacity between the image processing device 200 and the HMD 100 and generate an image in which the depth of the CG is appropriately expressed.

[0107] The above first to fifth embodiments illustrate examples in which stereo images in which the depth of the CG in space is appropriately expressed are generated using CG stereo images based on the first and second images and CG stereo depth information based on the first depth information and the second depth information. In this case, the HMD 100 generates a first viewpoint composite image (a composite image of an image of the space and a CG image) in which the depth of the CG in space seen from the first viewpoint is expressed, based on the first depth information and the first image. Furthermore, the HMD 100 generates a second viewpoint composite image (a composite image of an image of the space and a CG image) in which the depth of the CG in space seen from the second viewpoint is expressed, based on the second depth information and the second image.

[0108] Note that the present invention is not limited to generating a composite image of the first viewpoint and a composite image of the second viewpoint, and the HMD 100 may receive a composite image of the first viewpoint, which expresses the CG depth in the space viewed from the first viewpoint, from an external device. In this case, the HMD 100 may generate a composite image of the second viewpoint, which expresses the CG depth in the space viewed from the second viewpoint, based on the second depth information and the second image.

[0109] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0110] The above processor is a processor in a broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0111] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0112] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more of the functions.

[0113] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first image which is an image of a virtual object viewed from a first viewpoint; and an acquisition means for acquiring first depth information from an external device, the first depth information being information about a distance in a depth direction when viewing a body; an information generating means for generating second depth information, which is information regarding a distance in a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint, based on the first depth information; an image generating means for generating a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the first depth information, and for generating a composite image of the second viewpoint, which represents the depth of the virtual object in space viewed from the second viewpoint, based on the second depth information and the second image; An electronic device comprising: (Configuration 2) the acquisition means further acquires an image of the space seen from the first viewpoint; The image generating means generates a synthetic image of the first viewpoint, which represents the depth of the virtual object in the space as viewed from the first viewpoint, based on the first depth information and the first image. 2. The electronic device according to configuration 1. (Configuration 3) The space is a real space, The image generating means generates a composite image of the first viewpoint and a composite image of the second viewpoint based on a captured image of the real space. 3. The electronic device according to configuration 2. (Configuration 4) The space is a virtual space, The image generating means generates a composite image of the first viewpoint and a composite image of the second viewpoint based on the image of the virtual space. 3. The electronic device according to configuration 2. (Configuration 5) The image generating means generates the second image by performing perspective projection transformation on the first image. 2. The electronic device according to configuration 1. (Configuration 6) The information generating means generates the second depth information based on the second image generated by the image generating means. 6. The electronic device according to any one of configurations 1 to 5. (Configuration 7) The image generating means interpolates a missing area in the second image based on information about surrounding pixels. 7. The electronic device according to any one of configurations 1 to 6. (Configuration 8) The information generating means generates depth information of the real space based on the captured image. 8. The electronic device according to any one of configurations 3 to 7. (Configuration 9) The information generating means acquires depth information of the real space using at least one of a LiDAR sensor, a ToF sensor, and a millimeter-wave radar. 8. The electronic device according to any one of configurations 3 to 7. (Configuration 10) The information generating means generates the second depth information by performing perspective projection transformation on the first depth information. 10. The electronic device according to any one of configurations 1 to 9. (Configuration 11) the acquiring means further acquires differential depth information, which is a difference between the first depth information and the second depth information, from the external device; The information generating means generates the second depth information based on the first depth information and the differential depth information. 11. The electronic device according to any one of configurations 1 to 10. (Configuration 12) The information generating means interpolates the missing portion of the second depth information based on depth information of surrounding pixels. 12. The electronic device according to any one of configurations 1 to 11. (Configuration 13) further comprising a drawing means for drawing a second virtual object different from the virtual object; The image generating means generates an image that expresses a depth relationship between the virtual object and the second virtual object in the space viewed from the second viewpoint by synthesizing the second image and an image of the second virtual object based on depth information of the second virtual object, which is information about a distance in a depth direction when the second virtual object is viewed from the second viewpoint, and the second depth information. 2. The electronic device according to configuration 1. (Configuration 14) The acquisition means changes the second depth information and the information acquired from the external device to generate the second image according to a predetermined condition. 14. The electronic device according to any one of configurations 1 to 13. (Configuration 15) The predetermined condition includes at least one of a transmission capacity between the external device and the virtual object and a distance to the external device. 15. The electronic device according to configuration 14. (Configuration 16) an acquisition means for acquiring from an external device a first image, which is an image of a virtual object viewed from a first viewpoint, first depth information, which is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint, and a difference image, which is related to an image region of the virtual object viewed from a second viewpoint having a parallax with respect to the first viewpoint and which is not visible when viewed from the first viewpoint; an information generating means for generating second depth information, which is information regarding a distance in a depth direction when the virtual object is viewed from the second viewpoint, based on the first depth information; an image generating means for generating a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the difference image, and for generating a composite image of the second viewpoint, which represents the depth of the virtual object in the space viewed from the second viewpoint, based on the second depth information and the second image; An electronic device comprising: (method) acquiring, from an external device, a first image that is an image of a virtual object viewed from a first viewpoint, and first depth information that is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint; generating second depth information based on the first depth information, the second depth information being information about a distance in a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint; generating a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the first depth information, and generating a composite image of the second viewpoint, which represents the depth of the virtual object in space viewed from the second viewpoint, based on the second depth information and the second image; 1. A method for controlling an electronic device, comprising: (program) A program for causing a computer to function as each means of the electronic device according to any one of configurations 1 to 16. [Explanation of symbols]

[0114] 100: HMD, 200: image processing device, 104: communication unit, 107: stereo depth information generation unit, 109: depth determination unit, 110: synthesis unit

Claims

1. an acquisition means for acquiring, from an external device, a first image, which is an image of a virtual object viewed from a first viewpoint, and first depth information, which is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint; an information generating means for generating, based on the first depth information, second depth information which is information regarding a distance in a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint; an image generating means for generating a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the first depth information, and generating a composite image of the second viewpoint, which represents the depth of the virtual object in space viewed from the second viewpoint, based on the second depth information and the second image; An electronic device comprising:

2. the acquisition means further acquires an image of the space seen from the first viewpoint; The image generating means generates a synthetic image of the first viewpoint, which represents the depth of the virtual object in the space as viewed from the first viewpoint, based on the first depth information and the first image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

3. The space is a real space, The image generating means generates a composite image of the first viewpoint and a composite image of the second viewpoint based on a captured image of the real space.

3. The electronic device according to claim 2.

4. The space is a virtual space, The image generating means generates a composite image of the first viewpoint and a composite image of the second viewpoint based on the image of the virtual space.

3. The electronic device according to claim 2.

5. The image generating means generates the second image by performing perspective projection transformation on the first image.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

6. The information generating means generates the second depth information based on the second image generated by the image generating means.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

7. The image generating means interpolates a missing area in the second image based on information about surrounding pixels.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

8. The information generating means generates depth information of the real space based on the captured image.

4. The electronic device according to claim 3.

9. The information generating means acquires depth information of the real space using at least one of a LiDAR sensor, a ToF sensor, and a millimeter-wave radar.

4. The electronic device according to claim 3.

10. The information generating means generates the second depth information by performing perspective projection conversion on the first depth information. Generate information 2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

11. the acquiring means further acquires, from the external device, differential depth information that is a difference between the first depth information and the second depth information; The information generating means generates the second depth information based on the first depth information and the differential depth information.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

12. The information generating means interpolates a missing portion of the second depth information based on depth information of surrounding pixels.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

13. further comprising a drawing means for drawing a second virtual object different from the virtual object; The image generating means generates an image that expresses a depth relationship between the virtual object and the second virtual object in the space viewed from the second viewpoint by combining the second image and an image of the second virtual object based on depth information of the second virtual object, which is information about a distance in a depth direction when the second virtual object is viewed from the second viewpoint, and the second depth information.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

14. The acquisition means changes the second depth information and the information acquired from the external device to generate the second image in accordance with a predetermined condition.

2. The electronic device according to claim 1, wherein the electronic device is a semiconductor device.

15. The predetermined condition includes at least one of a transmission capacity between the external device and the virtual object and a distance to the external device.

15. The electronic device according to claim 14.

16. an acquisition means for acquiring from an external device a first image, which is an image of a virtual object viewed from a first viewpoint, first depth information, which is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint, and a difference image, which is related to an image region of the virtual object viewed from a second viewpoint having a parallax with respect to the first viewpoint and which is not visible when viewed from the first viewpoint; an information generating means for generating second depth information, which is information regarding a distance in a depth direction when the virtual object is viewed from the second viewpoint, based on the first depth information; an image generating means for generating a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the difference image, and for generating a composite image of the second viewpoint, which represents the depth of the virtual object in the space viewed from the second viewpoint, based on the second depth information and the second image; An electronic device comprising:

17. acquiring, from an external device, a first image that is an image of a virtual object viewed from a first viewpoint, and first depth information that is information about a distance in a depth direction when the virtual object is viewed from the first viewpoint; generating second depth information based on the first depth information, the second depth information being information regarding a distance in a depth direction when the virtual object is viewed from a second viewpoint having a parallax with respect to the first viewpoint; a second image, which is an image of the virtual object viewed from the second viewpoint, based on the first image and the first depth information; and a composite image of the second viewpoint, which expresses the depth of the virtual object in the space viewed from the second viewpoint, based on the second depth information and the second image. generating a 1. A method for controlling an electronic device, comprising:

18. A program for causing a computer to function as each of the means of the electronic device according to any one of claims 1 to 16.

Images