Display unit, method for controlling display unit, and program

Abstract

To reduce the amount of communication between an external device and a HMD when the HMD performs display according to information on the depth of an image created by the external device.SOLUTION: A display unit has a first display means and second display means, and is worn on the head of a user. The display unit has: receiving means that receives, from an external device, a first image and a second image having parallax to each other; first calculation means that calculates at least one of information on the depth of the first image and information on the depth of the second image; and display control means that controls the first display means to display an image based on the information on the depth of the first image, and controls the second display means to display an image based on the information on the depth of the second image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a display device, a control method for a display device, and a program. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly fuses the real world and the virtual world in real time. In MR technology, a video see-through type HMD (Head Mounted Display) is sometimes used. In MR technology, CG (Computer Graphics) can be superimposed on a real image of a subject (approximately matching the subject observed from the pupil position of the HMD user) captured by a video camera or the like.

[0003] A video see-through HMD captures an image of a subject using a charge-coupled device such as a CCD to obtain digital image data of the subject. A video see-through HMD presents an MR image (mixed reality image) superimposed with a CG image to a user by displaying the MR image on a display device (such as a liquid crystal or organic EL display).

[0004] Patent Document 1 proposes a technique in which an image generating device transmits CG depth information together with the CG to an HMD, and the HMD synthesizes the CG and a real image based on the depth information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-95916 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, since it is necessary to transmit depth information from the image generating device (external device) to the HMD, the amount of communication between the image generating device and the HMD increases.

[0007] An object of the present invention is to reduce the amount of communication between an external device and an HMD when the HMD displays an image generated by the external device in accordance with depth information. [Means for solving the problem]

[0008] One aspect of the present invention is a method for producing a composition comprising the steps of: A display device having a first display means and a second display means and worn on a user's head, A receiving means for receiving a first image and a second image having a parallax from an external device; a first calculation means for calculating at least one of depth information of the first image and depth information of the second image; a display control means for controlling a first display means to display an image based on the depth information of the first image, and for controlling a second display means to display an image based on the depth information of the second image; The display device is characterized by having:

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: A method for controlling a display device having a first display means and a second display means and worn on a user's head, comprising the steps of: A receiving step of receiving a first image and a second image having a parallax from an external device. and, a first calculation step of calculating at least one of depth information of the first image and depth information of the second image; a display control step of controlling a first display means to display an image based on the depth information of the first image, and controlling a second display means to display an image based on the depth information of the second image; The present invention relates to a method for controlling a display device, the method comprising the steps of: Effect of the Invention

[0010] According to the present invention, when an HMD performs display according to depth information of an image generated by an external device, it is possible to reduce the amount of communication between the external device and the HMD. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a configuration diagram of an image display system according to a first embodiment. [Diagram 2] FIG. 1 is a functional block diagram of an image display system according to a first embodiment. [Diagram 3] FIG. 4 is another functional block diagram of the image display system according to the first embodiment. [Figure 4] FIG. 1 is a diagram illustrating the synthesis according to embodiment 1. [Diagram 5] 4 is a flowchart of image synthesis processing according to the first embodiment. [Figure 6] FIG. 11 is a functional block diagram of an image display system according to a second embodiment. [Figure 7] 11 is a flowchart of an image synthesis process according to the second embodiment. [Figure 8] FIG. 11 is a functional block diagram of an image display system according to a third embodiment. [Figure 9] FIG. 13 is a diagram for explaining an example of reducing the amount of information according to the third embodiment. [Figure 10] 13 is a flowchart of an image synthesis process according to the third embodiment. [Figure 11] FIG. 11 is a functional block diagram of an image display system according to a fourth embodiment. [Figure 12] FIG. 13 is a diagram for explaining an example of reducing the amount of information according to the fourth embodiment. [Figure 13] 13 is a flowchart of an image synthesis process according to the fourth embodiment. [Figure 14] FIG. 13 is a diagram for explaining an example of reducing the amount of information according to the fifth embodiment. [Figure 15] 13 is a flowchart of an image synthesis process according to the fifth embodiment. [Figure 16] FIG. 13 is a functional block diagram of an image display system according to a sixth embodiment. [Figure 17] 23 is a flowchart of an image synthesis process according to the sixth embodiment. [Figure 18] FIG. 23 is a diagram for explaining an example of reducing the amount of information according to the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> 1 is a diagram showing the configuration of an image display system 1 according to embodiment 1. The image display system 1 includes an HMD (Head Mounted Display) 1201 and an image processing device 1210.

[0013] The HMD 1201 is a display device worn on the head of a user. The HMD 1201 communicates with an image processing device 1210 that constitutes a part of a small-scale network. The small-scale network is, for example, a wireless local area network (WLAN) or a wireless personal area network (WPAN). The communication between the HMD 1201 and the image processing device 1210 is not limited to wireless communication, and may be realized by wire.

[0014] The image processing device 1210 reproduces an image. The image processing device 1210 can also store images to be reproduced. The image processing device 1210 communicates with the HMD 1201. The image processing device 1210 includes an operation unit 1104 such as a keyboard. When data or a command is input to the image processing device 1210, the image processing device 1210 displays an image on the display device 1102 according to the input contents.

[0015] 2 is a functional block diagram of the image display system 1. The HMD 1201 has an imaging unit 1202 (imaging units 1202L and 1202R), a position and orientation calculation unit 1203, a communication unit 1204, and a display unit 1205 (display units 1205L and 1205R). The HMD 1201 has a synthesis unit 1206, a depth determination unit 1207, a CG depth calculation unit 1208, an imaging depth calculation unit 1209, and other functional units.

[0016] The imaging units 1202L and 1202R each capture a real space from a position similar to that of a human eye, and acquire an image of the real space (hereinafter referred to as a "real image"). Hereinafter, the image captured by the imaging unit 1202L of the real space is referred to as a "left-eye image," and the image captured by the imaging unit 1202R of the real space is referred to as a "right-eye image."

[0017] The position and orientation calculation unit 1203 calculates the position and orientation of the HMD 1201 based on the right-eye image and the left-eye image. The position and orientation calculation unit 1203 transmits information on the position and orientation of the HMD 1201 (hereinafter referred to as “position and orientation information”) to the image processing device 1210 via the communication unit 1204.

[0018] The communication unit 1204 transmits and receives images, control signals, and the like.

[0019] The display unit 1205L and the display unit 1205R each have a display device (such as a liquid crystal or organic EL display). The display unit 1205L and the display unit 1205R each control the display device to display an image (perform display control).

[0020] The synthesis unit 1206 synthesizes a real image with a CG image (image of a virtual object) acquired from the image processing device 1210 .

[0021] The depth determination unit 1207 compares the depth information of the real image with that of the CG image. Here, the "depth information" is information that indicates, for each pixel of an image, the depth (distance) from a reference position of the image (the position of the imaging device) to an object depicted in the pixel.

[0022] The CG depth calculation unit 1208 calculates (acquires) depth information of each of the two CG images (a CG to be synthesized with a left-eye image and a CG to be synthesized with a right-eye image) based on the two CG images acquired from the image processing device 1210. The CG depth calculation unit 1208 calculates the depth information of each of the two CG images by the same method as the imaging depth calculation unit 1209 (a method described later).

[0023] The imaging depth calculation unit 1209 calculates (acquires) depth information of the left eye image and the right eye image. The imaging unit 1202L is disposed at a position similar to that of a person's left eye, and the imaging unit 1202R is disposed at a position similar to that of a person's right eye. Therefore, the left eye image and the right eye image have a parallax with each other. Therefore, the imaging depth calculation unit 1209 calculates the depth to a real object (target object) at each pixel of each image by comparing the left eye image and the right eye image.

[0024] Methods for calculating depth from two images having parallax include the SAD (Sum of Absolute Difference) method and the SGM (Semi Global Matching) method. However, the method for calculating depth from two images having parallax is not particularly limited, and other methods may be used. The imaging depth calculation unit 1209 calculates the depth for each pixel of the left eye image and the right eye image.

[0025] In this way, the imaging depth calculation unit 1209 calculates the real object depth based on the right-eye image and the left-eye image. The depth to the object (subject) is calculated. However, as shown in FIG. 3, a distance sensor unit 1301 may measure the depth to the object. Then, the imaging depth calculation unit 1209 may calculate (acquire) depth information based on a measurement value of the distance sensor unit 1301. The distance sensor unit 1301 is a sensor other than the imaging unit 1202. The distance sensor unit 1301 is, for example, a LiDAR (Light Detection And Ranging), a ToF (Time of Flight), or a millimeter wave radar.

[0026] The image processing device 1210 is an external device different from the HMD 1201. The image processing device 1210 is a PC, a WS (workstation), a cloud (a cloud connectable to the HMD 1201 via a public network, etc.), etc. The image processing device 1210 has a communication unit 1211, a content DB (content database) 1213, a CG drawing unit 1212, and other functional units.

[0027] The communication unit 1211 transmits and receives images, control signals, and the like.

[0028] The CG rendering unit 1212 renders a CG image corresponding to a CG (virtual object) stored in the content DB 1213 in accordance with the position and orientation of the HMD 1201 .

[0029] The content DB 1213 stores CG (CG content).

[0030] In the first embodiment, the HMD 1201 transmits position and orientation information to the image processing device 1210, and the image processing device 1210 renders a CG image. However, for example, an external measurement sensor may measure the position and orientation of the HMD 1201 and transmit the position and orientation information to the image processing device 1210. Alternatively, the image processing device 1210 may calculate the position and orientation of the HMD 1201.

[0031] In the above configuration, a process flow will be described for displaying a composite image obtained by combining a CG image drawn by the image processing device 1210 with a real image acquired by the imaging unit 1202. By wearing the HMD 1201, a user can view a composite image according to his / her own position and posture.

[0032] (1) The position and orientation calculation unit 1203 calculates the position and orientation of the HMD 1201 based on the left-eye image and the right-eye image. Information on the position and orientation of the HMD 1201 (position and orientation information) is transmitted to the image processing device 1210 via the communication unit 1204.

[0033] (2) The CG rendering unit 1212 renders a CG image for the left eye and a CG image for the right eye based on the position and orientation information. After that, the CG image for the left eye (hereinafter referred to as the "left CG image") and the CG image for the right eye (hereinafter referred to as the "right CG image") are transmitted to the HMD 1201 via the communication unit 1211. Since the left CG image is an image to be synthesized with the left eye image and the right CG image is an image to be synthesized with the right eye image, the left CG image and the right CG image have a parallax with each other. Note that the left CG image and the right CG image have the same screen resolution (number of pixels) as the left eye image and the right eye image, respectively.

[0034] (3) The composition unit 1206 composes the left CG image with the left-eye image, and composes the right CG image with the right-eye image. A common composition method is to "use pixels of the real image in areas where the color of the CG image is a specified chromakey color, and use pixels of the CG image in areas where the color of the CG image is not the chromakey color." The composition unit 1206 can also receive an alpha value representing transparency information from the image processing device 1210 along with the CG image, and use this information to compose the real image and the CG image.

[0035] (4) The display unit 1205L displays an image obtained by combining a left CG image and a left-eye image (hereinafter referred to as a "left combined image"), and the display unit 1205R displays an image obtained by combining a right CG image and a right-eye image (hereinafter referred to as a "right combined image").

[0036] 4A to 4D are diagrams for explaining the synthesis performed by the synthesis unit 1206. In Fig. 4A to 4D, synthesis is explained using an image corresponding to only the left eye.

[0037] Fig. 4A shows a CG 1401 of a cylinder in a left CG image acquired from the image processing device 1210. Fig. 4B shows a left-eye image captured by the imaging unit 1202L of the real space. A table 1411, which is a real object, is captured in the left-eye image.

[0038] Here, when the CG 1401 is located closer to the viewer than the table 1411 (closer to the HMD 1201), a left composite image as shown in Fig. 4C may be generated. That is, a left composite image is generated in which part of the table 1411 is hidden by the CG 1401. On the other hand, when the CG 1401 is located further back than the table 1411, a left composite image as shown in Fig. 4D may be generated.

[0039] In order to realize such a synthesis process, information on the depth from the HMD 1201 to the CG 1401 and information on the depth from the HMD 1201 to the table 1411 are required.

[0040] For this reason, a depth determination unit 1207 compares the depth to the object at each pixel of the left eye image (hereinafter referred to as "object depth") with the depth to the CG at each pixel of the left CG image (hereinafter referred to as "CG depth"). A synthesis unit 1206 synthesizes the left CG image and the left eye image according to the object depth and the CG depth so that the object or the CG closer to the HMD 1201 is displayed in front.

[0041] The process of combining a CG image with a real image will be described with reference to the flowchart of FIG.

[0042] In step S1501, the imaging depth calculation unit 1209 calculates depth information of the left-eye image (object depth for each pixel) and depth information of the right-eye image based on the right-eye image and the left-eye image.

[0043] In step S1502, the CG depth calculation unit 1208 calculates depth information of the left CG image (CG depth for each pixel) and depth information of the right CG image based on the left CG image and the right CG image.

[0044] In step S1503, the depth determination unit 1207 compares the depth information of the left-eye image with the depth information of the left CG image for each corresponding pixel. Similarly, the depth determination unit 1207 compares the depth information of the right-eye image with the depth information of the right CG image for each corresponding pixel.

[0045] In step S1504, the synthesis unit 1206 synthesizes the left-eye image and the left CG image such that, for each pixel of the left synthesized image, the pixel of the image that indicates the smaller depth, either the object depth of the left eye image or the CG depth of the left CG image, is used. Similarly, the synthesis unit 1206 synthesizes the right-eye image and the right CG image such that, for each pixel of the right synthesized image, the pixel of the image that indicates the smaller depth, either the object depth of the right eye image or the CG depth of the right CG image, is used. In this way, the synthesis unit 1206 generates the left synthesized image and the right synthesized image.

[0046] It is not essential to generate the left and right composite images. For example, the display unit 1205L may display the smaller depth between the depth of the left simple image and the CG depth of the left CG image in each pixel. If the image is a left CG image, the pixels of the left CG image may be displayed. On the other hand, if the image showing the smaller depth of the left simple image or the CG depth of the left CG image is the left simple image, the display unit 1205L may display nothing in that pixel. In other words, the display unit 1205L may display only the CG that exists in front of the real object without displaying any real object.

[0047] According to the first embodiment, the HMD calculates the depth information of the CG image (the HMD does not receive the depth information of the CG image from the image processing device), so that the amount of communication between the HMD and the image processing device can be reduced. Also, it becomes possible to synthesize an image that more accurately expresses the depth relationship between the real image and the CG image. Therefore, the HMD is less susceptible to the effects of communication delays and can display a more appropriate synthetic image in real time.

[0048] <Embodiment 2> In the first embodiment, an example of displaying an MR image obtained by combining a real image and a CG image has been described. In the second embodiment, an example of displaying a VR (Virtual Reality) image obtained by combining two CG images will be described.

[0049] 6 is a functional block diagram of an image display system 2 according to embodiment 2. In embodiment 2, an HMD 1201 further includes a CG rendering unit 1601 and a calculation unit 1602 in addition to the functional units of the HMD 1201 according to embodiment 1. Also, a position and orientation calculation unit 1203 transmits position and orientation information of the HMD 1201 to the CG rendering unit 1601.

[0050] The calculation unit 1602 calculates the position and orientation of the controller held by the user's hand. In this embodiment, the calculation unit 1602 calculates the position and orientation of the controller based on the left-eye image and the right-eye image, similar to the case of calculating the position and orientation of the HMD 1201. The calculation unit 1602 may receive information on the position and orientation of the controller measured by an external measurement sensor or the like. Alternatively, the calculation unit 1602 may obtain information on the result of the controller's own calculation of its own position and orientation.

[0051] The CG rendering unit 1601 renders a CG image based on the position and orientation of the HMD 1201 and the controller. In the second embodiment, each of the image processing device 1210 and the HMD 1201 has a CG rendering unit. For example, the CG rendering unit 1212 renders a CG image (such as a background image) that spreads across the entire screen of the display unit 1205, and the CG rendering unit 1601 renders a simple CG image (such as an image of a controller held in a hand). Hereinafter, the CG image for the left eye generated by the CG rendering unit 1601 is referred to as a "left simple image," and the CG image for the right eye generated by the CG rendering unit 1601 is referred to as a "right simple image."

[0052] In the second embodiment, the two CG images (left simple image and right simple image) rendered by the CG rendering unit 1601 and the depth information of these two CG images are output to the depth determination unit 1207. The depth determination unit 1207 determines the CG that is closer to the HMD 1201, based on the depth information calculated by the CG depth calculation unit 1208 (depth information of the CG image acquired from the image processing device 1210) and the depth information of the CG image rendered by the CG rendering unit 1601. The synthesis unit 1206 synthesizes the left simple image with the left CG image, and also synthesizes the right simple image with the right CG image, based on the determination result of the depth determination unit 1207.

[0053] The flow chart of Fig. 7 explains the process of combining two CG images. Note that step S1502 is the same as step S1502 in Fig. 5, so the explanation will be omitted.

[0054] In step S1701, the CG rendering unit 1601 The CG drawing unit 1601 draws a right simple image and a left simple image based on the position and posture of the controller. The CG drawing unit 1601 also obtains (calculates) depth information of the right simple image and depth information of the left simple image.

[0055] In step S1702, the depth determination unit 1207 compares the depth information of the left simple image with the depth information of the left CG image for each corresponding pixel. Similarly, the depth determination unit 1207 compares the depth information of the right simple image with the depth information of the right CG image for each corresponding pixel.

[0056] In step S1703, the composition unit 1206 composes the left simple image and the left CG image such that, for each pixel of the left composite image, the pixel of the image showing the smaller depth between the depth of the left simple image and the CG depth of the left CG image is used. Similarly, the composition unit 1206 composes the right simple image and the right CG image such that, for each pixel of the right composite image, the pixel of the image showing the smaller depth between the depth of the right simple image and the CG depth of the right CG image is used. In this way, the composition unit 1206 generates the left composite image and the right composite image.

[0057] According to the second embodiment, in a system in which two devices each render a CG image, a composite image that correctly represents the depth relationship of the CG images can be synthesized without transmitting depth information from the image processing device to the HMD, thereby reducing the amount of communication between the HMD and the image processing device.

[0058] <Embodiment 3> In the third and subsequent embodiments, the image processing device 1210 transmits to the HMD 1201 information obtained by reducing the amount of information of the depth information of the left CG image and the depth information of the right CG image. Alternatively, the image processing device 1210 transmits to the HMD 1201 only one of the depth information of the left CG image and the depth information of the right CG image. For this reason, information (depth information; hereinafter referred to as "compressed information") having an amount of information smaller than the total amount of information of the depth information of the left CG image and the depth information of the right CG image is transmitted from the image processing device 1210 to the HMD 1201. Note that, hereinafter, information having depth information in pixel units (each of all pixels) in an image is referred to as "depth information", and information after the amount of information of the depth information has been reduced is not referred to as "depth information".

[0059] In the third embodiment, the image processing device 1210 transmits to the HMD 1201 information obtained by reducing the amount of depth information of the left CG image and the amount of depth information of the right CG image together as compressed information.

[0060] Fig. 8 is a functional block diagram of an image display system 3 according to embodiment 3. Unlike the image display system 2 shown in Fig. 2, the CG depth calculation unit 1208 is deleted and a depth information calculation unit 1801 is added. In addition, the image processing device 1210 transmits two CG images and compressed information to the HMD 1201.

[0061] In the third embodiment, the CG rendering unit 1212 acquires (calculates) depth information of the CG images (left CG image and right CG image) and reduces the amount of information of the depth information. The depth information includes information on a pixel-by-pixel basis of the CG image to indicate the depth of each pixel of the CG. In the third embodiment, information with the amount of depth information of the two CG images reduced is sent as compressed information from the image processing device 1210 to the HMD 1201. For this reason, the depth information calculation unit 1801 acquires (calculates) the depth information of the two CG images based on the compressed information.

[0062] 9A and 9B are diagrams for explaining an example of reducing the amount of information. In the third embodiment, the CG drawing unit 1212 reduces the vertical and horizontal screen resolution of the depth information (the number of pixels whose depth is indicated by the depth information).

[0063] 9A and 9B show an example of drawing a CG of a car, and are conceptual diagrams in which the depth information is provisionally replaced with a picture. For example, when the screen resolution (total number of pixels) of the left CG image is 3840×2160, the depth information has information for each pixel of the left CG image, so the screen resolution of the depth information is 3840×2160. Therefore, after the screen resolution of the depth information is reduced to 1920×1080 (0.5 times vertical and horizontal) or 960×540 (0.25 times vertical and horizontal), information with the reduced screen resolution of the depth information (compressed information) is transmitted from the image processing device 1210 to the HMD 1201. In other words, the depth information having the depth information of N pixels is converted into information having the depth information of M pixels (N / 4 or N / 16, etc.) which is smaller than N, and the converted information is transmitted to the HMD 1201.

[0064] The depth information calculation unit 1801 restores depth information to the screen resolution before the amount of information is reduced, based on the compression information acquired from the image processing device 1210. The method of reducing the screen resolution of the depth information may be any method such as simple thinning or averaging. On the other hand, the method of restoring the original screen resolution by the depth information calculation unit 1801 may be any method such as nearest neighbor or bilinear.

[0065] 9B shows an example in which the amount of information relating to the central region 1901 among the depth information is not reduced, but the amount of information relating to the area other than the central region 1901 is reduced. When a user wears the HMD 1201 and views an image, the user often looks at the center of the image. For this reason, the CG rendering unit 1212 does not reduce the amount of information relating to the central region of the image, but reduces the amount of information relating to the peripheral regions. This makes it possible to generate a composite image in which the depth relationship is more accurately expressed in the central region of the image.

[0066] As described above, the depth information usually requires information on a pixel-by-pixel basis. Therefore, when the depth information is 8-bit information for each pixel, the image processing device 1210 may reduce the information to 4-bit, for example.

[0067] The method for reducing the amount of depth information may be determined by negotiation at the start of connection between the image processing device 1210 and the HMD 1201. Furthermore, the image processing device 1210 may send data indicating the method for reducing the amount of information when transmitting compressed information to the HMD 1201.

[0068] The synthesis process according to the third embodiment will be described with reference to the flowchart of Fig. 10. The processes in steps S1501, S1503, and S1504 are similar to those in the steps with the same reference numerals shown in Fig. 5, and therefore will not be described.

[0069] In step S2001, the depth information calculation unit 1801 calculates (restores) the depth information of the left CG image and the depth information of the right CG image based on the compressed information. The compressed information is information (combined information) including information after the amount of depth information of the left CG image has been reduced and information after the amount of depth information of the right CG image has been reduced.

[0070] In addition, a method for reducing the amount of information of depth information will be explained in the third and subsequent embodiments. However, the amount of information of depth information may be further reduced by adding a technique such as encoding to the reduction method described in the third and subsequent embodiments.

[0071] According to the third embodiment, even when information regarding depth is transmitted to the HMD, the amount of information of the depth information is reduced, so that the amount of communication between the image processing device and the HMD can be reduced. Furthermore, by using the information of the depth information with reduced amount of information, an image that correctly expresses the depth relationship of the CG image can be synthesized.

[0072] <Embodiment 4> In the third embodiment, an example was described in which the amount of information is reduced by reducing the screen resolution or bit depth of the depth information. Here, the CG image and the real image are each formed from images of multiple frames. Therefore, in the fourth embodiment, the CG drawing unit 1212 reduces the amount of information of the depth information (depth information in the time direction) for each predetermined number of frames (fixed time period). In other words, when viewed in units of multiple frames (predetermined period), it can be said that information in which the amount of information of the depth information of the CG image is reduced is transmitted to the HMD 1201 as compressed information.

[0073] 11 is a functional block diagram of an image display system 4 according to embodiment 4. Unlike the image display system 3 shown in FIG. 8, information on the position and orientation of the HMD 1201 calculated by the position and orientation calculation unit 1203 (position and orientation information) is output to a depth information calculation unit 1801.

[0074] The depth information calculation unit 1801 obtains depth information of two CG images based on the position and orientation information of the HMD 1201 and the compressed information obtained from the image processing device 1210 .

[0075] 12 is a diagram for explaining a process of reducing the amount of information in the time direction. When time t is 0 or 2, the image processing device 1210 transmits the image of the CG image frame at that time and the depth information to the HMD 1201. On the other hand, when time t is 1 or 3, the image processing device 1210 transmits only the image of the CG image frame at that time, and does not transmit the depth information to the HMD 1201.

[0076] In the following, an example will be described in which the image processing device 1210 transmits depth information once every two times (two frames), but the image processing device 1210 may transmit the depth information at any pace, such as once every three times (three frames).

[0077] In this example, the depth information of a frame where time t is 1 is obtained by correcting the depth information of a frame where time t is 0. Based on the amount of change in the position and orientation of the HMD 1201 during the period when time t changes from 0 to 1, the depth information calculation unit 1801 performs processing called time warp or reprojection on the depth information of the frame where time t is 0. In the fourth embodiment, the depth information calculation unit 1801 corrects the depth information based on position and orientation information acquired based on a real image, but the depth information may also be corrected based on the angular velocity of the HMD 1201 (information acquired by a Gyro sensor) or the like.

[0078] The synthesis process according to the fourth embodiment will be described with reference to the flowchart in Fig. 13. The processes of steps S1501, S1503, and S1504 are similar to the processes of the steps with the same reference numerals shown in Fig. 5, and therefore will not be described. The process of the flowchart in Fig. 13 is executed at a time (frame) when depth information is not transmitted from the image processing device 1210. On the other hand, at a time (frame) when depth information is transmitted from the image processing device 1210, the process of the flowchart in Fig. 5 is executed.

[0079] In step S2301, the depth information calculation unit 1801 performs processing called time warp or reprojection on the depth information of the immediately previous frame (depth information of the left CG image and the right CG image) based on the position and orientation information of the HMD 1201. In this way, the depth information calculation unit 1801 acquires (calculates) the depth information of the current frame of the left CG image and the right CG image.

[0080] According to the fourth embodiment, the amount of communication between the image processing device and the HMD can be reduced by reducing the depth information in the time direction.

[0081] <Embodiment 5> In the third and fourth embodiments, it is necessary to transmit information with a reduced amount of depth information for the left CG image and information with a reduced amount of depth information for the right CG image. On the other hand, in the fifth embodiment, the image processing device 1210 does not transmit either the information related to the left CG image or the information related to the right CG image. The configuration of the image display system 5 according to the fifth embodiment is similar to the configuration of the image display system 3 shown in FIG.

[0082] Fig. 14A shows an example of compressed information transmitted from the image processing device 1210 according to the fifth embodiment to the HMD 1201. Meanwhile, Fig. 14B shows an example of compressed information transmitted from the image processing device 1210 described in the third and fourth embodiments to the HMD 1201. Thus, while information on two CG images is transmitted in the third and fourth embodiments, in the fifth embodiment, only depth information on one CG image is transmitted to the HMD 1201 as compressed information.

[0083] The depth information calculation unit 1801 of the HMD 1201 calculates the depth information of the other image based on the compressed information (depth information of one of the left and right CG images) received from the image processing device 1210. A general perspective projection transformation technique can be used to calculate this depth information of the other image. For example, the depth information of the right CG image can be generated by performing viewpoint transformation processing of perspective projection transformation on the depth information of the left CG image.

[0084] The synthesis process according to the fifth embodiment will be described with reference to the flowchart of Fig. 15. The processes in steps S1501, S1503, and S1504 are similar to those in the steps with the same reference numerals shown in Fig. 5, and therefore will not be described.

[0085] In step S2501, the depth information calculation unit 1801 calculates depth information of the other image based on the compressed information (depth information of one of the left CG image and the right CG image).

[0086] In the fifth embodiment, a method has been described in which the depth information of one of the left and right CG images is transmitted as compressed information, and the depth information of the other is calculated by perspective projection transformation using this depth information. However, average or total information of the depth information of the left CG image and the depth information of the right CG image may be transmitted as compressed information. Then, the depth information calculation unit 1801 may perform perspective projection transformation based on this average or total information to calculate the depth information of the left CG image and the depth information of the right CG image.

[0087] According to the fifth embodiment, since the depth information of only one of the left CG image and the right CG image is transmitted, it is possible to reduce the amount of communication between the image processing device and the HMD.

[0088] <Embodiment 6> In the first embodiment, a method for synthesizing an image that correctly expresses the depth relationship between a real image and a CG image without transmitting depth information to an HMD has been described. In the third to fifth embodiments, a method for transmitting information with a reduced amount of depth information has been described.

[0089] In the sixth embodiment, the HMD 1201 provisionally calculates depth information of the left CG image and depth information of the right CG image based on the left CG image and the right CG image (hereinafter, such depth information of the left CG image and depth information of the right CG image are collectively referred to as "calculated information"). Furthermore, the HMD 1201 acquires information in which the depth information of the left CG image has been reduced and information in which the amount of depth information of the right CG image has been reduced as compressed information from the image processing device 1210. The HMD 1201 acquires (calculates) more accurate depth information (depth information of the left CG image and depth information of the right CG image) based on the calculated information and compressed information.

[0090] FIG. 16 is a functional block diagram of an image display system 6 according to a sixth embodiment. A depth information calculation unit 2601 is added to the configuration shown in FIG. 2. The depth information calculation unit 2601 The position and orientation information of the HMD 1201 is obtained from the position and orientation calculation unit 1203. In the sixth embodiment, the image processing device 1210 transmits to the HMD 1201 compressed information in which the amount of depth information of two CG images is reduced.

[0091] The CG depth calculation unit 1208 calculates calculation information (depth information of the left CG image and depth information of the right CG image) based on the left CG image and the right CG image. Here, the depth information calculated by the CG depth calculation unit 1208 may have an error due to the accuracy of the image or an error due to the algorithm for calculating the depth. For example, when SAD (Sum of Absolute Difference) is used, matching is performed in which the sum of absolute values ​​of luminance differences is calculated for each block. Therefore, when calculating depth using SAD, depth may not be calculated correctly for areas with small luminance differences in the CG image. Therefore, the image processing device 1210 transmits compressed information (information in which the amount of information of depth information is reduced) to the HMD 1201 as information for correcting the calculation information. As a method for reducing the amount of information of depth information, the method described in any of the third to fifth embodiments can be adopted.

[0092] The depth information calculation unit 2601 corrects information in an area with low calculation accuracy (calculation accuracy lower than a predetermined accuracy) of the calculated information based on compressed information with a reduced amount of information.

[0093] The synthesis process according to the sixth embodiment will be described with reference to the flowchart of Fig. 17. The processes in steps S1501, S1503, and S1504 are similar to those in the steps with the same reference numerals shown in Fig. 5, and therefore will not be described.

[0094] In step S2701, CG depth calculation unit 1208 calculates calculation information (temporary depth information) based on the left CG image and the right CG image, similarly to step S1502.

[0095] In step S2702, the depth information calculation unit 2601 corrects the calculation information based on the compression information to obtain more accurate depth information of the left CG image and the right CG image. Specifically, the depth information calculation unit 2601 corrects information of an area with low calculation accuracy in the calculation information based on compressed information obtained from the image processing device 1210, which has been reduced in information.

[0096] According to the sixth embodiment, it is possible to synthesize an image that more accurately represents the depth relationship of the CG image, and further to reduce the amount of communication between the image processing device and the HMD.

[0097] <Embodiment 7> In the seventh embodiment, the image processing device 1210 switches the method of generating compressed information (method of reducing the amount of depth information) depending on the expected use scene of the HMD 1201. In other words, even when transmitting the same CG image, the image processing device 1210 switches the compressed information (the compressed information to be generated) depending on the expected use scene of the HMD 1201.

[0098] Fig. 18 is a table showing combinations of "envisioned usage scenes of the HMD 1201" and "methods of generating compressed information" in embodiment 7. Assumed scenes include, as shown in Fig. 18, a scene in which CG is displayed with a wider angle of view than a predetermined width, a scene in which CG that exists within a certain distance range is displayed, a scene in which the movement of the HMD 1201 is slow, a scene in which CG is displayed farther away than a predetermined distance, and the like.

[0099] For example, in a scene where the HMD 1201 does not move much or the movement is slow, there is little change in the image and depth information over time. For this reason, the image processing device 1210 reduces the amount of depth information in the time direction as in the fourth embodiment. When the HMD 1201 displays a CG image that is relatively far away, the parallax between the two CG images is small. For this reason, the image processing device As in the fifth embodiment, the image processing device 1210 transmits only the depth information of one side of the CG image as compressed information. On the other hand, in a scene in which CG within a certain distance range from the HMD 1201 is displayed on the HMD 1201, the depth indicated by the depth information is concentrated in that range. For this reason, the image processing device 1210 reduces the depth (amount of information per pixel). In this way, the image processing device 1210 switches the method of generating compressed information depending on the scene in which the HMD 1201 is used.

[0100] The timing at which the compressed information generation method is switched may be when the system is started, or when an application that renders CG is started. Alternatively, the timing at which the compressed information generation method is switched may be when the usage scene is changed. In this case, the image processing device 1210 determines the compressed information generation method through negotiation between the image processing device 1210 and the HMD 1201 at the timing at which the usage scene is changed. The image processing device 1210 may switch the compressed information generation method for each frame of a CG image. In this case, the image processing device 1210 notifies the HMD 1201 of information on the compressed information generation method before each frame.

[0101] In the seventh embodiment, an example in which one generation method is used according to the usage scene has been described, but multiple methods may be used when multiple usage scenes overlap. For example, when the HMD 1201 displays a distant CG image and also displays a CG image within a certain distance range on the HMD 1201, the image processing device 1210 may use a method of transmitting information of only one CG image with a reduced depth of depth information.

[0102] In the seventh embodiment, the amount of depth information can be appropriately reduced according to the usage scene, so that the amount of communication between the image processing device and the HMD can be reduced. Furthermore, the HMD can synthesize an image that correctly expresses the depth relationship of the CG image.

[0103] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0104] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." Therefore, unless a contradiction occurs, "equal to or greater than A" may be read as "equal to or greater than A (high; long; many)," and "equal to or less than A" may be read as "equal to or less than A (low; short; few)." And, "equal to or greater than A" may be read as "equal to or greater than A," and "equal to or less than A" may be read as "equal to or less than A."

[0105] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0106] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0107] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) A display device having a first display means and a second display means and worn on a user's head, A receiving means for receiving a first image and a second image having a parallax from an external device; a first calculation means for calculating at least one of depth information of the first image and depth information of the second image; a display control means for controlling a first display means to display an image based on the depth information of the first image, and for controlling a second display means to display an image based on the depth information of the second image; A display device comprising: (Configuration 2) the first calculation means calculates depth information of the first image and depth information of the second image based on the first image and the second image; 2. The display device according to configuration 1. (Configuration 3) The display device does not receive at least one of the depth information of the first image and the depth information of the second image from the external device. 3. The display device according to configuration 1 or 2. (Configuration 4) the display device does not receive any of the depth information of the first image and the depth information of the second image from the external device; 3. The display device according to configuration 1 or 2. (Configuration 5) the receiving means further receives from the external device first information, the first information being information about at least one of a depth in the first image and a depth in the second image, the amount of information being less in a predetermined period of time than a sum of an amount of information of the depth information of the first image and an amount of information of the depth information of the second image in a predetermined period of time; The first calculation means acquires depth information of the first image and depth information of the second image based on the first information. 4. The display device according to any one of configurations 1 to 3. (Configuration 6) The depth information of the first image and the depth information of the second image are each information indicating depths of N pixels, the first information includes second information regarding the first image, the second information indicating a depth of M pixels, the M being less than N pixels; The first calculation means calculates depth information of the first image based on the second information. 6. The display device according to configuration 5. (Configuration 7) the first information includes third information in which an amount of information of a central region of the first image is not reduced and an amount of information corresponding to a region not including the central region is reduced among depth information of the first image; The first calculation means calculates depth information of the first image based on the third information. R, 7. The display device according to configuration 5 or 6. (Configuration 8) each of the first image and the second image has a plurality of frames of images during the predetermined period; the first information includes depth information of a first frame of the first image and does not include depth information of a second frame of the first image during the predetermined period; the first calculation means calculates depth information of the second frame of the first image based on depth information of the first frame of the first image; 6. The display device according to configuration 5. (Configuration 9) the first information does not include depth information of the first image but includes depth information of the second image; The first calculation means calculates the depth information of the first image based on the depth information of the second image. 8. The display device according to any one of configurations 5 to 7. (Configuration 10) the first calculation means calculates depth information of the first image and depth information of the second image by correcting information calculated based on the first image and the second image based on the first information; 10. The display device according to any one of configurations 5 to 9. (Configuration 11) the first information is generated by the external device based on at least one of depth information of the first image and depth information of the second image; the external device switches the first information to be generated depending on a usage scene of the display device. 11. The display device according to any one of configurations 5 to 10. (Configuration 12) A generating means for generating a third image and a fourth image having a parallax with respect to each other; a second calculation means for calculating depth information of the third image and depth information of the fourth image; a synthesis means for generating a fifth image by synthesizing the first image and the third image based on depth information of the first image and depth information of the third image, and for generating a sixth image by synthesizing the second image and the fourth image based on depth information of the second image and depth information of the fourth image; and the display control means controls the first display means to display the fifth image, and controls the second display means to display the sixth image. 12. The display device according to any one of configurations 1 to 11. (Configuration 13) The generating means has a first imaging means and a second imaging means, the third image is an image captured by the first imaging means of a real space, The fourth image is an image of the real space captured by the second imaging means. 13. The display device according to configuration 12. (Configuration 14) The third image and the fourth image are each an image of a virtual object. 13. The display device according to configuration 12. (Configuration 15) the second calculation means calculates depth information of the third image and depth information of the fourth image based on the third image and the fourth image, or based on a depth to a subject measured by a distance sensor; 15. The display device according to any one of configurations 12 to 14. (Configuration 16) 16. The display device according to configuration 15, wherein the distance sensor is a LiDAR (Light Detection And Ranging), a ToF (Time of Flight), or a millimeter wave radar. (Configuration 17) The depth information is information indicating the depth of each pixel of the image. 17. The display device according to any one of configurations 1 to 16. (method) A method for controlling a display device having a first display means and a second display means and worn on a user's head, comprising the steps of: A receiving step of receiving a first image and a second image having a parallax from an external device; a first calculation step of calculating at least one of depth information of the first image and depth information of the second image; a display control step of controlling a first display means to display an image based on the depth information of the first image, and controlling a second display means to display an image based on the depth information of the second image; 13. A method for controlling a display device comprising the steps of: (program) A program for causing a computer to function as each of the means of the display device according to any one of configurations 1 or 15. [Explanation of symbols]

[0108] 1201: HMD (display device), 1210: image processing device (external device), 1204: communication unit, 1208: CG depth calculation unit, 1205L,1205R:Display section

Claims

1. A display device having a first display means and a second display means and worn on a user's head, a receiving means for receiving a first image and a second image having a parallax from an external device; a first calculation means for calculating at least one of depth information of the first image and depth information of the second image; a display control means for controlling a first display means to display an image based on depth information of the first image, and for controlling a second display means to display an image based on depth information of the second image; A display device comprising:

2. the first calculation means calculates depth information of the first image and depth information of the second image based on the first image and the second image; 2. The display device according to claim 1 .

3. the display device does not receive at least one of the depth information of the first image and the depth information of the second image from the external device; 3. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

4. the display device does not receive any of the depth information of the first image and the depth information of the second image from the external device; 3. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

5. the receiving means further receives from the external device first information, the first information being information regarding at least one of a depth in the first image and a depth in the second image, the amount of information being less in a predetermined period of time than a sum of an amount of information of the depth information of the first image and an amount of information of the depth information of the second image in a predetermined period of time; The first calculation means acquires depth information of the first image and depth information of the second image based on the first information.

3. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

6. The depth information of the first image and the depth information of the second image are each information indicating depths of N pixels, the first information includes second information regarding the first image, the second information indicating a depth of M pixels, the M being less than N; The first calculation means calculates depth information of the first image based on the second information.

6. The display device according to claim 5.

7. the first information includes third information in which an amount of information of a central region of the first image is not reduced and an amount of information corresponding to a region not including the central region is reduced, among depth information of the first image; The first calculation means calculates depth information of the first image based on the third information.

6. The display device according to claim 5.

8. each of the first image and the second image has a plurality of frames of images in the predetermined period; The first information includes a depth of a first frame of the first image during the predetermined period. information and does not include depth information of the second frame of the first image; the first calculation means calculates depth information of the second frame of the first image based on depth information of the first frame of the first image; 6. The display device according to claim 5.

9. the first information does not include depth information of the first image but includes depth information of the second image; The first calculation means calculates depth information of the first image based on depth information of the second image.

6. The display device according to claim 5.

10. the first calculation means calculates depth information of the first image and depth information of the second image by correcting information calculated based on the first image and the second image based on the first information; 6. The display device according to claim 5.

11. the first information is generated by the external device based on at least one of depth information of the first image and depth information of the second image; the external device switches the first information to be generated depending on a usage scene of the display device.

6. The display device according to claim 5.

12. A generating means for generating a third image and a fourth image having a parallax with respect to each other; a second calculation means for calculating depth information of the third image and depth information of the fourth image; a synthesis means for generating a fifth image by synthesizing the first image and the third image based on depth information of the first image and depth information of the third image, and for generating a sixth image by synthesizing the second image and the fourth image based on depth information of the second image and depth information of the fourth image; and the display control means controls the first display means to display the fifth image, and controls the second display means to display the sixth image.

3. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

13. The generating means includes a first imaging means and a second imaging means, the third image is an image captured by the first imaging means of a real space, The fourth image is an image of the real space captured by the second imaging means.

13. The display device according to claim 12.

14. the third image and the fourth image are each an image of a virtual object; 13. The display device according to claim 12.

15. the second calculation means calculates depth information of the third image and depth information of the fourth image based on the third image and the fourth image, or based on a depth to a subject measured by a distance sensor; 13. The display device according to claim 12.

16. 16. The display device according to claim 15, wherein the distance sensor is a Light Detection And Ranging (LiDAR), a Time of Flight (ToF), or a millimeter wave radar.

17. The depth information is information indicating the depth of each pixel of the image.

3. The display device according to claim 1, wherein the first and second electrodes are arranged in a first direction.

18. A method for controlling a display device having a first display means and a second display means and worn on a user's head, comprising the steps of: A receiving step of receiving a first image and a second image having a parallax from an external device; a first calculation step of calculating at least one of depth information of the first image and depth information of the second image; a display control step of controlling a first display means to display an image based on depth information of the first image, and controlling a second display means to display an image based on depth information of the second image; 13. A method for controlling a display device comprising the steps of:

19. A program for causing a computer to function as each of the means of the display device according to claim 1 or 2.

Images