Image creation device, display system, image creation method, and program

Abstract

To provide a technique that can generate image data with a small data size available for both an optical see-through system and a video see-through system.SOLUTION: An image creation device of the present invention generates image data of a virtual object to be displayed on a display of an optical see-through system or a video see-through system, and the image creation device has: generation means that generates output image data in which pixel values do not include transparency and a pixel value of the background is a specific pixel value interpreted as the pixel value of the background in both the display of the optical see-through system and the display of the video see-through system; and output means that outputs the output image data to the outside.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image generating device, a display system, an image generating method, and a program. [Background technology]

[0002] Mixed reality (MR) technology and augmented reality (AR) technology are known as technologies that seamlessly fuse real space and virtual space in real time. Virtual reality (VR) technology is also known as a technology that represents only virtual space. These technologies are sometimes used in head-mounted displays (HMDs), for example. For example, the position and orientation of the HMD are estimated, and an image (computer graphic (CG) image) of the virtual space as seen from the position and orientation according to the estimation result is displayed on the HMD.

[0003] There are two types of HMD: optical see-through and video see-through. With the optical see-through method, a virtual space image is displayed by superimposing (compositing) the virtual space image onto real space, which is not a video. With the video see-through method, the real space is imaged, the virtual space image is superimposed (composited) onto the real space image, and the superimposed image is displayed. Even if the CG image (virtual space image) to be displayed is the same, different video data (image data) must be used for the optical see-through and video see-through methods.

[0004] Patent Document 1 discloses an image generating device that generates RGBA data (video data in which each pixel value is an RGBA value) and transmits it to an HMD. In the RGBA values, the R value is a gradation value corresponding to red, the G value is a gradation value corresponding to green, the B value is a gradation value corresponding to blue, and the A value is a gradation value (alpha value) corresponding to transparency. Since the amount of information in RGBA data is large (compared to RGB data (video data in which each pixel value is an RGB value)), the load of transmitting RGBA data is large.

[0005] Patent Document 2 discloses an image generating device that performs chromakey processing on video data in which a virtual space image is superimposed on a real space image, and transmits the chromakey processed video data to a video see-through HMD. Even if this video data is used in an optical see-through HMD, the same CG image (virtual space image) as when used in a video see-through HMD is not displayed. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-95916 A [Patent Document 2] JP 2020-64592 A Summary of the Invention [Problem to be solved by the invention]

[0007] An object of the present invention is to provide a technique capable of generating image data with a small data size that can be used for both the optical see-through method and the video see-through method. [Means for solving the problem]

[0008] A first aspect of the present invention is an image generating device that generates image data of a virtual object to be displayed on an optical see-through or video see-through display device, and generates output image data in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that is interpreted as the pixel value of the background on both the optical see-through display device and the video see-through display device. and an output means for outputting the output image data to an external device.

[0009] A second aspect of the present invention is a display system including the above-mentioned image generation device and an optical see-through or video see-through display device, wherein the display device displays a virtual object based on output image data of the image generation device.

[0010] A third aspect of the present invention is an image generation method for generating image data of a virtual object to be displayed on an optical see-through or video see-through display device, the image generation method comprising the steps of: generating output image data in which each pixel value does not include transparency and in which the pixel values ​​of the background are specific pixel values ​​that can be interpreted as the pixel values ​​of the background on both an optical see-through display device and a video see-through display device; and outputting the output image data to the outside.

[0011] A fourth aspect of the present invention is a program for causing a computer to function as each of the means of the image generating device described above. Effect of the Invention

[0012] According to the present invention, it is possible to generate image data with a small data size that can be used for both the optical see-through method and the video see-through method. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a display system. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a display system. [Diagram 3] FIG. 1 is a diagram showing a configuration of a display system according to a first embodiment. [Figure 4] FIG. 11 is a diagram showing the configuration of a display system according to a second embodiment. [Diagram 5] FIG. 2 shows an image used in the display system of FIG. 1. [Figure 6] FIG. 3 shows an image used in the display system of FIG. 2. [Figure 7] FIG. 2 is a diagram showing an image used in the display system according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing an image used in a display system according to a second embodiment. [Figure 9] FIG. 11 is a diagram showing an image used in a display system according to a third embodiment. [Figure 10] FIG. 13 is a diagram showing an image used in a display system according to a fourth embodiment. [Figure 11] FIG. 13 is a diagram showing an image used in a display system according to a fifth embodiment. [Figure 12] FIG. 13 is a diagram showing an image used in the display system according to the sixth and seventh embodiments. [Figure 13] FIG. 13 is a diagram showing an image used in a display system according to an eighth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] (Embodiment 1) Fig. 1 is a block diagram showing the configuration of a display system including an image generating device 10 and an optical see-through head mounted display (HMD) 11. Fig. 5 is a schematic diagram showing an image (video) used in the display system of Fig. 1.

[0016] The image generating device 10 is a device that generates image data of a virtual object to be displayed on the HMD 11, and is, for example, a workstation, a personal computer, a tablet terminal, or a smartphone. The image generating device 10 may be a device that is worn by a user of the HMD 11 (for example, on the waist, shoulder, or back of the head). At least a part of the image generating device 10 may be provided in the HMD 11.

[0017] The position and orientation estimation unit 101 estimates at least one of the position and orientation of the HMD 11 based on information from the HMD 11. Since the load of this estimation is relatively large, the position and orientation estimation unit 101 is provided in the image generating device 10, which has many CPU and GPU resources. Note that the position and orientation estimation unit 101 may be provided in the HMD 11. Since the HMD 11 is worn on the head, the position and orientation of the HMD 11 can also be interpreted as the position and orientation of the user wearing the HMD 11.

[0018] The CG generation unit 102 performs rendering of computer graphics (CG) that are virtual objects based on the estimation result of the position and orientation estimation unit 101. As a result, an image (image data) with a black background is generated, such as image 50 in FIG. 5. The image data is, for example, RGB data (image data in which each pixel value is an RGB value). The R value is a gradation value corresponding to red, the G value is a gradation value corresponding to green, and the B value is a gradation value corresponding to blue. In the image 50, CG 501 and CG 502 are depicted, and the backgrounds of the CGs 501 and 502 are black. Then, the CG generation unit 102 outputs the generated image data to the HMD 11. Note that the CG generation unit 102 may generate an image that is not related to the position and orientation of the HMD 11.

[0019] The optical see-through type HMD 11 is a head-mounted display device that displays an image of a virtual space by superimposing (compositing) an image of a virtual object (virtual space) on a real space that is not an image. A part of the HMD 11 may be provided in an external device. The external device may be the image generating device 10, a device worn by the user of the HMD 11 (for example, on the waist, shoulder, or back of the head), or a device placed on a desk or floor.

[0020] The sensor unit 111 acquires information for estimating at least one of the position and orientation of the HMD 11. For example, the sensor unit 111 includes at least one of an acceleration sensor and an angular velocity sensor. The sensor unit 111 may include a geomagnetic sensor, an atmospheric pressure sensor, or an image sensor that captures an image of real space (the outside world). For example, the position and orientation estimation unit 101 detects markers or feature points from an image obtained by an image sensor, and estimates the position and orientation of the HMD 11 based on the detection result. The sensor unit 111 may have a sensor that obtains the distance to a reflection position from the time it takes for emitted light to be reflected and returned, such as a LiDAR sensor (for example, a TOF sensor).

[0021] The image correction unit 112 performs image processing on the image output from the image generation device 10 (CG generation unit 102). For example, when a delay occurs in the acquired image, the image correction unit 112 shifts or deforms the image based on information obtained by the sensor unit 111. This makes it possible to obtain an image with a pseudo-reduced delay. The image correction unit 112 may correct the resolution or color of the image, or may combine graphics (e.g., text) with the image.

[0022] The display unit 113 displays the image after the image processing by the image correction unit 112. The display unit 113 includes, for example, an LCD, LCOS, or OLED, and an eyepiece optical system. The display unit 113 may display an image based on the position and orientation of the HMD 11, or may display an image unrelated to the position and orientation of the HMD 11. A user wearing the HMD 11 can view a real space other than an image together with the image displayed on the display unit 113. For example, the display unit 113 has optical transparency, and a user wearing the HMD 11 can view a real space through the display unit 113.

[0023] The display unit 113 superimposes a virtual object on the real space by light, and the transmittance of light from the real space is maximum in the black area of ​​the displayed image. Therefore, when the image 50 in FIG. 5 is input to the HMD 11, the user wearing the HMD 11 visually recognizes the image 51. The black areas in image 50 are visible through the CGs 501 and 502, and real space is perceived. Furthermore, CGs 511 and 512 corresponding to CGs 501 and 502 are visible. Each pixel of CGs 511 and 512 is also visible with a transmittance according to the luminance of image 50. The lower the luminance of image 50, the higher the transmittance of image 51.

[0024] Fig. 2 is a block diagram showing the configuration of a display system including an image generating device 20 and a video see-through type HMD 21. Fig. 6 is a schematic diagram showing an image used in the display system of Fig. 2. In the explanation using Figs. 2 and 6, the contents common to the explanation using Figs. 1 and 5 will be omitted as appropriate.

[0025] The position and orientation estimation unit 201 estimates at least one of the position and orientation of the HMD 21 based on information from the HMD 21 .

[0026] The CG generation unit 202 performs rendering of CG, which is a virtual object, based on the estimation result of the position and orientation estimation unit 201. As a result, an image (image data) with a transparent background is generated, such as image 60 in FIG. 6. The image data is, for example, RGBA data (image data in which each pixel value is an RGBA value). The A value is a gradation value (alpha value) corresponding to transparency. In image 60, CG 601 and CG 602 are drawn, and the backgrounds of CG 601, 602 are transparent. Then, the CG generation unit 102 outputs the generated image data to the HMD 11.

[0027] The video see-through HMD 21 is a head-mounted display device that captures an image of real space, superimposes (composites) an image of a virtual object on the image of real space, and displays the superimposed image.

[0028] The sensor unit 211 acquires information for estimating at least one of the position and orientation of the HMD 21 .

[0029] The imaging unit 212 captures an image of the real space by capturing the image of the real space. The imaging unit 212 captures an image of the direction in which the user wearing the HMD 21 is facing. The imaging unit 212 includes, for example, an objective optical system and an image sensor. The imaging unit 212 is preferably compatible with color images and is preferably capable of capturing an image of a wider range than the range that can be displayed at one time on the display unit 215. The imaging unit 212 may be a monocular camera or a stereo camera.

[0030] The image processing unit 213 performs image processing on the image in real space acquired by the imaging unit 212. For example, a RAW image is obtained by the imaging unit 212, and development processing (e.g., Bayer interpolation) is performed by the image processing unit 213. The image processing unit 213 may perform image processing to improve image quality, such as brightness correction (shading correction), color correction, noise reduction, distortion correction, and angle of view correction.

[0031] The alpha blending unit 214 performs image processing similar to that of the image correction unit 112. Furthermore, the alpha blending unit 214 performs alpha blending, which blends the image of a virtual object output from the image generation device 20 (CG generation unit 202) with the image in real space output from the image processing unit 213. The alpha blending is a blending that takes into account the A value (transparency) of the image of the virtual object. For transparent regions in the image of the virtual object, pixel values ​​of the image in real space are applied, and for opaque regions in the image of the virtual object, pixel values ​​of the image of the virtual object are applied. And, for semi-transparent regions in the image of the virtual object, pixel values ​​obtained by blending the pixel values ​​of the image in real space and the pixel values ​​of the image of the virtual object are applied.

[0032] For example, image 61 is generated from image 60 in Fig. 6. In image 61, CGs 611 and 612 corresponding to CGs 601 and 602 are depicted, and in the background of CGs 611 and 612, The real space is depicted according to an A value (transparency) of 60.

[0033] The display unit 215 displays the image after image processing (alpha blending) by the alpha blending unit 214. A user wearing the HMD 21 cannot visually recognize a real space that is not an image. However, since reality is depicted in the image displayed on the display unit 215, the user can visually recognize the real space by visually recognizing the image displayed on the display unit 215.

[0034] An HMD that uses the optical see-through method cannot display low-brightness parts of virtual objects (the real space appears to be visible through the low-brightness parts), but an HMD that uses the video see-through method can display low-brightness parts of virtual objects. For this reason, in the past, different image data was used for the optical see-through method and the video see-through method, even if the virtual object to be displayed was the same. Furthermore, in the examples of Figures 2 and 6, RGBA data is transmitted. Since the amount of information in RGBA data is large (compared to RGB data), the load of transmitting RGBA data is large.

[0035] Therefore, in the first embodiment, image data with a small data size that can be used for both the optical see-through method and the video see-through method is generated. Fig. 3 is a block diagram showing the configuration of a display system according to the first embodiment. The display system in Fig. 3 includes an image generating device 30, an HMD 11 of the optical see-through method, and an HMD 31 of the video see-through method. Fig. 7 is a schematic diagram showing an image used in the display system in Fig. 3. In the explanation using Figs. 3 and 7, the contents common to the contents explained above will be omitted as appropriate.

[0036] The image generating device 30 is a device that generates image data of a virtual object to be displayed on the HMDs 11 and 31. The position and orientation estimation unit 301 estimates at least one of the position and orientation of the HMD 11 based on information from the HMD 11, and estimates at least one of the position and orientation of the HMD 31 based on information from the HMD 31. The CG generating unit 302 generates rendered image data by rendering the CG, which is a virtual object, based on the estimation result of the position and orientation estimation unit 301. The chromakey generating unit 303 generates chromakey image data that is output image data by performing a conversion process on the rendered image data generated by the CG generating unit 302. Then, the chromakey generating unit 303 outputs the chromakey image data to the outside of the image generating device 30. Note that the CG generating unit 302 and the chromakey generating unit 303 may be configured integrally.

[0037] For example, the CG generating unit 302 generates a rendering image (rendering image data) with a transparent background, such as image 70 in Fig. 7. The rendering image data is, for example, RGBA data. In image 70, CG 701 and CG 702 are depicted, and the backgrounds of CG 701 and 702 are transparent. CG 701 includes a blocked-up black area 703.

[0038] Then, in the conversion process, the chromakey generating unit 303 converts the pixel values ​​(transparent pixel values) of the background in the rendering image into specific pixel values. This conversion process generates a chromakey image (chromakey image data) in which each pixel value does not include transparency and the pixel values ​​of the background are specific pixel values. The chromakey image data is, for example, RGB data, YCbCr data, YUV data, or Lab data. In the first embodiment, the specific pixel values ​​are interpreted as background pixel values ​​in both the optical see-through type HMD 11 and the video see-through type HMD 31. The specific pixel values ​​are, for example, pixel values ​​that are displayed almost transparently in the optical see-through type HMD even when used as is, and are achromatic pixel values ​​whose luminance is equal to or less than a threshold value. The threshold value is, for example, 20% of the maximum luminance (maximum gradation value). The inventor of the present invention confirmed through an experiment that achromatic pixel values ​​whose luminance is equal to or less than 20% are displayed almost transparently in the optical see-through type HMD. In the first embodiment, the specific pixel value is assumed to be a black pixel value (for example, in the case of a 24-bit RGB value, a pixel value in which the R value, G value, and B value are all 0x000000).

[0039] Since the area 703 in the CG 701 is a black-out area, it is erroneously interpreted as a background area in the HMD 11 and the HMD 31. Therefore, it is preferable that the pixel value of the virtual object (CG) in the chromakey image data is not a specific pixel value (black pixel value). In the conversion process of the chromakey generation unit 303, the specific pixel value (black pixel value) of the rendering image data may be converted to a pixel value different from the specific pixel value. In this case, for example, the specific pixel value of the rendering image data is changed by an amount of change equal to or less than a threshold value. The threshold value is, for example, 20% of the range of pixel values ​​(tone values). The inventor of the present invention confirmed through an experiment that the impression of the virtual object does not change significantly if the amount of change of the pixel value is 20% or less. For example, in the conversion process of the chromakey generation unit 303, the black pixel value (0x000000) of the rendering image data is converted to a pixel value (0x000001) shifted by 1 bit.

[0040] Through these processes, image 71 is generated as a chromakey image from image 70 (rendered image) in Fig. 7. In image 71, CGs 711 and 712 corresponding to CGs 701 and 702 are depicted, and the pixel values ​​of the background of CGs 711 and 712 are black pixel values. The pixel values ​​of area 713 corresponding to blocked-up black area 703 are pixel values ​​of a color other than black (for example, dark gray).

[0041] The specific pixel value may be a pixel value that is estimated to be used less frequently in the rendered image data than a threshold value, or may be a pixel value that is used less frequently in the rendered image data than a threshold value. When the CG generating unit 302 performs rendering, the pixel value (color information) to be used in the rendered image data in the future is generally determined, so a pixel value that is unlikely to be used at that time may be used as the specific pixel value. When the CG generating unit 302 performs rendering, the pixel value (color information) used in the generated rendered image data is determined, so a pixel value that is unlikely to be used at that time may be used as the specific pixel value. The specific pixel value may be updated periodically (for example, for each frame). In these cases, the chromakey image data and additional information including chromakey information indicating that the specific pixel value is a pixel value of the background are output to the outside. The method of outputting the additional information is not particularly limited. For example, the additional information may be stored in a blank area of ​​the chromakey image data and output, or may be output separately from the chromakey image data.

[0042] The optical see-through type HMD 11 is the same as the HMD 11 in FIG.

[0043] The sensor unit 311, the imaging unit 312, the image processing unit 313, and the display unit 315 in the video see-through HMD 31 are the same as the sensor unit 211, the imaging unit 212, the image processing unit 213, and the display unit 215 in the HMD 21 in Fig. 2. Unlike the HMD 21, the HMD 31 has a chromakey composition unit 314.

[0044] The chromakey compositing unit 314 performs image processing similar to that of the image correction unit 112. Furthermore, the chromakey compositing unit 314 performs chromakey compositing, which combines a chromakey image (image of a virtual object) output from the image generating device 30 (chromakey generating unit 303) with an image in real space output from the image processing unit 313. In chromakey compositing, a specific pixel value in the chromakey image is interpreted as a pixel value of the background. For an area having a specific pixel value in the chromakey image, the pixel value of the image in real space is applied, and for an area having a pixel value different from the specific pixel value in the chromakey image, the pixel value of the chromakey image is applied. Information on the specific pixel value may be stored in the HMD 31 in advance, or may be notified to the HMD 31 from the image generating device 30. The display unit 315 displays the image after image processing (chromakey compositing) by the chromakey compositing unit 314.

[0045] When image 71 (chromakey image) of FIG. 7 is input to HMD 11, image 72 is viewed by the user wearing HMD 11. Image 72 is approximately the same as image 51 of FIG. 5. Furthermore, when image 71 (chromakey image) is input to HMD 31, image 73 is viewed by the user wearing HMD 31. Image 73 is approximately the same as image 61 of FIG. 6. In this way, the chromakey image data output by image generating device 30 can realize a desired display (a display similar to that of the conventional display) whether it is used in HMD 11 of the optical see-through type or in HMD 31 of the video see-through type. Furthermore, image generating device 30 can output image data having a smaller data size than RGBA data as the chromakey image data.

[0046] As described above, according to the first embodiment, output image data is generated in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that can be interpreted as the pixel value of the background in both the optical see-through display device and the video see-through display device. In this way, image data with a small data size that can be used in both the optical see-through and video see-through methods can be generated.

[0047] (Embodiment 2) Fig. 4 is a block diagram showing the configuration of a display system according to embodiment 2. The display system in Fig. 4 includes an image generating device 40, an HMD 41 of an optical see-through type, and an HMD 42 of a video see-through type. Fig. 8 is a schematic diagram showing an image used in the display system in Fig. 4. In the description of embodiment 2, content common to the content described in embodiment 1 will be omitted as appropriate.

[0048] The image generating device 40 (position and orientation estimation unit 401, CG generation unit 402, and chromakey generation unit 403) is substantially the same as the image generating device 30 (position and orientation estimation unit 301, CG generation unit 302, and chromakey generation unit 303) in FIG.

[0049] In the second embodiment, consider a case where a virtual object (CG) includes a transparent region (a region that should be transparent, for example, a shadow region) in a rendering image generated by the CG generation unit 402. For example, the CG generation unit 402 generates an image 80 (rendering image) in FIG. 8. In the image 80, CG 801 and CG 802 are depicted, and the backgrounds of the CGs 801 and 802 are transparent. The CG 801 includes a blocked-up black region 803. Furthermore, the image 80 includes a shadow region 804 of the CG 801 and a shadow region 805 of the CG 802.

[0050] The chromakey generating unit 403 converts pixel values ​​(transparent pixel values) of the background in the rendering image into a specific pixel value (first pixel value) and converts the specific pixel value of the rendering image into a pixel value different from the specific pixel value, similar to the chromakey generating unit 303 in FIG. 3. Furthermore, the chromakey generating unit 403 converts pixel values ​​of the transparent region of the virtual object in the rendering image into a second pixel value different from the first pixel value. This conversion process generates a chromakey image (chromakey image data) in which each pixel value does not include transparency, the pixel value of the background is the first pixel value, and the pixel value of the transparent region of the virtual object is the second pixel value. In the second embodiment, the first pixel value is interpreted as the pixel value of the background in both the optical see-through type HMD 41 and the video see-through type HMD 42. The second pixel value is interpreted as the pixel value of the transparent region of the virtual object in both the optical see-through type HMD 41 and the video see-through type HMD 42.

[0051] For example, the chromakey generating unit 403 generates an image 81 as a chromakey image from an image 80 (rendered image) in FIG. 8. In the image 81, CGs 811 and 812 corresponding to CGs 801 and 802 are depicted, and the pixel values ​​of the background of the CGs 811 and 812 are black pixel values. The pixel values ​​of an area 813 corresponding to a blocked-up black area 803 are a color other than black (e.g. The pixel values ​​of areas 814 and 815 corresponding to the shadow areas (semi-transparent areas) 804 and 805 are pixel values ​​of gray color.

[0052] As in the first embodiment, the specific pixel value (first pixel value) is not limited to a black pixel value, and the second pixel value is not limited to a gray pixel value.

[0053] The sensor unit 411, image correction unit 413, and display unit 414 in the optical see-through type HMD 41 are the same as the sensor unit 111, image correction unit 112, and display unit 113 in the HMD 11 in Figures 1 and 3. Unlike the HMD 11, the HMD 41 has a chromakey conversion unit 412.

[0054] The chromakey conversion unit 412 converts the first pixel value and the second pixel value in the chromakey image (image of a virtual object) output from the image generating device 30 (chromakey generation unit 303). In this conversion, the first pixel value is interpreted as a pixel value of the background, and the second pixel value is interpreted as a pixel value of a transparent area of ​​the virtual object. In the optical see-through type HMD 41, the lower the luminance, the higher the transmittance at which the image is displayed. Therefore, the chromakey conversion unit 412 converts the first pixel value into a black pixel value, and converts the second pixel value into a gray pixel value. The second pixel value may be converted into a pixel value having the same hue as the rendering image and a luminance equal to or lower than a threshold value. Information on the first pixel value and information on the second pixel value may be stored in advance in the HMD 41, or may be notified to the HMD 41 from the image generating device 40. The image data converted by the chromakey conversion unit 412 is, for example, RGB data, YCbCr data, YUV data, or Lab data.

[0055] It should be noted that, when the first pixel value is a black pixel value and the second pixel value is a gray pixel value (a pixel value whose luminance is equal to or less than a threshold value), the chromakey conversion unit 412 may be omitted.

[0056] As a result, for example, when image 81 (chromakey image) in Fig. 8 is input to the HMD 41, an image 82 is viewed by a user wearing the HMD 41. Image 82 is substantially the same as images 51 and 72 in Figs. 5 and 7. Unlike images 51 and 72, image 82 has regions 824 and 825 corresponding to regions 814 and 815 that are semi-transparent. In this way, it is possible to impart transparency to the transparent regions of a virtual object.

[0057] The sensor unit 421, imaging unit 422, image processing unit 423, alpha blending unit 425, and display unit 426 in the video see-through type HMD42 are the same as the imaging unit 212, image processing unit 213, alpha blending unit 214, and display unit 215 in the HMD21 in Fig. 2. Unlike the HMD21, the HMD42 has a chromakey conversion unit 424.

[0058] The chromakey conversion unit 424 converts the first pixel value and the second pixel value in the chromakey image (image of a virtual object) output from the image generating device 40 (chromakey generation unit 403). In this conversion, the first pixel value is interpreted as a pixel value of the background, and the second pixel value is interpreted as a pixel value of a transparent area of ​​the virtual object. In the HMD 42 of the video see-through type, the transmittance does not change depending on the luminance, and the image represented by the image data is displayed as it is. Therefore, the chromakey conversion unit 424 converts the first pixel value into a transparent pixel value and the second pixel value into a semi-transparent pixel value. The information of the first pixel value and the information of the second pixel value may be stored in advance in the HMD 42, or may be notified to the HMD 42 from the image generating device 40. The image data converted by the chromakey conversion unit 424 is, for example, RGBA data.

[0059] As a result, for example, when the image 81 (chromakey image) in FIG. 8 is input to the HMD 42, an image 83 is visually recognized by a user wearing the HMD 42. The image 83 is the same as the image in FIGS. The image 83 is substantially the same as the images 61 and 73. Unlike the images 61 and 73, the image 83 has semi-transparent regions 834 and 835 corresponding to the regions 814 and 815. In this manner, transparency can be imparted to the transparent regions of the virtual object.

[0060] As described above, the same effects as those of the first embodiment can be obtained in the second embodiment. Furthermore, according to the second embodiment, the transparent region of the virtual object can be displayed with transparency whether on an optical see-through display device or a video see-through display device.

[0061] (Embodiment 3) The display system according to the third embodiment has a similar configuration to the display system of Fig. 3. Fig. 9 is a schematic diagram showing an image used in the display system according to the third embodiment. In the description of the third embodiment, the contents common to the contents described in the above-mentioned embodiments will be omitted as appropriate.

[0062] By transmitting an image 90 including a white CG 901 on a black background to the HMD 11, the user can view an image 91 including the CG 901 superimposed on the real space. However, the CG 901 does not have transparency. By transmitting an image 92, which is an image in which each pixel value includes an A value and an image in which a semi-transparent, white CG 902 is included on the background, to the HMD 21, the user can view an image 93 in which the semi-transparent, white CG 902 is superimposed on an image in the real space. However, the load of transmitting an image in which each pixel value includes an A value is large.

[0063] Therefore, in the third embodiment, the chromakey generating unit 303 generates chromakey image data having a dot pattern consisting of a first dot of a specific pixel value and a second dot of a pixel value different from the specific pixel value in a transparent area of ​​a virtual object (CG). For example, an image 94 including a CG 903 expressed in a black and white dot pattern on a black background is generated and transmitted to the HMD 11 or HMD 31. Each pixel value of the image 94 does not include an A value. This allows the user to view an image 95 including the CG 903 superimposed on the real space. In each of the HMD 11 and HMD 31, black becomes transparent, so the CG 903 is viewed as a semi-transparent and white CG.

[0064] By changing the ratio between the number of first dots and the number of second dots, the apparent transparency of the CG 903 visually recognized by the user changes. The more black dots there are, the higher the apparent transparency, and the more white dots there are, the lower the apparent transparency. For example, in the case of dot pattern 96, since there are many black dots, the apparent transparency is high. In the case of dot pattern 97, since the number of black dots and the number of white dots are approximately the same, the apparent transparency is medium. In the case of dot pattern 98, since there are many white dots, the apparent transparency is low. Therefore, the chromakey generating unit 303 may determine the ratio between the number of first dots and the number of second dots in the dot pattern of the transparent region according to the transparency of the transparent region of the virtual object (the transparency of the rendering image).

[0065] As described in the first embodiment, the specific pixel value used as the pixel value of the first dot is not limited to a black pixel value. Also, the pixel value of the second dot is not limited to a white pixel value. For example, the pixel value of the second dot may be a pixel value having the same hue as the rendering image and a luminance equal to or greater than a threshold value.

[0066] As described above, the third embodiment can also provide the same effects as the second embodiment.

[0067] (Embodiment 4) The display system according to the fourth embodiment has a configuration similar to that of the display system shown in FIG. 10 is a schematic diagram showing an image used in a display system according to embodiment 4. In the description of embodiment 4, content common to the content described in the previous embodiments will be omitted as appropriate.

[0068] In the fourth embodiment, as in the third embodiment, a virtual object (CG) includes a transparent area in a rendering image generated by the CG generation unit 402, and the transparent area is expressed by a dot pattern in a chromakey image generated by the chromakey generation unit 403. For example, the CG generation unit 402 generates, as a rendering image, an image 92 in which each pixel value includes an A value (transparency) and which includes a semi-transparent and white CG 902 on a background. Then, the chromakey generation unit 403 generates, from the image 92, an image 94 including a CG 903 expressed by a black and white dot pattern on a black background as a chromakey image.

[0069] In the third embodiment, in the case of the optical see-through method, the image 94 is displayed as is, and in the case of the video see-through method, chromakey compositing is performed in which the image 94 is directly composited with an image in real space, and the result of the chromakey compositing is displayed. However, with this method, dots may stand out depending on the dot pattern. Therefore, in the fourth embodiment, the chromakey conversion units 412 and 424 convert the pixel values ​​of the transparent area expressed by the dot pattern according to the dot pattern.

[0070] The dot pattern 1001 in FIG. 10 is an enlarged version of CG903 expressed by a dot pattern, and neither the black pixel value nor the white pixel value includes the A value (transparency). The chromakey conversion unit 424 determines the transparency of CG903 from the arrangement of black dots and white dots in CG903 (dot pattern 1001), and converts the pixel value of CG903 to a pixel value including the A value corresponding to the transparency. The color of CG903 after conversion is, for example, white (the color of the second dot). As a result, the image 92 (rendered image) is restored as the image after conversion by the chromakey conversion unit 424. Then, the alpha blending unit 425 performs alpha blending of the image after conversion by the chromakey conversion unit 424 and the image in real space, and the image after alpha blending is displayed. As a result, the user can view the image 93 in which the semi-transparent and white CG902 is superimposed on the image in real space.

[0071] Similarly, the chromakey conversion unit 412 determines the transparency of CG903 from the arrangement of black dots and white dots in CG903 (dot pattern 1001), and converts the pixel values ​​of CG903 to pixel values ​​of luminance corresponding to the transparency. The color of CG903 after conversion is, for example, white (the color of the second dots). Then, an image after conversion by the chromakey conversion unit 424 (and image processing by the image correction unit 413) is displayed. As a result, the user can view an image including a semi-transparent and white CG superimposed on real space.

[0072] As described above, the fourth embodiment can also provide the same effects as the second and third embodiments. Furthermore, according to the fourth embodiment, the dot pattern is not displayed as is, so the dots of the dot pattern do not stand out.

[0073] (Embodiment 5) The display system according to the fifth embodiment has a configuration similar to that of the display system in Fig. 3 or Fig. 4. Fig. 11 is a schematic diagram showing an image used in the display system according to the fifth embodiment. In the description of the fifth embodiment, the contents common to the contents described in the above-mentioned embodiments will be omitted as appropriate.

[0074] In the fifth embodiment, the image generating device (chromakey generating unit) compresses chromakey image data and outputs it to the outside (HMD), and the HMD expands the compressed chromakey data for use. The image generating device (chromakey generating unit) then generates a dot pattern that takes these compression and expansion into consideration (e.g., a dot pattern that is compressed and expanded) as a dot pattern that represents a transparent area of ​​a virtual object. A dot pattern that is stretched to the desired state is used.

[0075] An example in which a display system according to the fifth embodiment has a configuration similar to that of the display system in FIG. 4 will be described.

[0076] The operation of the image generating device 40 will be described. The CG generating unit 402 generates, as a rendering image, an image 92 in which each pixel value includes an A value (transparency) and which includes a semi-transparent and white CG 902 on a background. The chromakey generating unit 403 generates, from the image 92, an image 94 as a chromakey image, which includes a CG 903 expressed as a black and white dot pattern on a black background. The chromakey generating unit 403 then compresses the image 94 to generate an image 1101. The dot pattern of the CG 903 changes due to the compression, and the CG 903 becomes a CG 1102.

[0077] The operation of the video see-through type HMD 42 will be described. The chromakey conversion unit 424 restores the image 94 (the chromakey image before compression) by expanding the image 1101 (the chromakey image after compression). Thereafter, as in the fourth embodiment, the chromakey conversion unit 424 restores the image 92 (the rendering image), and the alpha blending unit 425 generates the image 93 by alpha blending. Note that each of the compression and expansion may be a lossless conversion or a lossy conversion, and an image different from the chromakey image before compression may be generated by expansion. However, even in this case, the chromakey generation unit 403 generates the chromakey image before compression in consideration of the compression and expansion to be executed (using a dot pattern that becomes a desired state by compression and expansion).

[0078] As described above, according to the fifth embodiment, a dot pattern is used that takes into consideration the compression and expansion of the chromakey image. In this way, even when the chromakey image is compressed and expanded, the same effect as that of the fourth embodiment can be obtained.

[0079] (Embodiment 6) The display system according to the sixth embodiment has a similar configuration to the display system of Fig. 3. Fig. 12 is a schematic diagram showing an image used in the display system according to the sixth embodiment. In the description of the sixth embodiment, the contents common to the contents described in the above-mentioned embodiments will be omitted as appropriate.

[0080] In the sixth embodiment, the chromakey generating unit 303 generates chromakey image data so that pixel values ​​in a transparent region of a virtual object alternate between a specific pixel value and a pixel value different from the specific pixel value. The pixel value different from the specific pixel value is, for example, a pixel value having the same hue as the rendering image and a brightness equal to or higher than a threshold value. When it is desired that a user of the HMD visually recognize a semi-transparent and white CG, the chromakey generating unit 303 generates chromakey image data so as to repeatedly display and hide a non-transparent white CG.

[0081] In FIG. 12, the chromakey generating unit 303 generates a chromakey image by alternately generating an image 90 including a non-transparent white CG 901 and an image 1201 not including CG 901 (an image in which the area of ​​CG 901 is black). In this case, an image 91 including CG 901 superimposed on a real space (or an image of the real space) and an image 1202 in which only the real space is visible are presented to the user alternately. Due to the characteristics of the eye and the brain, humans view images that are smoothed in the time direction. Therefore, when the display and non-display of CG 901 are repeated at high speed, the user views an image 1204 including a semi-transparent and white CG 1203 superimposed on the real space (or an image of the real space).

[0082] By changing the speed at which the CG is switched between hidden (specific pixel values) and displayed (pixel values ​​different from the specific pixel values), the transparency perceived by the user changes. The unit 303 may generate chromakey image data so that the above switching is performed at a speed according to the transparency of the transparent region of the virtual object (the transparency of the rendering image).

[0083] As described above, the sixth embodiment can also provide the same effects as the second to fifth embodiments.

[0084] (Embodiment 7) The display system according to the seventh embodiment has a configuration similar to that of the display system in Fig. 3. Fig. 12 will also be referred to in the seventh embodiment. In the description of the seventh embodiment, details common to the details described in the previous embodiments will be omitted as appropriate.

[0085] In the seventh embodiment, the chromakey generating unit 303 generates chromakey image data for the right eye and chromakey image data for the left eye. The chromakey generating unit 303 uses a specific pixel value as the pixel value of the transparent region of the virtual object in one of the chromakey image data for the right eye and the chromakey image data for the left eye. The chromakey generating unit 303 uses a pixel value different from the specific pixel value as the pixel value of the transparent region of the virtual object in the other of the chromakey image data for the right eye and the chromakey image data for the left eye.

[0086] For example, the chromakey generating unit 303 generates an image 90 including a non-transparent white CG 901 as a chromakey image for the right eye, and generates an image 1201 not including CG 901 (an image in which the area of ​​CG 901 is black) as a chromakey image for the left eye. In this case, the image 91 including CG 901 superimposed on the real space (or an image of the real space) is presented to the right eye of the user, and an image 1202 in which only the real space is visible is presented to the left eye of the user. As a result, due to the characteristics of the eyes and brain, the user sees an image 1204 including a semi-transparent and white CG 1203 superimposed on the real space (or an image of the real space).

[0087] As described above, the same effects as those of the embodiments 2 to 6 can be obtained in the embodiment 7. Note that, although it is difficult to adjust the transparency as perceived by the user by the method of the embodiment 7 alone, the transparency as perceived by the user may be adjusted by combining the embodiment 7 with other embodiments.

[0088] (Embodiment 8) The display system according to the eighth embodiment has a similar configuration to the display system of Fig. 4. Fig. 13 is a schematic diagram showing an image used in the display system according to the eighth embodiment. In the description of the eighth embodiment, the contents common to the contents described in the above-mentioned embodiments will be omitted as appropriate.

[0089] In the eighth embodiment, the chromakey generating unit 403 outputs the chromakey image data and additional information including transparency information indicating transparent regions (e.g., transparent regions of the background and virtual object) to the outside. The method of outputting the additional information is not particularly limited. For example, the additional information may be stored in a blank region of the chromakey image data and output, or may be output separately from the chromakey image data. Then, the HMD displays an image expressing a desired transparency based on the additional information (transparency information) (e.g., an image in which the real space (or an image of the real space) can be viewed as the background and the transparent regions of the virtual object have transparency).

[0090] An example of the operation according to the eighth embodiment will be described. The CG generation unit 402 generates, as a rendering image, an image 92 in which each pixel value includes an A value (transparency) and which includes a semi-transparent and white CG 902 on a background. The chromakey generation unit 403 generates, from the image 92, an image 1301 including a white CG 1302 on a black background as a chromakey image. Then, the chromakey generation unit 403 adds additional information 130 including transparency information indicating transparent areas (for example, the background and transparent areas of a virtual object) to blank areas of the data of the image 1301. 3. In the video see-through HMD 42, the chromakey conversion unit 424 restores the image 92 (rendering image) based on the additional information 1303 (transparency information). Then, the alpha blending unit 425 generates an image 93 in which a semi-transparent and white CG 902 is superimposed on an image in real space by alpha blending.

[0091] In the optical see-through HMD 41, the chromakey conversion unit 412 reduces the luminance of the CG 1302 based on the additional information 1303 (transparency information). Then, the image after the luminance of the CG 1302 is reduced is displayed. As a result, the user can visually recognize an image including a semi-transparent and white CG superimposed on the real space.

[0092] As described above, the same effects as those of the second to seventh embodiments can be obtained in the eighth embodiment as well. Note that since the data size of the transparency information indicating the transparent area is smaller than the total data size of the transmittance information of each pixel, the transmission load of the transparency information is small.

[0093] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0094] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0095] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An image generating device that generates image data of a virtual object to be displayed on an optical see-through or video see-through display device, a generating means for generating output image data in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that can be interpreted as the pixel value of the background in both an optical see-through type display device and a video see-through type display device; an output means for outputting the output image data to an external device; An image generating device comprising: (Configuration 2) The pixel value of the virtual object in the output image data is not the specific pixel value. 2. The image generating device according to configuration 1. (Configuration 3) The generating means generates rendered image data with a transparent background by rendering a virtual object, and performs a conversion process for converting transparent pixel values ​​of the rendered image data into the specific pixel values ​​to generate the output image data. 3. The image generating device according to configuration 1 or 2. (Configuration 4) In the conversion process, the specific pixel value of the rendering image data is converted to a pixel value different from the specific pixel value, and a transparent pixel value of the rendering image data is converted to the specific pixel value. 4. The image generating device according to configuration 3. (Configuration 5) In the conversion process, the specific pixel value of the rendering image data is changed by an amount equal to or less than a threshold value. 5. The image generating device according to configuration 4. (Configuration 6) The specific pixel value is an achromatic pixel value whose luminance is equal to or less than a threshold value. 6. The image generating device according to any one of configurations 1 to 5. (Configuration 7) The specific pixel value is a black pixel value. 7. The image generating device according to configuration 6. (Configuration 8) The specific pixel value is a pixel value whose frequency of use in the rendering image data is estimated to be equal to or less than a threshold value. 6. The image generating device according to any one of configurations 3 to 5. (Configuration 9) The specific pixel value is a pixel value whose frequency of use in the rendering image data is equal to or less than a threshold value. 6. The image generating device according to any one of configurations 3 to 5. (Configuration 10) The output image data is image data in which each pixel value does not include transparency, the pixel value of the background is a first pixel value that is the specific pixel value, and the pixel value of the transparent region of the virtual object is a second pixel value that is interpreted by the display device as the pixel value of the transparent region. 10. The image generating device according to any one of configurations 1 to 9. (Configuration 11) The output image data has a dot pattern in a transparent area of ​​the virtual object, the dot pattern being made up of a first dot of the specific pixel value and a second dot of a pixel value different from the specific pixel value. 10. The image generating device according to any one of configurations 1 to 9. (Configuration 12) In the dot pattern, the ratio of the number of the first dots to the number of the second dots is a ratio according to the transparency of the transmissive region. 12. The image generating device according to claim 11, (Configuration 13) The output means compresses the output image data and outputs it to an outside. The dot pattern is a dot pattern that takes compression and expansion into consideration. 13. The image generating device according to configuration 11 or 12. (Configuration 14) The output image data is image data in which pixel values ​​in a transparent region of the virtual object alternate between the specific pixel value and a pixel value different from the specific pixel value. 10. The image generating device according to any one of configurations 1 to 9. (Configuration 15) The output image data is image data in which pixel values ​​of a transparent region of the virtual object alternate between the specific pixel value and a pixel value different from the specific pixel value at a speed according to the transparency of the transparent region. 15. The image generating device according to claim 14, (Configuration 16) the generating means generates output image data for a right eye and output image data for a left eye; In one of the output image data for the right eye and the output image data for the left eye, a pixel value in a transparent region of the virtual object is the specific pixel value, In the other of the right-eye output image data and the left-eye output image data, a pixel value in a transparent region of the virtual object is a pixel value different from the specific pixel value. 10. The image generating device according to any one of configurations 1 to 9. (Configuration 17) The output means outputs to an outside the image data and additional information including information indicating that the specific pixel value is a pixel value of the background. 17. The image generating device according to any one of configurations 1 to 16. (Configuration 18) The output means outputs the output image data and additional information including information indicating a transparent region to an outside. 18. The image generating device according to any one of configurations 1 to 17. (Configuration 19) The generating means stores the additional information in a blank area of ​​the output image data, The output means outputs to an external device the output image data in which the additional information is stored in the blank area. 19. The image generating device according to configuration 17 or 18. (Configuration 20) An image generating device according to any one of configurations 1 to 19, An optical see-through or video see-through display device Including, The display device displays a virtual object based on the output image data of the image generating device. A display system comprising: (Configuration 21) When the display device is an optical see-through display device, the display device converts the output image data into image data in which a specific pixel value is converted into a black pixel value, and displays a virtual object according to the image data; When the display device is a video see-through type display device, the display device converts the output image data into image data in which each pixel value includes transparency and a specific pixel value is converted into a transparent pixel value, and displays a virtual object according to the image data. 21. The display system of claim 20. (Configuration 22) the output image data has, in a transparent region of the virtual object, a dot pattern corresponding to the transparency of the transparent region; The display device converts pixel values ​​of the transmissive area in accordance with the dot pattern. 22. The display system according to claim 21, (method) 1. An image generation method for generating image data of a virtual object to be displayed on an optical see-through or video see-through display device, comprising: generating output image data in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that is interpreted as the pixel value of the background by both an optical see-through display device and a video see-through display device; outputting the output image data to an external device; 1. An image generating method comprising: (program) 20. A program for causing a computer to function as each of the means of the image generating device according to any one of configurations 1 to 19. [Explanation of symbols]

[0096] 30, 40: Image generating device 303, 403: Chromakey generating unit

Claims

1. An image generating device that generates image data of a virtual object to be displayed on an optical see-through or video see-through display device, a generating means for generating output image data in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that can be interpreted as the pixel value of the background in both an optical see-through type display device and a video see-through type display device; an output means for outputting the output image data to an external device; An image generating device comprising:

2. The pixel value of the virtual object in the output image data is not the specific pixel value.

2. The image generating device according to claim 1.

3. The generating means generates rendered image data with a transparent background by rendering a virtual object, and performs a conversion process for converting transparent pixel values ​​of the rendered image data into the specific pixel values ​​to generate the output image data.

2. The image generating device according to claim 1.

4. In the conversion process, the specific pixel value of the rendering image data is converted to a pixel value different from the specific pixel value, and a transparent pixel value of the rendering image data is converted to the specific pixel value.

4. The image generating apparatus according to claim 3.

5. In the conversion process, the specific pixel value of the rendering image data is changed by an amount equal to or less than a threshold value.

5. The image generating apparatus according to claim 4.

6. The specific pixel value is an achromatic pixel value whose luminance is equal to or less than a threshold value.

2. The image generating device according to claim 1.

7. The specific pixel value is a black pixel value.

7. The image generating apparatus according to claim 6.

8. The specific pixel value is a pixel value whose frequency of use in the rendering image data is estimated to be equal to or less than a threshold value.

4. The image generating apparatus according to claim 3.

9. The specific pixel value is a pixel value whose frequency of use in the rendering image data is equal to or less than a threshold value.

4. The image generating apparatus according to claim 3.

10. The output image data is image data in which each pixel value does not include transparency, the pixel value of the background is a first pixel value that is the specific pixel value, and the pixel value of the transparent region of the virtual object is a second pixel value that is interpreted by the display device as the pixel value of the transparent region.

2. The image generating device according to claim 1.

11. The output image data has a dot pattern in a transparent area of ​​the virtual object, the dot pattern being made up of a first dot of the specific pixel value and a second dot of a pixel value different from the specific pixel value.

2. The image generating device according to claim 1.

12. In the dot pattern, the ratio of the number of the first dots to the number of the second dots is a ratio according to the transparency of the transmissive region.

12. The image generating device according to claim 11.

13. The output means compresses the output image data and outputs it to an outside. The dot pattern is a dot pattern that takes compression and expansion into consideration.

12. The image generating device according to claim 11.

14. The output image data is image data in which pixel values ​​in a transparent region of the virtual object alternate between the specific pixel value and a pixel value different from the specific pixel value.

2. The image generating device according to claim 1.

15. The output image data is image data in which pixel values ​​of a transparent region of the virtual object alternate between the specific pixel value and a pixel value different from the specific pixel value at a speed according to the transparency of the transparent region.

15. An image generating apparatus according to claim 14.

16. the generating means generates output image data for a right eye and output image data for a left eye; In one of the output image data for the right eye and the output image data for the left eye, a pixel value in a transparent region of the virtual object is the specific pixel value, In the other of the right-eye output image data and the left-eye output image data, a pixel value in a transparent region of the virtual object is a pixel value different from the specific pixel value.

2. The image generating device according to claim 1.

17. The output means outputs to an outside the image data and additional information including information indicating that the specific pixel value is a pixel value of the background.

2. The image generating device according to claim 1.

18. The output means outputs the output image data and additional information including information indicating a transparent region to an outside.

2. The image generating device according to claim 1.

19. The generating means stores the additional information in a blank area of ​​the output image data, The output means outputs to an external device the output image data in which the additional information is stored in the blank area.

20. The image generating device according to claim 17 .

20. An image generating device according to any one of claims 1 to 19, An optical see-through or video see-through display device Including, The display device displays a virtual object based on the output image data of the image generating device. A display system comprising:

21. When the display device is an optical see-through display device, the display device converts the output image data into image data in which a specific pixel value is converted into a black pixel value, and displays a virtual object according to the image data; When the display device is a video see-through type display device, the display device converts the output image data into image data in which each pixel value includes transparency and a specific pixel value is converted into a transparent pixel value, and displays a virtual object according to the image data.

21. The display system of claim 20.

22. the output image data has, in a transparent region of the virtual object, a dot pattern corresponding to the transparency of the transparent region; The display device converts pixel values ​​of the transmissive area in accordance with the dot pattern.

22. The display system of claim 21.

23. 1. An image generation method for generating image data of a virtual object to be displayed on an optical see-through or video see-through display device, comprising: generating output image data in which each pixel value does not include transparency and the pixel value of the background is a specific pixel value that is interpreted as the pixel value of the background by both an optical see-through display device and a video see-through display device; outputting the output image data to an external device; 1. An image generating method comprising:

24. A program for causing a computer to function as each of the means of the image generating apparatus according to any one of claims 1 to 19.

Images