Image processing device, head-mounted display device, method for controlling image processing device, method for controlling head-mounted display device, and program

Abstract

An image processing apparatus according to the present invention is an image processing apparatus usable with a head-mounted display apparatus including a first imaging unit configured to obtain distance information, a second imaging unit different from the first imaging unit, and a display unit, the image processing apparatus includes: a first acquisition unit configured to acquire a first image of a subject captured by the first imaging unit; a second acquisition unit configured to acquire a second image of the subject captured by the second imaging unit; and a generation unit configured to generate a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

Description

Technical Field

[0001] The present invention relates to an image processing device, a head mounted display device, a control method of an image processing device, a control method of a head mounted display device, and a program. Background Art

[0002] As a head-mounted display device (e.g., HMD (head-mounted display)), a video see-through display device has been proposed. The video see-through display device has a camera unit and a display unit, and the camera unit captures the outside world (real space), and the display unit displays the image obtained by the camera unit (the image of the outside world). For example, a technology related to the video see-through display device is disclosed in Patent Document 1.

[0003] [Citation list]

[0004] [Patent Document]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-62593 Summary of the invention

[0006] Problem that the invention aims to solve

[0007] However, in a conventional head-mounted display device (video see-through), a scene different from the real scene is provided to the user. For example, because the position of the camera unit is different from the position of the user's eyes, a scene different from the real scene is provided to the user. In order to provide the user with a scene close to the real scene, it is also necessary to display a high-resolution color image. However, if the head-mounted display device is provided with two camera units capable of obtaining a high-resolution color image, wherein one of the two camera units is configured to obtain an image to be viewed by the right eye and the other of the two camera units is configured to obtain an image to be viewed by the left eye, the cost of the device will increase as the cost of the camera units increases. In addition, since it is necessary to process two color images (high resolution) obtained by the two camera units, the processing load increases, and expensive processors and memories are required.

[0008] The present invention provides a technology for obtaining an image suitable for display on a head-mounted display device.

[0009] Solutions for solving problems

[0010] In its first aspect, the present invention provides an image processing device that can be used together with a head-mounted display device, the head-mounted display device including a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the image processing device including: a first acquisition unit configured to acquire a first image of a subject captured by the first camera unit; a second acquisition unit configured to acquire a second image of the subject captured by the second camera unit; and a generation unit configured to generate a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

[0011] In its second aspect, the present invention provides a head-mounted display device, comprising: a camera unit configured to obtain distance information; a projection unit configured to project invisible light onto a camera range of the camera unit; and a display unit, wherein the projection unit repeatedly projects and does not project the invisible light onto the camera range of the camera unit.

[0012] In a third aspect, the present invention provides a head mounted display device, comprising: the first camera unit; the second camera unit; the display unit; and the above-mentioned image processing device.

[0013] In its fourth aspect, the present invention provides a control method for an image processing device, which can be used together with a head-mounted display device, wherein the head-mounted display device includes a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the control method including: a first acquisition step for acquiring a first image of a subject captured by the first camera unit; a second acquisition step for acquiring a second image of the subject captured by the second camera unit; and a generation step for generating a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

[0014] In its fifth aspect, the present invention provides a control method for a head-mounted display device, wherein the head-mounted display device includes a camera unit configured to obtain distance information, a projection unit configured to project invisible light onto the camera range of the camera unit, and a display unit, wherein the control method includes: controlling the projection unit to repeat the steps of projecting and not projecting the invisible light onto the camera range of the camera unit.

[0015] In its sixth aspect, the present invention provides a program for causing a computer to execute each step of the above-mentioned control method.

[0016] Effects of the Invention

[0017] According to the present invention, it is possible to obtain an image suitable for display on a head mounted display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A and Figure 1 B is an external view of the HMD 100 according to the first embodiment.

[0019] Figure 2 is a block diagram showing the functional configuration of the CPU 106 according to the first embodiment.

[0020] Figure 3 A and Figure 3 B is a diagram showing a state of coordinate projection according to the first embodiment.

[0021] Figure 4 is a data flow diagram of the display image generation process according to the first embodiment.

[0022] Figure 5 is a block diagram showing the functional configuration of the CPU 106 according to the second embodiment.

[0023] Figure 6 is a diagram showing image acquisition by a stereo camera according to the second embodiment.

[0024] Figure 7 is a diagram showing a positional relationship among a display unit, an eye, and a subject according to the third embodiment.

[0025] Figure 8 is a block diagram showing the functional configuration of the CPU 106 according to the third embodiment.

[0026] Fig. 9 is a data flow diagram of the display image generation process according to the third embodiment.

[0027] Fig.10 is a diagram showing a situation where interpolation is required according to the fourth embodiment.

[0028] Fig.11 is a diagram showing a configuration of a synthesizing unit according to a fourth embodiment.

[0029] Fig.12 is a diagram showing a situation where interpolation is required according to the fourth embodiment.

[0030] Fig.13 is a diagram showing a configuration of a synthesizing unit according to a fourth embodiment. DETAILED DESCRIPTION

[0031] <First Embodiment>

[0032] A first embodiment of the present invention will be described. In the first embodiment, an example in which the present invention is applied to an HMD (head-mounted display) will be described, but the present invention can also be applied to other head-mounted display devices. For example, a handheld display device that a user holds in his / her hand and wears (places) on his / her head is a head-mounted display device, and the present invention is also applicable to the handheld display device. The present invention is also applicable to a head-mounted display device in which a user views an image with both eyes and a head-mounted display device in which a user views an image with one eye. The present invention is also applicable to an electronic device other than a head-mounted display device as long as the electronic device can be used with the head-mounted display device. An image processing device to which the present invention is applied can be provided in a head-mounted display device or in an electronic device separated from the head-mounted display device. For example, the present invention is also applicable to a controller or a personal computer (PC) connected to the HMD by wire or wirelessly. The HMD according to the first embodiment is a video perspective HMD, and an image of the outside world (real space) is displayed by synthesizing graphics (e.g., a virtual subject) as needed.

[0033] Figure 1 A and Figure 1 B is an external view of the HMD 100 according to the first embodiment. Figure 1 A is a front perspective view of the HMD 100, and Figure 1 FIG. B is a rear perspective view of the HMD 100. Figure 1 As shown in FIG. 1A , the HMD 100 is provided with a headband 110. The user places the HMD 100 in contact with the eyes and fixes the HMD 100 to the head with the headband 110. The HMD 100 has a stereo camera 101, a monocular camera 102, a projection unit 103, a left display unit 104L, a right display unit 104R, a left eye camera unit 105L, and a right eye camera unit 105R. The HMD 100 has a CPU (Central Processing Unit) 106, a ROM (Read Only Memory) 107, and a RAM (Random Access Memory) 108.

[0034] The stereo camera 101 is a camera (imaging device) that captures the outside world to obtain distance information, and has a left imaging unit 101L and a right imaging unit 101R. The left imaging unit 101L and the right imaging unit 101R are each, for example, a global shutter type imaging unit. The monocular camera 102 is an imaging device (imaging unit) that faces the same direction as the stereo camera 101 and captures the outside world. The monocular camera 102 is, for example, a rolling shutter type imaging unit. From a certain point of view, the monocular camera 102 has higher performance than each of the left imaging unit 101L and the right imaging unit 101R. In the first embodiment, the left imaging unit 101L and the right imaging unit 101R are each an imaging unit that captures a grayscale image, and the monocular camera 102 is an imaging unit that captures a color image. The projection unit 103 faces the same direction as the stereo camera 101 and the monocular camera 102, and projects invisible light onto the imaging range of the stereo camera 101 and the monocular camera 102. In the first embodiment, the projection unit 103 is not used (an example using the projection unit 103 will be described in the second embodiment).

[0035] The left display unit 104L is arranged to face the left eye of the user wearing the HMD 100, and displays an image to be viewed by the left eye of the user. The right display unit 104R is arranged to face the right eye of the user wearing the HMD 100, and displays an image to be viewed by the right eye of the user. The left-eye imaging unit 105L is arranged in association with the left display unit 104L, and captures the left eye looking at the left display unit 104L. The right-eye imaging unit 105R is arranged in association with the right display unit 104R, and captures the right eye looking at the right display unit 104R. In the first embodiment, the left-eye imaging unit 105L and the right-eye imaging unit 105R are not used (an example of using the left-eye imaging unit 105L and the right-eye imaging unit 105R will be described in the third embodiment).

[0036] The CPU 106 is an information processing device (image processing device) that controls the entire HMD 100. The ROM 107 stores various data (for example, various programs and various parameters). The RAM 108 also stores various data. For example, the CPU 106 loads a program stored in the ROM 107 into the RAM 108 and executes the program.

[0037] Figure 2 1 is a block diagram showing a functional configuration of the CPU 106 according to the first embodiment. In the first embodiment, the CPU 106 has a depth map generation unit 201 , a coordinate conversion unit 202 , a synthesis unit 203 , a position / posture estimation unit 204 , and a rendering unit 205 .

[0038] The depth map generation unit 201 acquires two grayscale images with parallax (a grayscale image acquired by the left imaging unit 101L and a grayscale image acquired by the right imaging unit 101R) from the stereo camera 101. The depth map generation unit 201 then generates a depth map (a distance map, depth information, and distance information) from the two acquired grayscale images. The depth map is information indicating the distribution of the depth (distance) from the stereo camera 101 to the subject, and is, for example, an image having depth values ​​as pixel values. Figure 3 FIG. A shows an example of a depth distribution represented by a depth map. Figure 3 As shown in A of FIG. 1 , the depth map shows the depth distribution in the range where the imaging range of the left imaging unit 101L and the imaging range of the right imaging unit 101R overlap.

[0039] The coordinate conversion unit 202 converts the pixel coordinates of the grayscale image (left grayscale image) acquired by the left camera unit 101L into the coordinates of the color image (captured color image) acquired by the monocular camera 102 based on the depth map generated by the depth map generation unit 201. This conversion can also be considered as "projecting (coordinate projection) the coordinates of the left grayscale image onto the captured color image." This allows the coordinates of the left grayscale image to be associated with the coordinates of the captured color image. Similarly, the coordinate conversion unit 202 projects the coordinates of the grayscale image (right grayscale image) acquired by the right camera unit 101R onto the captured color image. This allows the coordinates of the right grayscale image to be associated with the coordinates of the captured color image. This coordinate projection on Figure 3 As shown in A. Figure 3 A shows a state where the coordinates of the right grayscale image are projected onto the captured color image. The coordinates of the left grayscale image are similarly projected onto the captured color image.

[0040] In addition, the coordinate conversion unit 202 projects the coordinates of the left grayscale image onto the image to be displayed on the left display unit 104L (display image, left display image) based on the depth map. This allows the coordinates of the left grayscale image to be associated with the coordinates of the left display image. Similarly, the coordinate conversion unit 202 projects the coordinates of the right grayscale image onto the image to be displayed on the right display unit 104R (display image, right display image). This allows the coordinates of the right grayscale image to be associated with the coordinates of the right display image. This coordinate projection is Figure 3 B is shown. Figure 3 B shows a state where the coordinates of the right grayscale image are projected onto the right display image. The coordinates of the left grayscale image are similarly projected onto the left display image.

[0041] Note that in the first embodiment, a left display image is generated to represent the scene of the subject viewed from a position (predetermined position, virtual viewpoint) different from the position of the left camera unit 101L and the position of the monocular camera 102. For example, a left display image is generated to represent the scene of the subject viewed from a position directly facing the left display unit 104L (a position where the left eye is considered to be located). Similarly, a right display image is generated to represent the scene of the subject viewed from a position (predetermined position, virtual viewpoint) different from the position of the right camera unit 101R and the position of the monocular camera 102. For example, a right display image is generated to represent the scene of the subject viewed from a position directly facing the right display unit 104R (a position where the right eye is considered to be located). The virtual viewpoint may be fixed or may be changed by the user.

[0042] Since the positional relationship among the stereo camera 101 (the left imaging unit 101L and the right imaging unit 101R), the monocular camera 102, the left display unit 104L, and the right display unit 104R is known, the above-described processing (association) can be performed by using the depth map.

[0043] The synthesis unit 203 acquires the grayscale image from the stereo camera 101, acquires the captured color image from the monocular camera 102, and acquires the result of the above association from the coordinate conversion unit 202. The synthesis unit 203 synthesizes the pixel values ​​of the pixels of the left grayscale image and the pixel values ​​of the pixels of the captured color image corresponding to the pixels of the left grayscale image based on the association result from the coordinate conversion unit 202. In the first embodiment, the synthesis unit 203 synthesizes the brightness value of the left grayscale image and the chromaticity value (value indicating chromaticity (hue and saturation)) of the captured color image. As a result, the pixel value of the pixel of the left display image corresponding to the pixel of the left grayscale image is determined, and a color image is generated as the left display image. Similarly, the synthesis unit 203 synthesizes the pixel value of the pixel of the right grayscale image and the pixel value of the pixel of the captured color image corresponding to the pixel of the right grayscale image based on the association result from the coordinate conversion unit 202. In the first embodiment, the synthesis unit 203 synthesizes the brightness value of the right grayscale image and the chromaticity value of the captured color image. As a result, the pixel values ​​of the pixels of the right display image corresponding to the pixels of the right grayscale image are determined, and a color image is generated as the right display image.

[0044] The position / attitude estimation unit 204 estimates at least one of the position and attitude of the HMD 100 based on the grayscale image obtained by the stereo camera 101. This process can use various known techniques (for example, SLAM (Simultaneous Localization and Mapping)).

[0045] The drawing unit 205 draws (synthesizes) a graphic (e.g., a virtual subject) on the left display image as needed, and displays the graphic on the left display unit 104L. Similarly, the drawing unit 205 draws a graphic on the right display image as needed, and displays the graphic on the right display unit 104R. For example, the graphic is synthesized based on the estimation result from the position / posture estimation unit 204.

[0046] Figure 4 FIG. 1 is a data flow diagram of the display image generation process according to the first embodiment. The arrows indicate the flow of data. Figure 4 , pixel-based processing of a left grayscale image, pixel-based processing of a right grayscale image, pixel-based processing of a captured color image, pixel-based processing of a left display image, and pixel-based processing of a right display image are shown respectively. Figure 4 The display image generation processing in is realized by CPU 106 expanding the program stored in ROM 107 into RAM 108 and executing the program. For example, when HMD 100 is started, Figure 4 The display image generation process in starts.

[0047] In step S401, the depth map generation unit 201 acquires the left grayscale image (luminance value of each pixel of the left grayscale image) and the right grayscale image (luminance value of each pixel of the right grayscale image) from the stereo camera 101. In addition, the synthesis unit 203 acquires the captured color image (chromaticity value of each pixel of the captured color image) from the monocular camera 102.

[0048] In step S402, the depth map generation unit 201 determines the depth corresponding to the pixel of the left grayscale image based on the left grayscale image and the right grayscale image acquired in step S401. The depth corresponding to the pixel of the left grayscale image is determined for each pixel of the left grayscale image. As a result, a left depth map indicating the depth of each pixel of the left grayscale image is generated. Similarly, the depth map generation unit 201 determines the depth corresponding to the pixel of the right grayscale image based on the left grayscale image and the right grayscale image acquired in step S401. The depth corresponding to the pixel of the right grayscale image is determined for each pixel of the right grayscale image. As a result, a right depth map indicating the depth of each pixel of the right grayscale image is generated.

[0049] In step S403, the coordinate conversion unit 202 projects the coordinates of each pixel of the left grayscale image onto the captured color image based on the left depth map generated in step S402. This allows the coordinates of each pixel of the left grayscale image to be associated with the coordinates of the captured color image. Similarly, the coordinate conversion unit 202 projects the coordinates of each pixel of the right grayscale image onto the captured color image based on the right depth map generated in step S402. This allows the coordinates of each pixel of the right grayscale image to be associated with the coordinates of the captured color image.

[0050] In step S404, the synthesis unit 203 synthesizes the brightness values ​​of the pixels of the left grayscale image with the chromaticity values ​​of the pixels of the captured color image corresponding to the pixels of the left grayscale image. Based on the association relationship obtained in step S403 (the association relationship between the coordinates of the left grayscale image and the coordinates of the captured color image), this processing is performed on each pixel of the left grayscale image. Similarly, the synthesis unit 203 synthesizes the brightness values ​​of the pixels in the right grayscale image with the chromaticity values ​​of the pixels in the captured color image corresponding to the pixels in the right grayscale image. Based on the association relationship obtained in step S403 (the association relationship between the coordinates of the right grayscale image and the coordinates of the captured color image), this processing is performed on each pixel of the right grayscale image.

[0051] In step S405, the coordinate conversion unit 202 projects the coordinates of each pixel of the left grayscale image onto the left display image based on the left depth map generated in step S402. This allows the coordinates of each pixel of the left grayscale image to be associated with the coordinates of the left display image. Similarly, the coordinate conversion unit 202 projects the coordinates of each pixel of the right grayscale image onto the right display image based on the right depth map generated in step S402. This allows the coordinates of each pixel of the right grayscale image to be associated with the coordinates of the right display image.

[0052] In step S406, the synthesis unit 203 determines the pixel value (synthesized pixel value) obtained in step S405 for the pixel of the left grayscale image corresponding to the pixel of the left display image as the pixel value of the pixel in the left display image. Based on the association relationship obtained in step S403 (the association relationship between the coordinates of the left grayscale image and the coordinates of the left display image), this processing is performed for each pixel of the left display image. Similarly, the synthesis unit 203 determines the pixel value (synthesized pixel value) obtained in step S405 for the pixel of the right grayscale image corresponding to the pixel of the right display image as the pixel value of the pixel in the right display image. Based on the association relationship obtained in step S403 (the association relationship between the coordinates of the right grayscale image and the coordinates of the right display image), this processing is performed for each pixel of the right display image.

[0053] As described above, according to the first embodiment, by using distance information, an appropriate display image for representing a scene close to an actual scene can be obtained. For example, a display image for showing a scene of a subject viewed from a position where the user's eyes are assumed to be located can be obtained. In addition, since the left camera unit 101L and the right camera unit 101R of the stereo camera 101 each use a camera unit having lower performance than the monocular camera 102 from a certain perspective, increases in processing load and device costs can be suppressed.

[0054] The imaging device for acquiring distance information is not limited to a stereo camera. For example, a ToF (Time of Flight) camera can be used as an imaging device (imaging unit) for acquiring distance information. In this case, an image captured by a ToF camera can be used as a depth map (distance information).

[0055] An example of generating two display images (left display image and right display image) corresponding to two images acquired by a stereo camera, respectively, has been described, but the present invention is not limited thereto. For example, a left display image may be generated by the above method, and the left display image may also be used as a right display image. In addition, only one of the left depth map and the right depth map may be generated, and the left display image and the right display image may be generated using the generated one depth map.

[0056] The imaging unit of the stereo camera does not have to be an imaging unit that obtains grayscale images, and the monocular camera does not have to be an imaging unit that obtains color images. For example, an imaging unit that obtains color images can be used as each of the imaging units of the stereo camera and the monocular camera. An imaging unit that obtains color images can be used as the imaging unit of the stereo camera, and an imaging unit that obtains grayscale images can be used as the monocular camera. The resolution of the image obtained by the monocular camera can be higher than the resolution of the image obtained by the imaging unit of the stereo camera. In this case, a display image having a higher resolution than the resolution of the image obtained by the imaging unit of the stereo camera can be generated based on the above-mentioned association relationship of the coordinates.

[0057] The method for synthesizing the pixel values ​​of the image obtained by the imaging unit of the stereo camera and the pixel values ​​of the image obtained by the monocular camera is not limited to the above method. For example, the chromaticity value of the image obtained by the imaging unit of the stereo camera may be considered, or the brightness value of the image obtained by the monocular camera may be considered. The pixel value of the image obtained by the monocular camera may be determined as the pixel value of the display image.

[0058] The image obtained by the camera unit of the stereo camera may be an HDR (high dynamic range) image, the image obtained by the monocular camera may be an SDR (standard dynamic range) image, and the displayed image may be an HDR image. If the image obtained by the camera unit of the stereo camera is an HDR image, a depth map (distance information) may be obtained with stable accuracy. The image obtained by the camera unit of the stereo camera may be an SDR image, the image obtained by the monocular camera may be an HDR image, and the displayed image may be an HDR image. By using a camera unit for obtaining an SDR image, the cost of the device can be reduced.

[0059] The virtual viewpoint is a position different from the position of the imaging unit of the stereo camera and the position of the monocular camera, but the present invention is not limited thereto. For example, the position of the imaging unit of the stereo camera may be a virtual viewpoint. If a high-resolution color image can be displayed, a scene close to a real scene can be provided to the user. Therefore, when a high-resolution grayscale image is obtained by the imaging unit of the stereo camera, an image obtained by coloring the grayscale image obtained by the imaging unit can be used as a display image. When a low-resolution color image is obtained by the imaging unit of the stereo camera, an image obtained by increasing the resolution of the color image obtained by the imaging unit can be used as a display image. When a low-resolution grayscale image is obtained by the imaging unit of the stereo camera, an image obtained by coloring the grayscale image obtained by the imaging unit and increasing the resolution of the grayscale image can be used as a display image.

[0060] <Second Embodiment>

[0061] A second embodiment of the present invention will be described. Note that hereinafter, description of the same points as in the first embodiment (eg, the same configuration and processing as in the first embodiment) will be omitted, and differences from the first embodiment will be described.

[0062] In general, when acquiring a depth map (distance information), a high-precision depth (distance) cannot be obtained for a portion of the grayscale image obtained by the stereo camera (the portion does not have (has few) features, for example, a portion with a uniform brightness value). Therefore, in the second embodiment, the projection unit 103 projects a pattern image of invisible light (for example, a dot pattern of infrared light). The stereo camera 101 can detect invisible light, and the projected pattern image appears in the grayscale image obtained by the stereo camera 101. For example, by considering the position and shape of the pattern image appearing in the grayscale image, a high-precision depth map can be obtained. In the example of the dot pattern, a high-precision depth map can be obtained by considering the arrangement and spacing of multiple points in the dot pattern appearing in the grayscale image. The projected pattern image does not appear in the color image obtained by the monocular camera 102.

[0063] However, if a grayscale image in which a pattern image appears is used, the pattern image will also appear in the displayed image. Therefore, in the second embodiment, the projection unit 103 repeats the projection and non-projection of the pattern image (invisible light). The stereo camera 101 (each of the left camera unit 101L and the right camera unit 101R) repeats the imaging to obtain a first grayscale image in which a pattern image appears and a second grayscale image in which a pattern image does not appear. Then, the CPU 106 determines a depth map from the first grayscale image, and determines the pixel value of the displayed image based on the depth map by synthesizing the pixel value of the second grayscale image and the pixel value of the captured color image.

[0064] Figure 5 1 is a block diagram showing a functional configuration of the CPU 106 according to the second embodiment. In the second embodiment, the CPU 106 has a depth map generation unit 201, a coordinate conversion unit 202, a synthesis unit 203, a position / posture estimation unit 204, a drawing unit 205, and an image selection unit 501.

[0065] The image selection unit 501 repeatedly acquires the left grayscale image and the right grayscale image from the stereo camera 101. The image selection unit 501 outputs the left grayscale image and the right grayscale image as the first grayscale image (grayscale image with the pattern image) to the depth map generation unit 201. Then, the image selection unit 501 outputs the left grayscale image and the right grayscale image as the second grayscale image (grayscale image without the pattern image) to the synthesis unit 203 and the position / attitude estimation unit 204.

[0066] For example, Figure 6 As shown, the stereo camera 101 alternately acquires the first grayscale image and the second grayscale image as the left grayscale image. Therefore, the image selection unit 501 alternately acquires the first grayscale image and the second grayscale image from the stereo camera 101 as the left grayscale image. Similarly, the stereo camera 101 alternately acquires the first grayscale image and the second grayscale image as the right grayscale image. Therefore, the image selection unit 501 alternately acquires the first grayscale image and the second grayscale image from the stereo camera 101 as the right grayscale image.

[0067] Note that although an example of alternately acquiring a first grayscale image of one frame and a second grayscale image of one frame has been described, the present invention is not limited thereto. For example, the acquisition of a first grayscale image of one frame and the acquisition of second grayscale images of multiple frames (e.g., two frames, three frames, or five frames) may be performed alternately. The acquisition of first grayscale images of multiple frames and the acquisition of second grayscale images of one frame may be performed alternately. The acquisition of first grayscale images of multiple frames and the acquisition of second grayscale images of multiple frames may be performed alternately.

[0068] If the acquisition frequency of the second grayscale image (the grayscale image without the pattern image) increases, the frame rate of the displayed image will also increase. Therefore, when the smoothness of the displayed image is prioritized, it is preferred to increase the frequency of acquiring the second grayscale image. If the frequency of acquiring the second grayscale image increases, the frequency of acquiring the first grayscale image (the grayscale image with the pattern image) will decrease. However, since the depth map can be updated based on the change of the grayscale image without using the first grayscale image, there is no problem even if the frequency of acquiring the first grayscale image is low.

[0069] In the second embodiment, the display frame rate of the display image is lower than the frame rate of the image pickup performed by the stereo camera 101 (each of the left image pickup unit 101L and the right image pickup unit 101R). For example, the frame rate of the display is 60fps, 90fps, or 120fps, and the frame rate of the image pickup is 90fps, 120fps, or 240fps. The frame rate of the image pickup performed by the monocular camera 102 can be consistent with the frame rate of the display of the display image. Therefore, the stereo camera 101 (each of the left image pickup unit 101L and the right image pickup unit 101R) can capture images at a higher frame rate than the monocular camera 102.

[0070] As described above, according to the second embodiment, an image in which invisible light is captured is used to obtain (judge) distance information, and an image in which invisible light is not captured is used to determine the pixel value of the display image. In this way, high-precision distance information can be obtained while suppressing the appearance of invisible light in the display image. In addition, based on the high-precision distance information, a more appropriate display image (an appropriate display image representing a scene closer to the actual scene) can be obtained.

[0071] Note that images captured with invisible light can be used to obtain a depth map through stereo measurement, and images not captured with invisible light can be used to obtain position or pose through SLAM.

[0072] The multiple display modes that can be selectively set for the HMD 100 may include: a display mode in which an image of the outside world (real space) is displayed on each of the left display unit 104L and the right display unit 104R; and a display mode in which an image of the outside world is not displayed. The display mode in which an image of the outside world is displayed is, for example, an AR (augmented reality) mode. The display mode in which an image of the outside world is not displayed is, for example, a display mode in which an image of a virtual space is displayed on each of the left display unit 104L and the right display unit 104R. The display mode in which an image of a virtual space is displayed is, for example, a VR (virtual reality) mode.

[0073] When the AR mode is set, the projection unit 103 can repeatedly project and not project the invisible light, and when the AR mode is not set (for example, when the VR mode is set), the invisible light can be continuously projected. By doing so, when the AR mode is not set, a high-precision depth map (distance information) can be continuously acquired, and processing based on the depth map can be appropriately performed.

[0074] When the AR mode is not set (for example, when the VR mode is set), the stereo camera 101 (each of the left camera unit 101L and the right camera unit 101R) may capture images at a lower frame rate than when the AR mode is set. For example, when the AR mode is not set, the stereo camera 101 may capture images so that the frequency of capturing images in which invisible light is captured is reduced. The stereo camera 101 may capture images so that only images in which invisible light is captured are captured. By doing so, the power consumption of the HMD 100 can be reduced.

[0075] <Third Embodiment>

[0076] A third embodiment of the present invention will be described. Note that hereinafter, description of the same points as in the first embodiment (eg, the same configuration and processing as in the first embodiment) will be omitted, and differences from the first embodiment will be described.

[0077] The eye positions of the user wearing the HMD 100 (the relative position of the left eye with respect to the left display unit 104L and the relative position of the right eye with respect to the right display unit 104R), that is, the actual viewpoint, depend on the user. Figure 7 As shown, when the relative position of the user's eye relative to the display unit changes, even if the relative position of the subject relative to the display unit does not change, the relative position of the subject relative to the eye changes, and the image to be displayed on the display unit changes. Therefore, in the third embodiment, the CPU 106 determines the viewpoint of the left eye (the relative position of the left eye relative to the left display unit 104L) based on the image of the left eye (left eye image) captured by the left eye camera unit 105L, and generates a left display image for representing the scene of the subject viewed from the viewpoint. Similarly, the CPU 106 determines the viewpoint of the right eye (the relative position of the right eye relative to the right display unit 104R) based on the image of the right eye (right eye image) captured by the right eye camera unit 105R, and generates a right display image for representing the scene of the subject viewed from the viewpoint.

[0078] The human eye has a characteristic of being able to detect the color in the central field of view with good sensitivity but not being able to detect the color in the peripheral field of view with good sensitivity. Therefore, in the third embodiment, the CPU 106 processes the portion of the left display image corresponding to the central field of view of the left eye in a manner different from the portion of the left display image corresponding to the peripheral field of view of the left eye based on the left eye image. As in the first embodiment, the CPU 106 determines the pixel value of the portion of the left display image corresponding to the central field of view of the left eye by synthesizing the pixel value of the left grayscale image and the pixel value of the captured color image. However, the CPU 106 determines the pixel value of the portion of the left display image corresponding to the peripheral field of view of the left eye based on the pixel value of the left grayscale image. As a result, only the portion of the left display image corresponding to the central field of view of the left eye is colored, and the pixel value of the portion of the left display image corresponding to the peripheral field of view of the left eye corresponds to the grayscale pixel value.

[0079] Similarly, the CPU 106 processes the portion of the right display image corresponding to the central field of view of the right eye in a manner different from the portion of the right display image corresponding to the peripheral field of view of the right eye based on the right eye image. As in the first embodiment, the CPU 106 determines the pixel value of the portion of the right display image corresponding to the central field of view of the right eye by synthesizing the pixel value of the right grayscale image and the pixel value of the captured color image. However, the CPU 106 determines the pixel value of the portion of the right display image corresponding to the peripheral field of view of the right eye based on the pixel value of the right grayscale image. As a result, only the portion of the right display image corresponding to the central field of view of the right eye is colored, and the pixel value of the portion corresponding to the peripheral field of view of the right eye corresponds to the grayscale pixel value.

[0080] Figure 8 1 is a block diagram showing a functional configuration of the CPU 106 according to the third embodiment. In the third embodiment, the CPU 106 has a depth map generation unit 201, a coordinate conversion unit 202, a synthesis unit 203, a position / posture estimation unit 204, a rendering unit 205, and an eye image processing unit 801.

[0081] The eye image processing unit 801 obtains the left eye image from the left eye camera unit 105L and uses the left eye image for processing. For example, the eye image processing unit 801 detects the left eye from the left eye image and determines the viewpoint of the left eye (the relative position of the left eye relative to the left display unit 104L). The eye image processing unit 801 also obtains the gaze information (gaze detection) related to the gaze of the left eye based on the left eye image. The gaze information includes, for example, at least one of the angle information indicating the gaze direction and the position information indicating the gaze position on the display surface of the left display unit 104L. Similarly, the eye image processing unit 801 obtains the right eye image from the right eye camera unit 105R and uses the right eye image for processing. Various known technologies can be used for the processing of the eye image processing unit 801. For example, multiple light sources can be provided around the left eye camera unit 105L, and the gaze information can be obtained based on the positional relationship between the multiple corneal reflection images corresponding to the multiple light sources and the pupil.

[0082] The processing of the synthesis unit 203 is substantially the same as that of the first embodiment. However, the left display image is generated based on the viewpoint of the left eye (not a predetermined virtual viewpoint but a viewpoint of the left eye determined by the eye image processing unit 801). In addition, based on the gaze information of the left eye acquired by the eye image processing unit 801, a left display image is generated, wherein in the left display image, only the portion corresponding to the central field of view of the left eye is colored and the pixel value of the portion corresponding to the peripheral field of view of the left eye corresponds to the grayscale pixel value. Similarly, the right display image is generated based on the viewpoint of the right eye (not a predetermined virtual viewpoint but a viewpoint of the right eye determined by the eye image processing unit 801). In addition, based on the gaze information of the right eye acquired by the eye image processing unit 801, a right display image is generated, wherein in the right display image, only the portion corresponding to the central field of view of the right eye is colored and the pixel value of the portion corresponding to the peripheral field of view of the right eye corresponds to the grayscale pixel value.

[0083] Fig. 9 FIG. 4 is a data flow diagram of the display image generation process according to the third embodiment. The arrows indicate the flow of data. Fig. 9 , pixel-based processing of a left grayscale image, pixel-based processing of a captured color image, and pixel-based processing of a left display image are shown respectively. Fig. 9 The display image generation processing in is realized by CPU 106 expanding the program stored in ROM 107 into RAM 108 and executing the program. For example, when HMD 100 is started, Fig. 9 The display image generation processing in starts. Note that the method for generating the right display image is the same as the method for generating the left display image, so the description of the method for generating the right display image is omitted.

[0084] The processing in steps S901 to S903 is the same as that in the first embodiment ( Figure 4 ) is the same as the processing in steps S401 to S403.

[0085] In step S911 , the eye image processing unit 801 detects the position of the left eye from the left eye image obtained by the left eye imaging unit 105L, and determines the viewpoint of the left eye (the relative position of the left eye with respect to the left display unit 104L).

[0086] In step S912 , the eye image processing unit 801 determines the posture (gaze direction) of the left eye based on the left eye image.

[0087] In step S913, the coordinate conversion unit 202 determines the portion of the display surface of the left display unit 104L corresponding to the central field of view of the left eye based on the posture of the left eye determined in step S912. For example, the coordinate conversion unit 202 determines the portion corresponding to the predetermined viewing angle centered on the gaze direction of the left eye as the portion corresponding to the central field of view. The predetermined viewing angle is, for example, a viewing angle of more than 30 degrees and less than 50 degrees. Then, based on the left depth map generated in step S902, the coordinate conversion unit 202 projects the portion of the display surface of the left display unit 104L corresponding to the central field of view of the left eye onto the left grayscale image. In this way, the portion of the left grayscale image corresponding to the central field of view of the left eye is determined.

[0088] In step S904, similar to the first embodiment ( Figure 4 ), the synthesis unit 203 synthesizes the brightness values ​​of the pixels in the left grayscale image and the chrominance values ​​of the pixels in the captured color image corresponding to the pixels in the left grayscale image. However, the processing of step S904 is performed only on the portion determined in step S913 (the portion of the left grayscale image corresponding to the central field of view of the left eye).

[0089] In step S905, similar to the first embodiment ( Figure 4 ), the coordinate conversion unit 202 projects the coordinates of each pixel in the left grayscale image onto the left display image. However, the coordinate conversion unit 202 performs the projection based on the viewpoint of the left eye determined in step S911 rather than based on a predetermined virtual viewpoint.

[0090] The processing of step S906 is similar to that of the first embodiment ( Figure 4). However, as described above, the processing of step S904 is performed only on the portion of the left grayscale image corresponding to the central field of view of the left eye. Therefore, the processing of step S906 generates a left display image in which only the portion corresponding to the central field of view of the left eye is colored and the pixel values ​​of the portion corresponding to the peripheral field of view of the left eye correspond to the grayscale pixel values.

[0091] As described above, according to the third embodiment, the viewpoint of the user is determined based on the image of the user's eyes. Then, a display image is generated for representing the scene of the subject viewed from the determined viewpoint. In this way, a more appropriate display image (an appropriate display image representing a scene closer to the actual scene) can be obtained. In addition, pixel values ​​are synthesized only for the portion corresponding to the user's central field of view. In this way, a more appropriate display image (an appropriate display image representing a scene closer to the actual scene) can also be obtained. By limiting the portion for which pixel values ​​are synthesized, the processing load can also be reduced.

[0092] Note that it is not necessary to perform gaze detection. In this case, for example, CPU 106 determines the pixel value of the center part of the left display image by synthesizing the pixel value of the left grayscale image and the pixel value of the captured color image. Then, CPU 106 determines the pixel value of the peripheral part (part other than the center part) of the left display image from the pixel value of the left grayscale image. Similarly, CPU 106 determines the pixel value of the center part of the right display image by synthesizing the pixel value of the right grayscale image and the pixel value of the captured color image. Then, CPU 106 determines the pixel value of the peripheral part (part other than the center part) of the right display image from the pixel value of the right grayscale image. In this way, a more suitable display image (a suitable display image representing a scene closer to the actual scene) can also be obtained. Considering the range of the user's gaze change, a part wider than the central field of view can be predetermined as the center part. The center part can be fixed or can be changed by the user.

[0093] Generally, when acquiring a depth map (distance information), it is impossible to obtain a high-precision depth (distance) of the peripheral part of the image. Therefore, processing based on the depth map may not be properly performed in the peripheral part. For example, when coloring the peripheral part of a grayscale image, misalignment between the color and the subject may occur. According to the above configuration in which pixel values ​​are not synthesized for the peripheral part, the occurrence of such a problem can be suppressed.

[0094] <Fourth Embodiment>

[0095] A fourth embodiment of the present invention will be described. Note that, hereinafter, descriptions of points that are the same as those in the first embodiment (e.g., the same configuration and processing as those in the first embodiment) will be omitted, and differences from the first embodiment will be described. hereinafter, matters related to the right eye will be described. Matters related to the left eye are similar to those related to the right eye, so descriptions of matters related to the left eye will be omitted.

[0096] When acquiring a depth map (distance information), the depth (distance) of some positions in the display image may not be acquired. In the fourth embodiment, the CPU 106 (synthesizing unit 203) determines the pixel value of such a position by interpolation.

[0097] For example, a position visible from the right eye (viewpoint) may be hidden by the right camera unit 101R and not appear in the right grayscale image. In this case, for a position in the right display image that is not shown in the right grayscale image, the depth (distance) cannot be acquired and the pixel value cannot be determined. This state is Fig.10 Shown in.

[0098] The configuration of the synthesis unit 203 for solving the first problem is as follows Fig.11 As shown. Fig.11 In the embodiment, the synthesis unit 203 has a gridding unit 1101 and a texture mapping unit 1102. Fig.11 The synthesis unit 203 in also performs the same processing (synthesis of pixel values) as the synthesis unit 203 in the first embodiment.

[0099] The gridding unit 1101 sets a plurality of regions ( Fig.10 Meshing in ), where the vertices of the multiple areas are positions of the right grayscale image or the captured color image whose depths (distances) are indicated by the depth map (distance information).

[0100] The texture mapping unit 1102 performs texture mapping using a plurality of textures corresponding to the plurality of regions set by the gridding unit 1101. In this way, the pixel values ​​of the positions in the right display image whose depths are not indicated by the depth map are determined. In the fourth embodiment, a plurality of regions of the right grayscale image after pixel value synthesis, i.e., a plurality of regions of the colored right grayscale image, are used as a plurality of textures.

[0101] Note that multiple areas of the captured color image can be used as multiple textures. The number of multiple positions (multiple vertices) for meshing can be the same as the number of pixels in the right grayscale image, or can be less than the number of pixels in the right grayscale image. By reducing the number of multiple positions for meshing, the processing load can be reduced. The interpolation method is not limited to the above method. For example, without texture mapping, the depth value can be interpolated so that the pixel value of the undetermined position (the position where the pixel value is undetermined) of the display image can be determined in the same manner as the pixel value of the determined position (the position where the pixel value is determined). After obtaining the right display image where some pixel values ​​have not yet been determined, the pixel value of the undetermined position can be determined by using the interpolation processing (e.g., linear interpolation) of the pixel values ​​around the undetermined position of the right display image.

[0102] In addition, if Fig.12 As shown, the end of the range corresponding to the right display image may be outside the acquisition range of the depth map (the range where the imaging range of the left imaging unit 101L and the imaging range of the right imaging unit 101R overlap). In this case, the depth cannot be acquired for the end of the right display image, and the pixel value cannot be determined.

[0103] The configuration of the synthesis unit 203 for solving the second problem is as follows Fig.13 As shown. Fig.13 In the figure, the synthesis unit 203 has a pixel value synthesis unit 1301 and an interpolation unit 1302.

[0104] The pixel value synthesis unit 1301 performs the same processing (synthesis of pixel values) as the synthesis unit 203 of the first embodiment. In addition, the pixel value synthesis unit 1301 generates a mask image for representing an undetermined area (an area where pixel values ​​have not yet been determined) of the right display image.

[0105] The interpolation unit 1302 performs inpainting processing on the right display image generated by the pixel value synthesis unit 1301. When the right display image generated by the pixel value synthesis unit 1301 is an incomplete right display image including an undetermined area, the pixel value of the undetermined area is determined by inpainting, and a complete right display image is generated. Various known techniques can be used for inpainting. The interpolation unit 1302 is, for example, a calculator (trained model) that uses AI (artificial intelligence) (such as GAN (generative adversarial network) etc.) for inpainting. In a fourth embodiment, the right display image and the mask image generated by the pixel value synthesis unit 1301 are input to the interpolation unit 1302, and the complete right display image is output from the interpolation unit 1302.

[0106] The first question can also be solved by using Fig.13 The synthesis unit 203 in the embodiment can be used to solve the problem.

[0107] As described above, according to the fourth embodiment, a complete display image is generated by interpolation. In this way, a more appropriate display image (an appropriate display image representing a scene closer to a real scene) can be obtained.

[0108] Note that the synthesis unit 203 can generate a display image with a colored portion by synthesizing pixel values, and color the other portions of the display image based on the coloring result of the portion. In order to color the other portions, a calculator (trained model) that performs coloring using AI (e.g., GAN) can be used. For example, the calculator receives an input of a display image with a colored portion and a mask image representing the color distribution of the portion, and outputs a complete display image.

[0109] The above-described embodiments (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0110] <Other embodiments>

[0111] The present invention can also be implemented by supplying a program for implementing one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium and causing one or more processors in a computer of the system or device to read and execute the program. In addition, the present invention can also be implemented by a circuit (e.g., ASIC) for implementing one or more functions.

[0112] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0113] (Configuration 1)

[0114] An image processing device capable of being used together with a head-mounted display device, the head-mounted display device comprising a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the image processing device comprising:

[0115] A first acquisition unit, configured to acquire a first image of the subject captured by the first camera unit;

[0116] a second acquiring unit configured to acquire a second image of the subject photographed by the second camera unit; and

[0117] A generating unit configured to generate a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

[0118] (Configuration 2)

[0119] The image processing apparatus according to configuration 1, wherein:

[0120] The first image is a grayscale image,

[0121] The second image is a color image, and

[0122] The generating unit generates a color image as the display image.

[0123] (Configuration 3)

[0124] The image processing device according to configuration 1 or 2, wherein:

[0125] The first camera unit is a camera unit of a stereo camera.

[0126] (Configuration 4)

[0127] An image processing device according to any one of Configurations 1 to 3, wherein:

[0128] The second imaging unit is an imaging unit of a monocular camera.

[0129] (Configuration 5)

[0130] An image processing device according to any one of configurations 1 to 4, wherein:

[0131] The first camera unit includes a left camera unit and a right camera unit, and

[0132] The second camera unit is arranged between the left camera unit and the right camera unit.

[0133] (Configuration 6)

[0134] An image processing device according to any one of Configurations 1 to 5, wherein:

[0135] The generation unit generates the display image by synthesizing a graphic with an image of the outside world generated based on the distance information and the second image.

[0136] (Configuration 7)

[0137] The image processing apparatus according to configuration 6, further comprising:

[0138] an estimating unit configured to estimate at least one of a position and a posture of the head mounted display device,

[0139] The generating unit generates the graphic based on at least one of the position and the posture.

[0140] (Configuration 8)

[0141] The image processing device according to configuration 7, wherein:

[0142] The estimating unit estimates at least one of the position and the posture based on the first image.

[0143] (Configuration 9)

[0144] An image processing device according to any one of Configurations 1 to 8, wherein:

[0145] The second image has a higher resolution than the first image, and

[0146] The generating unit generates a display image having a higher resolution than a resolution of the first image.

[0147] (Configuration 10)

[0148] An image processing device according to any one of configurations 1 to 9, wherein:

[0149] The first camera unit is a global shutter camera unit, and

[0150] The second camera unit is a rolling shutter type camera unit.

[0151] (Configuration 11)

[0152] The image processing device according to any one of Configurations 1 to 10, wherein:

[0153] The first imaging unit captures images at a frame rate higher than a frame rate of the second imaging unit.

[0154] (Configuration 12)

[0155] The image processing device according to any one of configurations 1 to 11, wherein:

[0156] The first image is an SDR image,

[0157] The second image is a HDR image, and

[0158] The displayed image is an HDR image.

[0159] (Configuration 13)

[0160] The image processing device according to any one of Configurations 1 to 12, wherein:

[0161] The head mounted display device includes a projection unit configured to repeatedly project and not project invisible light onto an imaging range of the first imaging unit.

[0162] (Configuration 14)

[0163] The image processing apparatus according to configuration 13, wherein:

[0164] The first imaging unit repeatedly performs imaging to obtain a first image in which the invisible light is captured and a first image in which the invisible light is not captured.

[0165] (Configuration 15)

[0166] The image processing apparatus according to configuration 14, wherein:

[0167] The generating unit determines distance information from the first image captured by the invisible light.

[0168] (Configuration 16)

[0169] The image processing apparatus according to configuration 15, wherein:

[0170] The generation unit generates the display image based on the distance information, the first image and the second image, neither of which captures the invisible light.

[0171] (Configuration 17)

[0172] An image processing device according to any one of Configurations 13 to 16, wherein:

[0173] In a case where a display mode for displaying an image of the outside world on the display unit is set, the projection unit repeatedly projects and does not project the invisible light onto the imaging range of the first imaging unit, and

[0174] When the display mode is not set, the projection unit continues to project the invisible light onto the imaging range of the first imaging unit.

[0175] (Configuration 18)

[0176] An image processing device according to any one of Configurations 1 to 17, wherein:

[0177] The head mounted display device includes a third camera unit configured to capture eyes of a user wearing the head mounted display device, and

[0178] The generation unit generates a display image representing a view of the subject viewed from a position based on a third image of the eye captured by the third imaging unit.

[0179] (Configuration 19)

[0180] A head mounted display device, comprising:

[0181] A camera unit configured to obtain distance information;

[0182] a projection unit configured to project invisible light onto an imaging range of the imaging unit; and

[0183] Display unit,

[0184] The projection unit repeatedly projects and does not project the invisible light onto the imaging range of the imaging unit.

[0185] (Configuration 20)

[0186] The image processing apparatus according to configuration 19, wherein:

[0187] The imaging unit repeatedly performs imaging to obtain an image in which the invisible light is captured and an image in which the invisible light is not captured.

[0188] (Configuration 21)

[0189] The head mounted display device according to configuration 19 or 20, wherein:

[0190] In a case where a display mode for displaying an image of the outside world on the display unit is set, the projection unit repeatedly projects and does not project the invisible light onto the imaging range of the imaging unit, and

[0191] When the display mode is not set, the projection unit continues to project the invisible light onto the imaging range of the imaging unit.

[0192] (Configuration 22)

[0193] A head mounted display device, comprising:

[0194] the first camera unit;

[0195] the second camera unit;

[0196] the display unit; and

[0197] An image processing device according to any one of configurations 1 to 18.

[0198] (Method 23)

[0199] A control method for an image processing device, the image processing device being usable together with a head mounted display device, the head mounted display device comprising a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the control method comprising:

[0200] A first acquisition step is used to acquire a first image of the subject captured by the first camera unit;

[0201] A second acquisition step is used to acquire a second image of the subject photographed by the second camera unit; and

[0202] A generating step for generating a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

[0203] (Method 24)

[0204] A control method for a head mounted display device, the head mounted display device comprising a camera unit configured to obtain distance information, a projection unit configured to project invisible light onto a camera range of the camera unit, and a display unit, the control method comprising:

[0205] The method is used for controlling the projection unit so that the projection unit repeats the steps of projecting and not projecting the invisible light onto the imaging range of the imaging unit.

[0206] (Procedure 25)

[0207] A program for causing a computer to execute the various steps of the control method according to method 23 or 24.

[0208] The present invention is not limited to the above-described embodiments, and various modifications and corrections may be made thereto without departing from the spirit and scope of the present invention. Therefore, in order to disclose the scope of the present invention, the following claims are attached.

[0209] This application claims priority based on Japanese Patent Application No. 2022-103268 filed on June 28, 2022, the disclosure of which is hereby incorporated by reference in its entirety.

[0210] [reference numerals list]

[0211] 100 HMD

[0212] 101 Stereo Camera

[0213] 102 Monocular Camera

[0214] 103 Projection unit

[0215] 104 Left display unit

[0216] 104 Right display unit

[0217] 106 CPU

Claims

1. An image processing device capable of being used together with a head mounted display device, the head mounted display device comprising a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the image processing device comprising: A first acquisition unit, configured to acquire a first image of the subject captured by the first camera unit; A second acquiring unit, configured to acquire a second image of the subject captured by the second camera unit; as well as A generating unit configured to generate a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

2. The image processing device according to claim 1, wherein: The first image is a grayscale image, The second image is a color image, and The generating unit generates a color image as the display image.

3. The image processing device according to claim 2, wherein: The first camera unit is a camera unit of a stereo camera.

4. The image processing device according to claim 3, wherein: The second imaging unit is an imaging unit of a monocular camera.

5. The image processing device according to claim 4, wherein: The first camera unit includes a left camera unit and a right camera unit, and The second camera unit is arranged between the left camera unit and the right camera unit.

6. The image processing device according to claim 1, wherein: The generation unit generates the display image by synthesizing a graphic with an image of the outside world generated based on the distance information and the second image.

7. The image processing apparatus according to claim 6, further comprising: an estimating unit configured to estimate at least one of a position and a posture of the head mounted display device, The generating unit generates the graphic based on at least one of the position and the posture.

8. The image processing device according to claim 7, wherein: The estimation unit estimates at least one of the position and the posture based on the first image.

9. The image processing apparatus according to claim 1, wherein: The second image has a higher resolution than the first image, and The generating unit generates a display image having a higher resolution than a resolution of the first image.

10. The image processing apparatus according to claim 1, wherein: The first camera unit is a global shutter camera unit, and The second camera unit is a rolling shutter type camera unit.

11. The image processing apparatus according to claim 1, wherein: The first imaging unit captures images at a frame rate higher than a frame rate of the second imaging unit.

12. The image processing apparatus according to claim 1, wherein: The first image is an SDR image, The second image is a HDR image, and The displayed image is an HDR image.

13. The image processing apparatus according to claim 1, wherein: The head mounted display device includes a projection unit configured to repeatedly project and not project invisible light onto an imaging range of the first imaging unit.

14. The image processing apparatus according to claim 13, wherein: The first imaging unit repeatedly performs imaging to obtain a first image in which the invisible light is captured and a first image in which the invisible light is not captured.

15. The image processing apparatus according to claim 14, wherein: The generating unit determines distance information from the first image captured by the invisible light.

16. The image processing apparatus according to claim 15, wherein: The generation unit generates the display image based on the distance information, the first image and the second image, neither of which captures the invisible light.

17. The image processing apparatus according to claim 13, wherein: In a case where a display mode for displaying an image of the outside world on the display unit is set, the projection unit repeatedly projects and does not project the invisible light onto the imaging range of the first imaging unit, and When the display mode is not set, the projection unit continues to project the invisible light onto the imaging range of the first imaging unit.

18. The image processing apparatus according to claim 1, wherein: The head mounted display device includes a third camera unit configured to capture eyes of a user wearing the head mounted display device, and The generation unit generates a display image representing a view of the subject viewed from a position based on a third image of the eye captured by the third imaging unit.

19. A head mounted display device, comprising: A camera unit configured to obtain distance information; a projection unit configured to project invisible light onto an imaging range of the imaging unit; as well as Display unit, The projection unit repeatedly projects and does not project the invisible light onto the imaging range of the imaging unit.

20. The image processing apparatus according to claim 19, wherein: The imaging unit repeatedly performs imaging to obtain an image in which the invisible light is captured and an image in which the invisible light is not captured.

21. The head mounted display device according to claim 19, wherein: In a case where a display mode for displaying an image of the outside world on the display unit is set, the projection unit repeatedly projects and does not project the invisible light onto the imaging range of the imaging unit, and When the display mode is not set, the projection unit continues to project the invisible light onto the imaging range of the imaging unit.

22. A head mounted display device, comprising: the first camera unit; the second camera unit; the display unit; as well as An image processing device according to any one of claims 1 to 18.

23. A control method for an image processing device, the image processing device being usable together with a head mounted display device, the head mounted display device comprising a first camera unit configured to obtain distance information, a second camera unit different from the first camera unit, and a display unit, the control method comprising: A first acquisition step is used to acquire a first image of the subject captured by the first camera unit; A second acquisition step is used to acquire a second image of the subject taken by the second camera unit; as well as A generating step for generating a display image to be displayed on the display unit based on the distance information corresponding to the first image and the second image.

24. A method for controlling a head mounted display device, the head mounted display device comprising a camera unit configured to obtain distance information, a projection unit configured to project invisible light onto a camera range of the camera unit, and a display unit, the control method comprising: The method is used for controlling the projection unit so that the projection unit repeats the steps of projecting and not projecting the invisible light onto the imaging range of the imaging unit.

25. A program for causing a computer to execute each step of the control method according to claim 23 or 24.

Images