Control apparatus, control method, and storage medium

Abstract

To enable a user to confirm an image with sufficiently high image quality when photographing (recording) the image.SOLUTION: A control device according to the present disclosure is a control device for controlling a head-mount type display device including a first imaging section, a second imaging section, a first display section, and a second display section, the control device including a setting section adapted to set either one of a plurality of modes including a first mode and a second mode, and a display control section adapted to display an image taken by the first imaging section with first image quality on the first display section in the first mode, and a control unit configured to control the second display unit to display an image captured by the second image capturing unit with a first image quality, and in a second mode, control the first display unit or the second display unit to stop one of the first image capturing unit and the second image capturing unit and display an image captured by the other image capturing unit with a second image quality higher than the first image quality.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device that controls a head-mounted display device such as a head-mounted display (HMD). [Background technology]

[0002] Mixed reality (MR) technology using a head-mounted display (HMD) has been proposed, as well as technology for recording a composite image, which is a combination of computer graphics (CG) and a background image, as a still image.

[0003] Patent Document 1 discloses a technique for setting a shooting mode using hand gestures. Patent Document 2 discloses a technique for determining which of two image capturing units to use when capturing a still image based on image quality information of each image capturing unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-6502 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-227846 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with the conventional techniques disclosed in Patent Documents 1 and 2, there are cases where a user cannot check an image (for example, a still image) with a sufficiently high image quality when the image is captured (recorded).

[0006] An object of the present invention is to enable a user to check an image with a sufficiently high image quality when the image is captured (recorded). [Means for solving the problem]

[0007] The control device of the present invention is a control device that controls a head-mounted display device having a first imaging means arranged to correspond to the user's right eye, a second imaging means arranged to correspond to the user's left eye, a first display means that presents an image to the user's right eye, and a second display means that presents an image to the user's left eye, and is characterized by having a setting means that sets one of a plurality of modes including a first mode and a second mode, and a control means that, in the first mode, controls so that an image captured by the first imaging means at a first image quality is displayed on the first display means and an image captured by the second imaging means at the first image quality is displayed on the second display means, and that, in the second mode, stops one of the first imaging means and the second imaging means and controls so that an image captured by the other imaging means at a second image quality higher than the first image quality is displayed on the first display means or the second display means. [Effects of the Invention]

[0008] According to the present invention, a user can check an image with a sufficiently high image quality when capturing (recording) an image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an HMD. [Figure 2] 10 is a timing chart showing a series of processes up to displaying a composite image. [Figure 3] 10 is a timing chart showing a series of processes up to displaying a composite image. [Figure 4] 1 is a flowchart according to the first embodiment. [Figure 5] 10 is a flowchart according to a second embodiment. [Figure 6] 10 is a flowchart according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Embodiment 1> A first embodiment of the present invention will be described below. Fig. 1 is a block diagram of an HMD 100 according to the first embodiment.

[0011] The lens 101 is provided to correspond to the left eye of a user wearing the HMD 100 on their head. The lens 101' is provided to correspond to the right eye of a user wearing the HMD 100 on their head. The lenses 101 and 101' are twin lenses that can capture right and left images (background images) with parallax. Reflected light from a subject entering the lens 101 is focused as an optical image on the image sensor 103. Similarly, reflected light from the subject entering the lens 101' is focused as an optical image on the image sensor 103'. The timing generators 102 and 102' generate timing signals. The image sensors 103 and 103' are image sensors formed with elements such as CCDs and CMOS elements that convert optical images into electrical signals. The image sensor 103 operates based on the timing signal output from the timing generator 102 and photoelectrically converts the focused reflected light (optical image of the subject) into an analog signal.

[0012] The analog signal processing unit 104 converts the analog signal from the image sensor 103 into a digital signal by A / D conversion and outputs the digital signal. The digital gain unit 106 performs predetermined signal processing on the digital signal output from the analog signal processing unit 104 and transmits the digital signal to the image processing unit 110. Similarly, the analog signal processing unit 104' converts the analog signal from the image sensor 103' into a digital signal and outputs the digital signal, and the digital gain unit 106' performs predetermined signal processing on the digital signal output from the analog signal processing unit 104' and transmits the digital signal to the image processing unit 110.

[0013] The camera control unit 105 controls the timing generator 102, the image sensor 103, the analog signal processing unit 104, the digital gain unit 106, etc. For example, the camera control unit 105 controls the accumulation time of the image sensor 103 and the gain of the digital gain unit 106. Similarly, the camera control unit 105' controls the timing generator 102', the image sensor 103', the analog signal processing unit 104', the digital gain unit 106', etc.

[0014] The image display unit 107 is provided to present (display) an image to the left eye of a user wearing the HMD 100 on their head. The image display unit 107' is provided to present an image to the right eye of a user wearing the HMD 100 on their head. The image display units 107, 107' are display elements capable of two-eye VR display, etc. The display elements can be, but are not limited to, an EL (Electro Luminescence) panel, an LCD, etc. Two-eye VR display is performed, for example, when a VR180 video is played and displayed on the HMD 100. By performing "two-eye VR display" using a VR image for the left eye and a VR image for the right eye that have parallax from each other, it is possible to view those VR images in stereoscopic view.

[0015] The image processing unit 110 performs various image processing. For example, the image processing unit 110 performs image processing to cancel aberrations caused by the optical systems of the image sensors 103 and 103' and the optical systems of the image display units 107 and 107', and image processing to synthesize any CG with the digital image output from the digital gain units 106 and 106'. Furthermore, the image processing unit 110 can acquire distance information of the environment where the user is located by measuring distance using a stereo camera (or the image sensors 103 and 103'), not shown, and can acquire user posture information from a posture sensor, not shown. Therefore, the image processing unit 110 can select the CG to be synthesized using the distance information and posture information. By changing the position, orientation, and size, it is possible to generate images that make CG objects that are not actually present in the scene appear as if they were actually there.

[0016] The image processing unit 110 can store (record) image data of a composite image (an image obtained by combining a real space image with CG) in the memory 113 via the memory control unit 112. The image processing unit 110 can also display the composite image on the image display units 107 and 107'. The composite image obtained by combining the digital image output from the digital gain unit 106 with CG is displayed on the image display unit 107, and the composite image obtained by combining the digital image output from the digital gain unit 106' with CG is displayed on the image display unit 107'.

[0017] Information such as CG images is stored in a content database (DB) 111. The image processing unit 110 can switch the CG data read from the content DB 111.

[0018] The image processing unit 110 also functions as a control unit that controls the entire HMD 100. The image processing unit 110 sets one of a plurality of operation modes as the operation mode of the HMD 100. The operation mode may be set automatically or in response to a user operation. In the first embodiment, the user can change settings such as the operation mode of the HMD 100 by using an operation unit (not shown). The plurality of operation modes include, for example, a shooting mode and a playback mode. The shooting mode includes a video shooting mode for shooting video and a still image shooting mode for shooting still images. In the playback mode, a two-eye VR display or the like is performed.

[0019] FIG. 2 is a timing chart showing a series of processes until a composite image is displayed in the playback mode.

[0020] First, the image sensor 103, the analog signal processor 104, and the digital gain unit 106 perform a process 201 to acquire RAW data of image B, which is a background image (a captured image of real space). Similarly, the image sensor 103', the analog signal processor 104', and the digital gain unit 106' perform a process 202 to acquire RAW data of image A, which is a background image.

[0021] Next, the image processing unit 110 performs image processing 203 on the two pieces of RAW data. The image processing unit 203 includes development processing, correction processing such as distortion correction processing, and CG compositing processing. As a result, a composite image A is obtained by compositing image A with CG, and a composite image B is obtained by compositing image B with CG.

[0022] Finally, the image display unit 107 updates (204) the display image to composite image B output from the image processing unit 110. Similarly, the image display unit 107′ updates (205) the display image to composite image A output from the image processing unit 110.

[0023] These processes are performed periodically in response to timing signals periodically output from the timing generators 102 and 102'.

[0024] 3 is a timing chart showing a series of processes until a composite image is displayed in still image shooting mode. As shown in FIG. 3, in the first embodiment, in still image shooting mode, the image sensor 103, analog signal processor 104, digital gain unit 106, and image display unit 107 are stopped. Then, the image capturing frame rate of the image sensor 103' is changed to a frame rate higher than that in playback mode, and the background image obtained by the image sensor 103' (a composite image based on the background image obtained by the image sensor 103') is displayed and recorded.

[0025] In addition, in still image shooting mode, the image sensor 103', analog signal processing unit 104', digital gain unit 106', and image display unit 107' may be stopped. Then, the image capturing frame rate of the image sensor 103 may be changed to a frame rate higher than that in playback mode, and the background image obtained by the image sensor 103 (a composite image based on the background image obtained by the image sensor 103) may be displayed and recorded. In addition, in moving image shooting mode, similar processing may be performed to record a moving image.

[0026] FIG. 4 is a flowchart illustrating a series of processes from obtaining a background image to displaying a composite image.

[0027] In S401, the image processing unit 110 (control unit) determines whether or not the current operation mode of the HMD 100 is the still image capturing mode. The image processing unit 110 waits until the still image capturing mode is set, and when the still image capturing mode is set, the process proceeds to S402.

[0028] In S402, the camera control unit 105 stops the imaging process of the imaging element 103 and the like.

[0029] In S403, the camera control unit 105' changes the imaging frame rate of the image sensor 103' to a frame rate that is twice as high as that in playback mode. Note that the imaging frame rate of the image sensor 103' after the change only needs to be higher than that in playback mode, and is not limited to twice the frame rate in playback mode.

[0030] In S404, the camera control unit 105' sets a gain corresponding to the image capturing frame rate set in S403 in the digital gain unit 106', thereby obtaining a properly exposed RAW image (background image).

[0031] In S405, the image processing unit 110 performs image processing such as development processing and distortion correction processing on the RAW image (background image) obtained in S404.

[0032] In S406, the image processing unit 110 performs CG synthesis processing on the background image after the image processing in S405.

[0033] In S407, the image processing unit 110 displays the composite image generated in S406 on the image display unit 107'. The composite image generated in S406 may be displayed on the image display unit 107, or may be displayed on both the image display unit 107 and the image display unit 107'. Furthermore, the image processing unit 110 records the composite image generated in S406 as a still image in the memory 113 in response to a shooting instruction from the user. The composite image may be recorded in an internal storage medium such as the memory 113, or may be recorded in an external storage medium.

[0034] As described above, according to the first embodiment, by stopping one of the two image sensors, the image quality of the image captured by the other image sensor can be significantly improved. This makes it possible to display or record images with sufficiently high image quality when capturing (recording) the image. In the first embodiment, images can be displayed at a very high frame rate. When capturing video, video can be recorded at a very high frame rate.

[0035] <Embodiment 2> A second embodiment of the present invention will now be described. In the first embodiment, image quality is improved by increasing the image capture frame rate. In the second embodiment, image quality is improved by HDR compositing, which combines two images captured sequentially with different exposures. Fig. 5 is a flowchart illustrating a series of processes from obtaining a background image to displaying a composite image in the second embodiment.

[0036] In steps S401 to S405, the same processing as that performed in the first embodiment (FIG. 4) is performed.

[0037] In S501, the camera control unit 105 sets a gain lower than that in S404 in the digital gain unit 106'. As a result, a RAW image (background image) with exposure darker than the appropriate exposure (underexposure) is obtained.

[0038] In S502, the image processing unit 110 performs image processing such as development processing and distortion correction processing on the RAW image (background image) obtained in S501.

[0039] In S503, the image processing unit 110 performs HDR composition, which combines the properly exposed RAW image (background image) obtained in S404 and the underexposed RAW image (background image) obtained in S501.

[0040] In S406 and S407, the same processing as that performed in the first embodiment (FIG. 4) is performed. However, in S406, CG composition processing is performed on the background image obtained by HDR composition in S503.

[0041] As described above, according to the second embodiment, the image capturing frame rate is doubled and HDR compositing is performed between a properly exposed image and an underexposed image, thereby enabling images with a wide dynamic range to be displayed while maintaining the display frame rate.

[0042] <Embodiment 3> A third embodiment of the present invention will be described. In the first embodiment, the still image capturing mode is set in response to a user operation. In the third embodiment, the still image capturing mode is set by detecting the user's blinking (the same operation as the operation in the still image capturing mode in the first embodiment is performed). Fig. 6 is a flowchart illustrating a series of processes from detecting the user's blinking to displaying a composite image in the third embodiment.

[0043] In S601, the image processing unit 110 (control unit) determines whether or not the user of the HMD 100 has one eye closed based on information from a pupil state detection camera (not shown). The image processing unit 110 waits for the user to close one eye, and when the user closes one eye, the process proceeds to S602.

[0044] In S602, the camera control unit corresponding to the closed eye, either of the camera control units 105 and 105', stops the imaging process of the imaging element corresponding to the closed eye, either of the imaging elements 103 and 103'.

[0045] In S603, the camera control unit corresponding to the open eye, either of the camera control units 105 and 105', changes the frame rate of the image sensor corresponding to the open eye, either of the image sensor 103 and the image sensor 103', to twice the frame rate in playback mode.

[0046] In S604, the camera control unit corresponding to the open eye sets a gain corresponding to the imaging frame rate set in S603 to the digital gain unit 106′, thereby obtaining a properly exposed RAW image (background image).

[0047] In S405 and S406, the same processing as that performed in the first embodiment (FIG. 4) is performed.

[0048] In S605, the image processing unit 110 displays the composite image generated in S406 on the image display unit corresponding to the open eye, either the image display unit 107 or the image display unit 107'.

[0049] As described above, according to the third embodiment, the operation in still image capture mode is automatically performed in response to the user's blinking, thereby saving the user the trouble of switching the operation mode using a setting screen or the like.

[0050] In addition, in the first to third embodiments, imaging by one of the two imaging elements is stopped and the imaging frame rate of the other imaging element is increased, but the method of improving the imaging quality is not limited to this. For example, imaging by one of the two imaging elements may be stopped and the imaging resolution of the other imaging element may be increased. There may be a mode for increasing the frame rate and a mode for increasing the resolution.

[0051] The camera control units 105, 105' and the image processing unit 110 may be built into the HMD 100, or may be provided in an external device of the HMD 100 (for example, a personal computer or controller connected to the HMD 100 by wire or wirelessly).

[0052] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0053] The above processor is a processor in the broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0054] Although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely represents one embodiment of the present invention, and each embodiment can be combined as appropriate.

[0055] <Other embodiments> The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more functions.

[0056] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a first imaging means provided to correspond to the right eye of the user; a second imaging means provided to correspond to the left eye of the user; a first display means for presenting an image to the right eye of the user; a second display means for presenting an image to the left eye of the user; A control device for controlling a head-mounted display device having a setting means for setting one of a plurality of modes including a first mode and a second mode; In the first mode, control is performed so that an image captured by the first imaging means with a first image quality is displayed on the first display means, and an image captured by the second imaging means with the first image quality is displayed on the second display means; In the second mode, one of the first imaging means and the second imaging means is stopped, and an image captured by the other imaging means with a second image quality higher than the first image quality is displayed on the first display means or the second display means. Control means and A control device comprising: (Configuration 2) the image captured with the first image quality is an image captured at a first frame rate, The image captured at the second image quality is an image captured at a second frame rate that is higher than the first frame rate. 2. The control device according to configuration 1, (Configuration 3) The second frame rate is twice the frame rate of the first frame rate. 3. The control device according to configuration 2. (Configuration 4) In the second mode, the control means performs HDR compositing, which combines two images with different exposures captured sequentially by the other of the imaging means, and controls the first display means or the second display means to display the image obtained by the HDR compositing. 4. The control device according to configuration 3, (Configuration 5) the image captured at the first image quality is an image captured at a first resolution, The image captured at the second image quality is an image captured at a second resolution higher than the first resolution. 5. The control device according to any one of configurations 1 to 4. (Configuration 6) the plurality of modes includes a third mode; the image captured at the first image quality is an image captured at a first resolution, In the third mode, one of the first imaging means and the second imaging means is stopped, and an image captured by the other imaging means at a second resolution higher than the first resolution is controlled to be displayed on the first display means or the second display means. 5. The control device according to any one of configurations 2 to 4. (Configuration 7) The second mode is a still image shooting mode. 7. The control device according to any one of configurations 1 to 6. (Configuration 8) a detection means for detecting that the user has closed one eye, When the detecting means detects that the user has closed one eye, the setting means sets the second mode. 8. The control device according to any one of configurations 1 to 7. (method) a first imaging means provided to correspond to the right eye of the user; a second imaging means provided to correspond to the left eye of the user; a first display means for presenting an image to the right eye of the user; a second display means for presenting an image to the left eye of the user; A method for controlling a head-mounted display device having a setting step of setting one of a plurality of modes including a first mode and a second mode; a first control step of controlling, in the first mode, to display an image captured by the first imaging means with a first image quality on a first display means and to display an image captured by the second imaging means with the first image quality on a second display means; a second control step of controlling, in the second mode, to stop one of the first imaging means and the second imaging means and to display an image captured by the other imaging means at a second image quality higher than the first image quality on the first display means or the second display means; A control method comprising: (program) A program for causing a computer to function as each means of the control device according to any one of configurations 1 to 8. [Explanation of symbols]

[0057] 100: HMD 103, 103': Image sensor 105, 105': Camera control unit 107, 107': Image display unit 110: Image processing unit

Claims

1. a first imaging means provided to correspond to the right eye of the user; a second imaging means provided to correspond to the left eye of the user; a first display means for presenting an image to the right eye of the user; a second display means for presenting an image to the left eye of the user; A control device for controlling a head-mounted display device having a setting means for setting one of a plurality of modes including a first mode and a second mode; In the first mode, the image captured by the first imaging means with a first image quality is displayed on the first display means, and the image captured by the second imaging means with the first image quality is displayed on the second display means; In the second mode, one of the first imaging means and the second imaging means is stopped, and an image captured by the other imaging means with a second image quality higher than the first image quality is displayed on the first display means or the second display means. Control means and A control device comprising:

2. the image captured with the first image quality is an image captured at a first frame rate, The image captured at the second image quality is an image captured at a second frame rate higher than the first frame rate.

2. The control device according to claim 1.

3. The second frame rate is twice the frame rate of the first frame rate.

3. The control device according to claim 2.

4. In the second mode, the control means performs HDR compositing, which combines two images with different exposures captured sequentially by the other of the imaging means, and controls the first display means or the second display means to display the image obtained by the HDR compositing.

4. The control device according to claim 3.

5. the image captured at the first image quality is an image captured at a first resolution, The image captured at the second image quality is an image captured at a second resolution higher than the first resolution.

2. The control device according to claim 1.

6. the plurality of modes includes a third mode; the image captured at the first image quality is an image captured at a first resolution, In the third mode, one of the first imaging means and the second imaging means is stopped, and an image captured by the other imaging means at a second resolution higher than the first resolution is displayed on the first display means or the second display means.

3. The control device according to claim 2.

7. The second mode is a still image shooting mode.

2. The control device according to claim 1.

8. a detection means for detecting that the user has closed one eye, When the detecting means detects that the user has closed one eye, the setting means sets the second mode.

2. The control device according to claim 1.

9. a first imaging means provided to correspond to the right eye of the user; a second imaging means provided to correspond to the left eye of the user; a first display means for presenting an image to the right eye of the user; a second display means for presenting an image to the left eye of the user; A method for controlling a head-mounted display device having a setting step of setting one of a plurality of modes including a first mode and a second mode; a first control step of controlling, in the first mode, to display an image captured by the first imaging means with a first image quality on the first display means and to display an image captured by the second imaging means with the first image quality on the second display means; a second control step of controlling, in the second mode, to stop one of the first imaging means and the second imaging means and to display an image captured by the other imaging means at a second image quality higher than the first image quality on the first display means or the second display means; A control method comprising:

10. A program for causing a computer to function as each of the means of the control device according to any one of claims 1 to 8.

Images