Image generation device and head-mounted display

Abstract

This image generation device comprises: a camera (13) capable of outputting a pixel signal of an imaging pixel region for each line; a signal acquisition unit (301) that acquires a captured video signal based on the pixel signal and stores the captured video signal in a buffer (302); a light source that emits light on the basis of the captured video signal stored in the buffer (302); a scanning unit that generates a frame image by scanning with the light from the light source; and a detection unit for detecting a line of sight. The signal acquisition unit (301) acquires the captured video signal with a first definition and stores the captured video signal in the buffer (302) with regard to a first pixel region including a position corresponding to the line of sight in the imaging pixel region, and with regard to a second pixel region other than the first pixel region, acquires the captured video signal with a second definition lower than the first definition and stores the captured video signal in the buffer (302).

Description

Image generating device and head-mounted display

[0001] The present invention relates to an image generating device that generates an image by scanning light and a head-mounted display.

[0002] Conventionally, as an image generating device that generates an image by scanning light, for example, a head-mounted display such as goggles or glasses that realize AR (Augmented Reality) or VR (Virtual Reality) has been known. In these devices, for example, light based on a video signal is irradiated onto a translucent display, and the reflected light is irradiated onto a user's eyes. Alternatively, light based on the video signal is irradiated directly onto a user's eyes.

[0003] Japanese Patent Application Laid-Open No. 2006-124494 (Patent Document 1) describes an apparatus that realizes a first line density in a first portion of an image and a second line density lower than the first line density in a second portion of the image by controlling the rotation of the fast axis and the slow axis of a MEMS mirror, and determines the position of the first portion of the image based on the line of sight of the eye. As a result, the resolution of the image in the second portion that does not correspond to the line of sight is lower than the resolution of the image in the first portion that corresponds to the line of sight, thereby reducing eye fatigue for the user.

[0004] U.S. Patent No. 9,986,215

[0005] In the head-mounted display described above, a video signal obtained by capturing an image in front of the user can be used as a video signal for modulating light for generating an image. This allows the user to grasp the scenery in front of them from the captured image even if the goggles or glasses described above are not particularly see-through.

[0006] In this case, it is preferable to make the image definition of the portion corresponding to the user's line of sight different from the image definition of the other portion so that the user can view the image more comfortably.

[0007] However, because the user's line of sight can change dynamically, if a video signal with resolution adjusted according to the line of sight is reconstructed from one frame of captured video signal output from a camera, it takes time to reconstruct the video signal, resulting in a delay in the display of the frame image. Such a display delay results in distortion of the frame image, causing a sense of discomfort to the user.

[0008] In view of such problems, the present invention aims to provide an image generating device and a head-mounted display that can smoothly switch the image resolution between a first image area near the user's line of sight and a second image area elsewhere.

[0009] A first aspect of the present invention relates to an image generating device. The image generating device according to this aspect includes a camera capable of outputting pixel signals of each pixel in an imaging pixel area on a line-by-line basis, a signal acquisition unit that acquires an imaging video signal based on the pixel signals and stores the acquired signal in a buffer, a light source that emits light for generating a frame image based on the imaging video signal stored in the buffer, a scanning unit that scans the light emitted from the light source to generate the frame image, and a detection unit that detects a user's line of sight. The signal acquisition unit acquires the imaging video signal at a first resolution for a first pixel area in the imaging pixel area that includes a position corresponding to the line of sight and stores the acquired signal in the buffer, and acquires the imaging video signal at a second resolution lower than the first resolution for a second pixel area other than the first pixel area and stores the acquired signal in the buffer.

[0010] According to the image generating device of this aspect, the resolution of the captured video signal stored in the buffer from the captured pixel area is switched in accordance with the user's line of sight, which allows for smooth switching of the image resolution between the first image area near the user's line of sight and the other second image area in the displayed frame image.

[0011] A second aspect of the present invention relates to a head-mounted display, which includes the image generation device according to the first aspect and a frame that holds the image generation device.

[0012] The head-mounted display according to this aspect has the same effects as those of the first aspect. In addition, by wearing the head-mounted display on the head, the user can grasp the scenery captured by the camera, etc., from the frame image generated by the image generating device.

[0013] As described above, the present invention can provide an image generating device and a head-mounted display that can smoothly switch the image resolution between a first image area near the user's line of sight and a second image area elsewhere.

[0014] The effects and significance of the present invention will become more apparent from the following description of the embodiments, however, the embodiments shown below are merely examples of how the present invention can be implemented, and the present invention is not limited to the embodiments described below.

[0015] FIG. 1 is a perspective view schematically illustrating the configuration of AR glasses according to an embodiment. FIG. 2 is a diagram schematically illustrating the configuration of a projection unit according to an embodiment. FIG. 3 is a block diagram illustrating the configurations of a projection unit and a detection unit according to an embodiment. FIG. 4 is a block diagram illustrating the configuration of a signal processing unit according to an embodiment. FIG. 5 is a flowchart illustrating a storage process of an imaged video signal performed by a signal acquisition unit according to an embodiment. FIG. 6 is a diagram schematically illustrating an operation of writing a video signal to a buffer according to an embodiment. FIG. 7 is a diagram schematically illustrating a process of combining an imaged video signal and an input video signal in a signal synthesis unit according to an embodiment. FIG. 8 is a diagram schematically illustrating a light scanning method according to an embodiment. FIG. 9 is a diagram schematically illustrating a method of setting a first pixel region based on a viewpoint according to a first modification. FIG. 10 is a flowchart illustrating a storage process of an imaged video signal performed by a signal acquisition unit according to a second modification. FIG. 11 is a diagram schematically illustrating an operation of writing a video signal to a buffer according to the second modification. FIG. 12 is a diagram schematically illustrating a process of combining an imaged video signal and an input video signal in a signal synthesis unit according to the second modification. Fig. 13 is a flowchart showing a storage process of an imaged video signal executed by a signal acquisition unit according to Modification Example 3. Fig. 14 is a diagram schematically showing a write operation of a video signal to a buffer according to Modification Example 3. Fig. 15 is a diagram schematically showing a synthesis process of an imaged video signal and an input video signal in a signal synthesis unit according to Modification Example 3. Fig. 16 is a diagram schematically showing a write operation of a video signal to a buffer according to another modification.

[0016] However, the drawings are for illustrative purposes only and do not limit the scope of the present invention.

[0017] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, an example in which the present invention is applied to an image generation device for a head-mounted display is shown. Examples of head-mounted displays include AR glasses, AR goggles, VR glasses, and VR goggles. The head-mounted display in the following embodiments is AR glasses. However, the following embodiments are one embodiment of the present invention, and the present invention is not limited to the following embodiments. For example, the present invention is not limited to image generation devices for head-mounted displays, but can also be applied to image generation devices such as in-vehicle head-up displays.

[0018] FIG. 1 is a perspective view schematically illustrating the configuration of AR glasses 1. As shown in FIG.

[0019] 1 also shows mutually orthogonal X, Y, and Z axes in addition to the front, back, left, right, up, and down directions of the AR glasses 1. The positive X-axis direction, positive Y-axis direction, and positive Z-axis direction correspond to the rightward, backward, and upward directions of the AR glasses 1, respectively.

[0020] The AR glasses 1 include a frame 2, a pair of image generating devices 3, and a pair of mirrors 4. The AR glasses 1 are worn on the user's head, similar to ordinary eyeglasses.

[0021] The frame 2 holds a pair of image generating devices 3 and a pair of mirrors 4. The frame 2 is composed of a front portion 2a and a pair of support portions 2b. The pair of support portions 2b extend rearward from the right and left ends of the front portion 2a. When the frame 2 is worn by a user, the front portion 2a is positioned in front of a pair of eyes E of the user. The frame 2 is composed of an opaque material. The frame 2 may also be composed of a transparent material.

[0022] The pair of image generation devices 3 are arranged symmetrically with respect to the YZ plane passing through the center of the AR glasses 1. The image generation devices 3 generate images in the eyes E of a user wearing the AR glasses 1 on their head.

[0023] The mirror 4 is a mirror with a concave reflective surface, and is installed on the inner surface of the front surface 2a of the frame 2. The mirror 4 almost totally reflects the light projected from the corresponding projection unit 11 and guides it to the user's eye E. In other words, the mirror 4 constitutes an optical system for guiding the light from the image generation device 3 to the eye E of the user wearing the AR glasses 1 (head-mounted display) on the head. If the light from the image generation device 3 is directly irradiated onto the user's eye E, the mirror 4 may be omitted.

[0024] The image generating device 3 includes a projection unit 11 , a detection unit 12 , and a camera 13 .

[0025] The projection unit 11 is installed on the inner surface of the support unit 2b. The projection unit 11 projects light modulated by a video signal onto the corresponding mirror 4. The light from the projection unit 11 reflected by the mirror 4 is irradiated onto the fovea centralis, which is located at the center of the retina in the eye E. This allows the user to visually grasp the frame image 20 (see FIG. 2) generated by the image generation device 3.

[0026] The pair of detectors 12 are installed on the inner surface of the front surface 2a between the pair of mirrors 4. The detectors 12 are used to detect the line of sight of the user.

[0027] The pair of cameras 13 are installed on the outer surface of the front surface 2a in front of the pair of mirrors 4. The cameras 13 capture an image within the field of view of the cameras 13. In this embodiment, the field of view of the cameras 13 is in front of the AR glasses 1.

[0028] FIG. 2 is a diagram schematically illustrating the configuration of the projection unit 11. As shown in FIG.

[0029] The projection unit 11 includes light sources 101 , 102 , and 103 , collimator lenses 111 , 112 , and 113 , apertures 121 , 122 , and 123 , a mirror 131 , dichroic mirrors 132 and 133 , a first scanning unit 140 , a relay optical system 150 , and a second scanning unit 160 .

[0030] Light sources 101, 102, and 103 are, for example, semiconductor laser light sources. Light source 101 emits laser light with a red wavelength in the range of 635 nm to 645 nm, light source 102 emits laser light with a green wavelength in the range of 510 nm to 530 nm, and light source 103 emits laser light with a blue wavelength in the range of 440 nm to 460 nm.

[0031] In this embodiment, a color image is generated as the frame image 20 described below, and therefore the projection unit 11 is equipped with light sources 101, 102, and 103 capable of emitting red, green, and blue laser light. When a monochromatic image is displayed as the frame image 20, the projection unit 11 may be equipped with only one light source corresponding to the color of the image. Alternatively, the projection unit 11 may be configured to include two light sources with different emission wavelengths. These light sources may be LEDs.

[0032] The light beams emitted from the light sources 101, 102, and 103 are converted into parallel light beams by collimator lenses 111, 112, and 113, respectively. The light beams transmitted through the collimator lenses 111, 112, and 113 are shaped into approximately circular beams by apertures 121, 122, and 123, respectively.

[0033] Mirror 131 almost totally reflects the red light that has passed through aperture 121. Dichroic mirror 132 reflects the green light that has passed through aperture 122 and transmits the red light reflected by mirror 131. Dichroic mirror 133 reflects the blue light that has passed through aperture 123 and transmits the red light and green light that have passed through dichroic mirror 132. Mirror 131 and the two dichroic mirrors 132 and 133 are arranged to align the optical axes of the light of each color emitted from light sources 101, 102, and 103.

[0034] The first scanning unit 140 reflects the light that has passed through the dichroic mirror 133. The first scanning unit 140 is, for example, a MEMS (Micro Electro Mechanical System) mirror. The first scanning unit 140 is configured to rotate a first mirror 141, onto which the light that has passed through the dichroic mirror 133 is incident, around an axis 141a parallel to the Z-axis direction in response to a drive signal. The rotation of the first mirror 141 changes the reflection direction of the light. As a result, the light reflected by the first mirror 141 is scanned along a scanning line extending in the X-axis direction (horizontal direction) on the retina of the eye E, as described below.

[0035] The relay optical system 150 directs light reflected by the first scanning unit 140 toward the center of the second mirror 161 of the second scanning unit 160. That is, light incident on the first scanning unit 140 is deflected by the first mirror 141 at a predetermined deflection angle. The relay optical system 150 directs light at each deflection angle toward the center of the second mirror 161. The relay optical system 150 also has multiple mirrors, and reflects the light reflected by the first scanning unit 140 by the multiple mirrors to direct the light toward the second scanning unit 160. This makes it possible to achieve a long optical path length inside the relay optical system 150 and reduce the deflection angle of the light when viewed from the second mirror 161.

[0036] The second scanning unit 160 reflects the light that has passed through the relay optical system 150. The second scanning unit 160 is, for example, a MEMS mirror. The second scanning unit 160 has a configuration that rotates a second mirror 161, onto which the light that has passed through the relay optical system 150 is incident, around an axis 161a parallel to the XY plane in response to a drive signal. The rotation of the second mirror 161 changes the reflection direction of the light. As a result, the scanning line on the retina of the eye E along which the light is scanned by the first scanning unit 140 is changed to the Z-axis direction (vertical direction) as described below.

[0037] The light reflected by the second scanning unit 160, that is, the light emitted from the projection unit 11, is reflected by the mirror 4 and forms a frame image 20 on the retina of the eye E.

[0038] FIG. 3 is a block diagram showing the configuration of the projection unit 11 and the detection unit 12. As shown in FIG.

[0039] The detection unit 12 includes a light source 12a and an image sensor 12b, and is connected to the control unit 201 of the projection unit 11. The light source 12a is, for example, an LED that emits light in the infrared wavelength range. The image sensor 12b is, for example, a CMOS image sensor or a CCD image sensor. The light source 12a irradiates light onto the user's eye E in response to an instruction from the control unit 201. The image sensor 12b captures an image of the user's eye E in response to an instruction from the control unit 201, and outputs the captured image to the control unit 201.

[0040] In response to instructions from the control unit 201, the camera 13 captures an image of its field of view, generates a video signal, and outputs the generated video signal to the signal processing unit 300 of the corresponding projection unit 11. In Fig. 1 , the left camera 13 outputs the generated video signal to the signal processing unit 300 of the left projection unit 11, and the right camera 13 outputs the generated video signal to the signal processing unit 300 of the right projection unit 11.

[0041] The camera 13 includes an image sensor and an optical system that forms an image of the field of view on the image sensor. The camera 13 is capable of outputting pixel signals for each pixel included in the image sensor's image pickup pixel area on a line-by-line basis. As will be described later, in this embodiment, the number of lines for writing captured video signals to the buffer 302 (see FIG. 4) in the signal processing unit 300 is limited. This will be described later with reference to FIGS. 5 and 6.

[0042] The projection unit 11 includes a control unit 201, a first mirror driving circuit 211, a second mirror driving circuit 212, a first mirror monitor sensor 213, a second mirror monitor sensor 214, a signal processing unit 300, a line memory 221, and a laser driving circuit 222.

[0043] The control unit 201 includes an arithmetic processing unit such as a CPU or FPGA, and a memory. The control unit 201 detects the user's line of sight based on the captured image from the detection unit 12, for example, by the dark pupil method, the bright pupil method, or the corneal reflex method. The control unit 201 acquires the viewpoint position in the frame image 20 formed on the user's retina based on the detected line of sight of the user. The control unit 201 also controls the signal processing unit 300 to process video signals from the camera 13 and external devices.

[0044] The first mirror drive circuit 211 drives the first mirror 141 of the first scanning unit 140 in response to a drive signal from the control unit 201. The second mirror drive circuit 212 drives the second mirror 161 of the second scanning unit 160 in response to a drive signal from the control unit 201.

[0045] The first mirror monitor sensor 213 is installed on the first mirror 141 and outputs a detection signal corresponding to the rotation of the first mirror 141 to the control unit 201. The second mirror monitor sensor 214 is installed on the second mirror 161 and outputs a detection signal corresponding to the rotation of the second mirror 161 to the control unit 201. Based on the detection signals from the first mirror monitor sensor 213 and the second mirror monitor sensor 214, the control unit 201 outputs drive signals to the first mirror drive circuit 211 and the second mirror drive circuit 212 so that the first mirror 141 and the second mirror 161 rotate with desired drive waveforms.

[0046] The signal processing unit 300 processes the video signals from the camera 13 and external devices, and outputs one line of video signals to the line memory 221. The configuration of the signal processing unit 300 will be described later with reference to FIG.

[0047] The line memory 221 outputs the video signal for one line output from the signal processing unit 300 to the laser driving circuit 222. The laser driving circuit 222 drives the light sources 101, 102, and 103 to emit light modulated by the video signal for one line output from the line memory 221.

[0048] FIG. 4 is a block diagram showing the configuration of the signal processing unit 300.

[0049] The signal processing unit 300 includes a signal acquisition unit 301 , a buffer 302 , an input processing unit 310 , a first buffer 321 , a second buffer 322 , and a signal synthesis unit 330 .

[0050] The signal acquisition unit 301 acquires pixel signals of each pixel from the camera 13, generates an imaged video signal based on the pixel signals, and stores the generated imaged video signal in the buffer 302. The buffer 302 is a memory that temporarily stores one frame's worth of imaged video signal.

[0051] The input processing unit 310 performs thinning processing on a digital video signal input from an external device. The video signal from the external device is, for example, a video signal related to CG (Computer Graphics). This video signal has the same resolution (number of lines) as the image sensor of the camera 13. The input processing unit 310 outputs one frame of video signal input from the external device directly to the first buffer 321. The input processing unit 310 also performs thinning processing on one frame of video signal input from the external device and outputs the resulting signal to the second buffer 322.

[0052] The first buffer 321 temporarily stores a video signal input from an external device, i.e., a first input video signal for one frame that has not been thinned out by the input processing unit 310. The second buffer 322 temporarily stores a second input video signal for one frame that has been thinned out by the input processing unit 310. The first input video signal stored in the first buffer 321 is an input video signal for high resolution, and the second input video signal stored in the second buffer 322 is an input video signal for low resolution.

[0053] The signal combining unit 330 selectively combines the first input video signal for high resolution stored in the first buffer 321 and the second input video signal for low resolution stored in the second buffer 322 with the captured video signal stored in the buffer 302 to generate a combined video signal for one frame. The signal combining method in the signal combining unit 330 will be described later with reference to Fig. 7. The signal combining unit 330 outputs the generated combined video signal to the line memory 221 line by line.

[0054] However, if the entire frame image 20 generated by the projection unit 11 in Fig. 2 has a high resolution, the user's eyes E will become tired easily. Therefore, in this embodiment, the resolution of the frame image 20 is set low in the area outside the range including the user's viewpoint. This reduces the user's eye fatigue.

[0055] However, because the user's line of sight can change dynamically, if an image signal whose resolution has been adjusted according to the line of sight is reconstructed from one frame of image signal output from the camera 13, it takes time to reconstruct the image, causing a delay in the display of the frame image. Such a delay in display results in distortion of the frame image, causing a sense of discomfort to the user.

[0056] Therefore, in this embodiment, a control is performed that enables smooth generation of an imaged video signal whose resolution is adjusted according to the line of sight. This control will be described below.

[0057] First, the imaging operation of the camera 13 and the transfer of signals to the signal acquisition unit 301 will be described.

[0058] The camera 13 has a plurality of pixels arranged vertically and horizontally in an imaging pixel area, which is the light receiving area of ​​the imaging element. The camera 13 performs imaging operations at a predetermined imaging cycle. The camera 13 resets its own imaging element at the start of the imaging cycle, thereby resetting the pixel signal (charge) of each pixel. The camera 13 then exposes each pixel for a predetermined exposure period. As a result, pixel signals (charges) corresponding to the amount of received light are accumulated in each of the plurality of pixels arranged in the imaging pixel area. Under the control of the signal acquisition unit 301, the camera 13 outputs pixel signals of each pixel for each horizontally arranged line of pixels.

[0059] The signal acquisition unit 301 resets the buffer 302 before the end of the exposure period in the camera 13. The buffer 302 has set therein storage addresses corresponding to the pixels of the image sensor of the camera 13. When the buffer 302 is reset, the data at all storage addresses becomes 0.

[0060] Thereafter, when the exposure period of camera 13 ends, signal acquisition unit 301 executes a process of acquiring an image capture video signal based on the pixel signal and storing (overwriting) it in buffer 302. At this time, signal acquisition unit 301 converts the pixel signal acquired from each pixel of camera 13 into digital data expressed in a predetermined gradation (for example, 256 gradations) that defines the luminance resolution, and acquires this digital data as an image capture video signal for each pixel.

[0061] Next, the storage process of the image signal in the buffer 302 will be described.

[0062] FIG. 5 is a flowchart showing the storage process of the captured video signal executed by the signal acquisition unit 301.

[0063] The signal acquisition unit 301 acquires the gaze detection result from the control unit 201 (S11). The signal acquisition unit 301 sets, in the imaging pixel area, a first pixel area including a position corresponding to the gaze and a second pixel area other than the first pixel area (S12). In this embodiment, the first pixel area and the second pixel area are set by dividing the imaging pixel area in a direction perpendicular to the line in which pixels are arranged horizontally in the imaging pixel area.

[0064] The signal acquisition unit 301 acquires captured image signals from pixel signals of all lines in the first pixel region and writes the acquired captured image signals to storage addresses in the buffer 302 corresponding to the pixels of each line (S13). The signal acquisition unit 301 thins out the lines in the second pixel region to set the lines to be read out (S14). The signal acquisition unit 301 acquires captured image signals from pixel signals of the lines set to be read out and writes the acquired captured image signals to storage addresses in the buffer 302 corresponding to the pixels of each line (S15). This causes the signal acquisition unit 301 to end the storage process of the captured image signals for the current imaging cycle.

[0065] FIG. 6 is a diagram showing a schematic diagram of the video signal writing operation according to the process of FIG.

[0066] The left side of Fig. 6 shows the accumulation state of pixel signals in the imaging pixel region R0 (at the end of the exposure period) and the setting states of the first pixel region R1 and the second pixel region R2, while the right side of Fig. 6 shows the writing state of the imaging video signals in the buffer 302.

[0067] In the diagram on the left side of Figure 6, P10 is a position (viewpoint) corresponding to the user's line of sight detected by the control unit 201. L1 is a line in which pixels are arranged horizontally, and pixel signals can be read out for each line L1. For convenience, 17 lines L1 are shown here, but the actual number of lines L1 included in the imaging pixel region R0 is several levels greater.

[0068] 5, the signal acquisition unit 301 sets a first pixel region R1 having a predetermined width in a direction perpendicular to the line L1 and centered on the viewpoint P10, and sets the remaining area as a second pixel region R2. The first pixel region R1 includes a predetermined number of lines L1 according to the width.

[0069] 5, the signal acquisition unit 301 reads out pixel signals of each line L1 included in the first pixel region R1, A / D converts these pixel signals into data of a predetermined gradation (for example, 256 gradations), and acquires an imaged video signal D1 for each pixel. Furthermore, the signal acquisition unit 301 writes the acquired imaged video signal D1 for each pixel to a corresponding storage address in the first region R11 of the buffer 302 corresponding to these lines L1.

[0070] In step S14 of Fig. 5, the signal acquisition unit 301 thins out the lines L1 in the second pixel region R2 to set the lines L1 to be read. Furthermore, in step S15 of Fig. 5, the signal acquisition unit 301 reads out pixel signals from each line L1 set to be read, A / D converts these pixel signals into data with the same gradation as the first pixel region R1, and acquires an imaged video signal D1 for each pixel. The signal acquisition unit 301 then writes the acquired imaged video signal D1 for each pixel to the corresponding storage address in the second region R12 of the buffer 302 that corresponds to these lines L1.

[0071] The first region R11 is a region on the buffer 302 that corresponds to the first pixel region R1 of the imaging pixel region R0, and the second region R12 is a region on the buffer 302 that corresponds to the second pixel region R2 of the imaging pixel region R0.

[0072] Through the above process, the captured video signal D1 based on the pixel signals of all lines L1 included in the first pixel region R1 is written to the first region R11 of the buffer 302. Furthermore, the captured video signal D1 is written to only the storage addresses corresponding to the lines L1 set as read targets in step S14 of Fig. 5 in the second region R12 of the buffer 302. Data D0 in the other storage addresses in the second region R12 of the buffer 302 remains the data at the time of resetting the buffer 302, i.e., data with a value of 0.

[0073] In step S14 of FIG. 5, the signal acquisition unit 301 preferably performs thinning in units of two adjacent lines L1.

[0074] That is, in the image sensor used in the camera 13, all colors are usually composed of two adjacent lines L1. Therefore, if lines are thinned out one by one, the colors included in the thinned lines will be lost, which may result in a decrease in color reproducibility of the frame image 20. In contrast, if thinning is performed in units of two adjacent lines L1 as described above, the loss of only one color is suppressed. Therefore, the color reproducibility of the frame image can be improved.

[0075] FIG. 7 is a diagram schematically showing the processing in the signal synthesis unit 330 in FIG.

[0076] The left side of Fig. 7 shows the storage state of the first input video signal D11 in the first buffer 321, and the storage state of the second input video signal D12 and the zero-value data D10 in the second buffer 322. The right side of Fig. 7 shows the storage state of the captured video signal D1 in the buffer 302. The total number of lines for data storage in the first buffer 321, the second buffer 322, and the buffer 302 are all the same.

[0077] The first buffer 321 stores one frame of the first input video signal D11. The first input video signal D11 is one frame of the input video signal that is stored in the first buffer 321 without being thinned out. The second buffer 322 stores one frame of the second input video signal D12 and zero-value data D10. The second input video signal D12 is the input video signal that remains after one frame of the input video signal has been thinned out using the same rule as for the second region R12 of the buffer 302. The data D10 is the zero-value data that was present at the time of reset, remaining as it was because the second input video signal D12 was thinned out and not written.

[0078] The signal combining unit 330 acquires the gaze detection result from the control unit 201 and identifies a first region R11 and a second region R12 on the buffer 302. The signal combining unit 330 combines, for each line, the captured video signal D1 stored in the first region R11 with the first input video signal D11 stored in the region of the first buffer 321 corresponding to this first region R11. The signal combining unit 330 also combines, for each line, the captured video signal D1 and data D0 stored in the second region R12 with the second input video signals D12 and D10 stored in the region of the second buffer 322 corresponding to this second region R12.

[0079] In this way, the signal combining unit 330 generates a combined video signal for one frame by combining the captured video signal and the input video signal. The signal combining unit 330 outputs the combined video signal for one line to the line memory 221 in FIG. 4, starting from the first line. As a result, the light sources 101 to 103 are driven in accordance with the combined video signal for each line. The light emitted from the light sources 101 to 103 is scanned along each scanning line by the first scanning unit 140 and the second scanning unit 160. As a result, a frame image is displayed.

[0080] FIG. 8 is a diagram schematically showing a light scanning method.

[0081] The first scanning unit 140 repeatedly rotates the first mirror 141 at a constant cycle. During the period in which the first mirror 141 rotates in the outward direction, light is emitted based on a composite video signal for one line, and light of each color is scanned along the scanning line. During the period in which the first mirror 141 rotates in the backward direction, the light sources 101 to 103 are in a non-driven state. The second scanning unit 160 drives the second mirror 161 so that the scanning lines are arranged at a constant pitch. The number of scanning lines is the same as the number of lines L1 included in the imaging pixel region R0 in FIG. 6.

[0082] The light emitted by the composite video signal for each line scans sequentially from the top scanning line to the bottom scanning line, thereby displaying a frame image 20. When scanning has reached the end of the bottom scanning line, the second mirror 161 is driven in reverse, and the scanning position returns to the beginning of the top scanning line. Scanning is then performed in the same manner to display the next frame image.

[0083] Through the above-described processing, in the first image region R21 corresponding to the first region R11 (see FIG. 7 ) of the buffer 302, light is emitted for all scanning lines based on the composite video signal obtained by combining the captured video signal D1 and the first input video signal D11. Therefore, an image is displayed in all scanning lines in the first image region R21. This increases the resolution (line density) of the image in the first image region R21.

[0084] In contrast, the second image region R22 corresponding to the second region R12 (see FIG. 7 ) of the buffer 302 includes scanning lines to which a composite video signal obtained by combining the captured video signal D1 and the second input video signal D12 is assigned, as well as scanning lines to which a zero-value composite video signal obtained by combining zero-value data D0 and zero-value data D10 is assigned, due to the above-described thinning process performed on the captured video signal and the input video signal. Therefore, in the second image region R22, no image is displayed on the scanning lines to which a zero-value composite video signal is assigned, i.e., the scanning lines indicated by the dashed arrows in FIG. 8 , but images are displayed on the remaining scanning lines. This results in a lower image resolution (line density) in the second image region R22.

[0085] The first image region R21 is an region that includes the user's viewpoint P20. Therefore, according to the above processing, the definition (resolution) of the image displayed in the first image region R21 that includes the viewpoint P20 can be maintained high, while the definition (resolution) of the image displayed in the remaining second image region R22 can be lowered. This allows the user to clearly view the image near their line of sight while reducing eye strain.

[0086] <Effects of the embodiment> According to the above embodiment, the following effects are achieved.

[0087] As shown in Figure 6, the signal acquisition unit 301 acquires the imaged video signal D1 at a first resolution (high resolution) for a first pixel region R1 of the imaged pixel region R0 that includes a position corresponding to the line of sight (viewpoint P10) and stores it in the buffer 302, and acquires the imaged video signal D1 at a second resolution (low resolution) that is lower than the first resolution (high resolution) for a second pixel region R2 of the imaged pixel region R0 other than the first pixel region R1 and stores it in the buffer 302.

[0088] According to this configuration, the definition (resolution) of the captured video signal D1 stored in the buffer 302 from the captured pixel region R0 is switched in accordance with the user's line of sight. Therefore, in the displayed frame image 20, the image definition (resolution) can be smoothly switched between the first image region R21 near the user's line of sight and the second image region R22.

[0089] As shown in Figure 6, the signal acquisition unit 301 divides the imaging pixel region R0 in a direction perpendicular to the line L1 to set a first pixel region R1 and a second pixel region R2, and sets a first resolution (high resolution) and a second resolution (low resolution) by making the frequency of lines L1 that do not store the imaging video signal D1 in the buffer 302 different between the first pixel region R1 and the second pixel region R2.

[0090] 8, this configuration makes it possible to make the resolution (definition) of the first image region R21 corresponding to the first pixel region R1 higher than the resolution (definition) of the second image region R22 corresponding to the second pixel region R2 in the frame image 20. This makes it possible to make the first image region R21 corresponding to the user's line of sight easier to see, while reducing eye fatigue on the user caused by the image in the second image region R22.

[0091] In this configuration, control may be further performed to reduce the difference in brightness between the first image region R21 and the second image region R22. For example, as shown in FIG. 8 , if the density of scanning lines for image display in the first image region R21 is twice the density of scanning lines for image display in the second image region R22, the control unit 201 may perform control to increase the output of the light sources 101 to 103 in the second image region R22 to approximately twice the output of the light sources 101 to 103 in the first image region R21. This reduces the difference in brightness between the first image region R21 and the second image region R22.

[0092] As shown in Figure 6, the signal acquisition unit 301 acquires the image signal D1 from all lines L1 for the first pixel region R1 and stores it in the buffer 302, and for the second pixel region R2, thins out the lines L1 at a predetermined interval and acquires the image signal D1 from the remaining lines L1 and stores it in the buffer 302.

[0093] According to this configuration, the resolution of the first image region R21 and the resolution of the second image region R22 can be made different through simple processing.

[0094] As described above, it is preferable that the signal acquiring section 301 performs thinning in units of two adjacent lines L1.

[0095] In the imaging element used in the camera 13, all colors are usually composed of two adjacent lines L1. Therefore, if one line is thinned out at a time, the color contained in the thinned line L1 will be lost, which may result in a decrease in color reproducibility of the frame image 20. In contrast, with the above configuration, thinning is performed in units of two adjacent lines L1, which prevents the loss of only one color. Therefore, the color reproducibility of the frame image 20 can be improved.

[0096] As shown in Figures 4 and 7, the image generating device 3 includes an input processing unit 310 that outputs a first input video signal D11 for constructing a frame image of a first definition (high resolution) and a second input video signal D12 for constructing a frame image of a second definition (low resolution) based on a video signal from an external device, and a signal combining unit 330 that combines the first input video signal D11 with the captured video signal D1 stored in the buffer 302 at the first definition (high resolution) and combines the second input video signal D12 with the captured video signal D1 stored in the buffer 302 at the second definition (low resolution).

[0097] With this configuration, an external image such as a CG image can be displayed superimposed on the image captured by the camera 13. Furthermore, a high-definition (high-resolution) external image is superimposed on the first image region R21 corresponding to the first pixel region R1, and a low-definition (low-resolution) external image is superimposed on the second image region R22 corresponding to the second pixel region R2. This makes it possible to make the first image region R21 corresponding to the user's line of sight easier to see, even when an external image is superimposed, while reducing eye fatigue caused by the image in the second image region R22.

[0098] As shown in Figure 1, the AR glasses 1 (head-mounted display) include an image generating device 3 and a frame 2 that holds the image generating device 3, and further include a mirror 4 (optical system) that guides the light scanned by the first scanning unit 140 and the second scanning unit 160 to the eyes E of the user who wears the AR glasses 1 (head-mounted display) on their head.

[0099] With the head-mounted display of this configuration, a user can wear the head-mounted display on their head and grasp the scenery captured by the camera 13 through the frame image 20 generated by the image generation device 3.

[0100] <Modification 1> In the above embodiment, the first pixel region R1 in the imaging pixel region R0 is set to a range of a predetermined width that includes the viewpoint P10 based on the user's line of sight, but this is not limited to this, and the first pixel region R1 may be set to one of a plurality of regions that are prepared in advance based on the viewpoint P10.

[0101] FIG. 9 is a diagram schematically showing a method for setting the first pixel region R1 based on a viewpoint P10 according to the first modification.

[0102] In the imaging pixel region R0, when the viewpoint P10 is included in one of the viewpoint regions R01 to R05, the signal acquisition unit 301 sets the region corresponding to the viewpoint region including the viewpoint P10 among the regions R1a to R1e corresponding to the viewpoint regions R01 to R05, respectively, as the first pixel region R1.

[0103] Viewpoint regions R01 to R05 are obtained by dividing the imaging pixel region R0 into five regions in a direction perpendicular to line L1. Regions R1a to R1e are regions set corresponding to viewpoint regions R01 to R05, and each has a width greater than that of viewpoint regions R01 to R05. When any of regions R1a to R1e is set as the first pixel region R1, the region of imaging pixel region R0 other than that region is set as the second pixel region R2.

[0104] <Effects of Modified Example 1> According to the configuration of Modified Example 1, the area including the viewpoint P10 among the pre-prepared areas R1a to R1e is set as the first pixel area R1, so that the first pixel area R1 including the viewpoint P10 can be smoothly set.

[0105] Modification Example 2 differs from the above embodiment in the method of differentiating the first definition when an imaged video signal is acquired from a first pixel region R1 including viewpoint P10 and stored in buffer 302 from the second definition when an imaged video signal is acquired from a second pixel region R2 other than first pixel region R1 and stored in buffer 302. Specifically, in Modification Example 2, the definition is adjusted by differentiating the gradation of the imaged video signal stored in buffer 302 between first pixel region R1 and second pixel region R2.

[0106] In addition, in Modification Example 2, the processing in the input processing unit 310 in Fig. 4 is modified from that in the above embodiment. That is, the processing for writing the input video signal to the first buffer 321 is the same as in the above embodiment, but the processing for writing the input video signal to the second buffer 322 is different from that in the above embodiment. The input processing unit 310 reduces the gradation of one frame of input video signal input from an external device without thinning it out, and writes the reduced gradation to the second buffer 322.

[0107] FIG. 10 is a flowchart showing the storage process of the captured video signal executed by the signal acquisition unit 301 according to the second modification.

[0108] The processing of steps S21 and S22 in Fig. 10 is the same as the processing of steps S11 and S12 in Fig. 5. The signal acquisition unit 301 acquires the line of sight detection result from the control unit 201 (S21), and sets, in the imaging pixel area R0, a first pixel area R1 including a position corresponding to the line of sight (viewpoint P10) and a second pixel area R2 other than that (S22).

[0109] The signal acquisition unit 301 converts the pixel signals of all lines in the first pixel region R1 into high-gradation data (captured video signals) and writes the converted data to storage addresses in the buffer 302 corresponding to the pixels of each line (S23). The signal acquisition unit 301 also converts the pixel signals of all lines in the second pixel region R2 into low-gradation data (captured video signals) and writes the converted data to storage addresses in the buffer 302 corresponding to the pixels of each line (S24). This causes the signal acquisition unit 301 to end the storage process of the captured video signals for the current imaging cycle.

[0110] FIG. 11 is a diagram schematically showing the video signal writing operation according to the process of FIG. 10 according to the second modification.

[0111] The left side of Fig. 11 shows the accumulation state of pixel signals in the imaging pixel region R0 (at the end of the exposure period) and the setting states of the first pixel region R1 and the second pixel region R2. The accumulation state of pixel signals in the imaging pixel region R0 is the same as the state shown on the left side of Fig. 6. The right side of Fig. 11 shows the state of the buffer 302 after the imaging video signal has been written.

[0112] 10, the signal acquisition unit 301 sets a first pixel region R1 having a predetermined width in a direction perpendicular to the line L1 and centered on the viewpoint P10, and sets the remaining area as a second pixel region R2. The first pixel region R1 includes a predetermined number of lines L1 according to the width.

[0113] 10, the signal acquisition unit 301 sequentially reads out pixel signals from all lines L1 included in the first pixel region R1, A / D converts these pixel signals into data expressed with a high luminance resolution (e.g., 256 gradations), and acquires an imaged video signal D1 for each pixel. Furthermore, the signal acquisition unit 301 writes the acquired imaged video signal D1 for each pixel to a corresponding storage address in the first region R11 of the buffer 302 corresponding to these lines L1.

[0114] 10, the signal acquisition unit 301 sequentially reads out pixel signals from all lines L1 included in the second pixel region R2, A / D converts these pixel signals into data expressed with a low luminance resolution (for example, two gradations), and acquires an imaged video signal D2 for each pixel. Furthermore, the signal acquisition unit 301 writes the acquired imaged video signal D2 for each pixel to a corresponding storage address in the second region R12 of the buffer 302 corresponding to these lines L1.

[0115] FIG. 12 is a diagram schematically showing the processing in the signal synthesis unit 330 of FIG.

[0116] The left side of Fig. 12 shows the storage state of the first input video signal D11 in the first buffer 321 and the storage state of the second input video signal D12 in the second buffer 322. The right side of Fig. 12 shows the storage state of the captured video signals D1 and D2 in the buffer 302. The total number of lines for data storage in the first buffer 321, the second buffer 322, and the buffer 302 are all the same.

[0117] One frame of the first input video signal D11 is stored in the first buffer 321. The storage state of the first input video signal D11 in the first buffer 321 is the same as in the case of FIG.

[0118] The second buffer 322 stores one frame of the second input video signal D12. However, as described above, the second input video signal D12 is the first input video signal D11 whose gradation has been reduced by the input processing unit 310 in Fig. 4. The first input video signal D11 is expressed in the same gradation as the imaged video signal D1 (for example, 256 gradations), and the second input video signal D12 is expressed in the same gradation as the imaged video signal D2 (for example, 2 gradations).

[0119] The signal combining unit 330 acquires the gaze detection result from the control unit 201 and identifies a first region R11 and a second region R12 on the buffer 302. The signal combining unit 330 combines, for each pixel, the captured video signal D1 stored in the first region R11 and the first input video signal D11 stored in the region of the first buffer 321 corresponding to the first region R11. The signal combining unit 330 also combines, for each pixel, the captured video signal D2 stored in the second region R12 and the second input video signal D12 stored in the region of the second buffer 322 corresponding to the second region R12.

[0120] In this way, the signal combining unit 330 generates a combined video signal for one frame by combining the captured video signal and the input video signal. The signal combining unit 330 outputs the combined video signal for one line to the line memory 221 in FIG. 4, starting from the first line. As a result, the light sources 101 to 103 are driven in accordance with the combined video signal for each line. The light emitted from the light sources 101 to 103 is scanned along the scanning lines by the first scanning unit 140 and the second scanning unit 160. As a result, the frame image 20 is displayed.

[0121] The scanning method of the light emitted from each of the light sources 101 to 103 is the same as the method in Fig. 8. In Modification Example 2, the composite video signal applied when the light scans the first image region R21 is a high-gradation signal, so an image with high gradation is displayed in the first image region R21. On the other hand, the composite video signal applied when the light scans the second image region R22 is a low-gradation signal, so an image with low gradation is displayed in the second image region R22.

[0122] Therefore, according to the process of the second modification, the resolution (gradation) of the image displayed in the first image region R21 including the viewpoint P20 can be maintained high, while the resolution (gradation) of the image displayed in the remaining second image region R22 can be lowered. This allows the user to clearly view the image near the line of sight and alleviate eye fatigue.

[0123] <Effects of Modified Example 2> As shown in FIG. 11 , the signal acquisition unit 301 acquires an imaged video signal D1 at a first resolution (high gradation) for a first pixel region R1 of the imaged pixel region R0 that includes a position corresponding to the line of sight (viewpoint P10) and stores the imaged video signal D1 in the buffer 302, and acquires an imaged video signal D2 at a second resolution (low gradation) that is lower than the first resolution for a second pixel region R2 of the imaged pixel region R0 other than the first pixel region R1 and stores the imaged video signal D2 in the buffer 302.

[0124] According to this configuration, the resolution (gradation) of the captured video signals D1 and D2 stored in the buffer 302 from the captured pixel region R0 is switched in accordance with the user's line of sight. Therefore, in the displayed frame image 20, the image resolution (gradation) can be smoothly switched between the first image region R21 near the user's line of sight and the second image region R22.

[0125] As shown in FIG. 11, the signal acquisition unit 301 sets the first definition and the second definition by varying the gradation that defines the luminance resolution of the captured video signal.

[0126] According to this configuration, the gradation (resolution) of the first image region R21 corresponding to the first pixel region R1 can be made higher than the gradation (resolution) of the second image region R22 corresponding to the second pixel region R2 in the frame image 20. This makes it easier to see the first image region R21 corresponding to the user's line of sight, while reducing eye fatigue caused by the image in the second image region R22.

[0127] Modification Example 3 Modification Example 3 differs from Modification Example 2 in the method of setting the first pixel region R1 and the second pixel region R2.

[0128] Fig. 13 is a flowchart showing the storage process of the captured video signal executed by the signal acquisition unit 301 according to the modified example 3. Fig. 14 is a diagram schematically showing the write operation of the video signal according to the modified example 3.

[0129] The processing of step S21 in Fig. 13 is the same as that of Fig. 10 . In step S22 in Fig. 13 , the signal acquisition unit 301 sets a first pixel region R1 with a predetermined width in directions perpendicular and parallel to the line L1, centered on the viewpoint P10, as shown in the left diagram of Fig. 14 , and sets the remaining region as a second pixel region R2. For example, the width of the first pixel region R1 in the direction parallel to the line L1 is limited compared to the case of Fig. 11 . The width of the first pixel region R1 in the direction perpendicular to the line L1 may also be different from that in Fig. 11 .

[0130] 13 , the signal acquisition unit 301 sequentially reads out pixel signals of all lines L1 that overlap the first pixel region R1, A / D converts the pixel signals of pixels included in the first pixel region R1 among these pixel signals into data expressed with a luminance resolution of high gradation (e.g., 256 gradations) to acquire an imaged video signal D1, and A / D converts the pixel signals of pixels included in the second pixel region R2 into data expressed with a luminance resolution of low gradation (e.g., 2 gradations) to acquire an imaged video signal D2. Furthermore, the signal acquisition unit 301 writes the acquired imaged video signals D1 and D2 for each pixel to the corresponding storage addresses in the buffer 302 that correspond to these pixels.

[0131] 13 , the signal acquisition unit 301 sequentially reads out pixel signals from all lines L1 included in the second pixel region R2 other than the above, A / D converts these pixel signals into data expressed with a luminance resolution of low gradation (for example, two gradations), and acquires an imaged video signal D2 for each pixel. Furthermore, the signal acquisition unit 301 writes the acquired imaged video signal D2 for each pixel to the corresponding storage address in the second region R12 of the buffer 302 corresponding to these lines L1.

[0132] As a result of the processing of steps S25 and S26, as shown in the right diagram of Figure 14, a high-gradation image video signal D1 is stored in a first region R11 of the buffer 302 corresponding to the first pixel region R1, and a low-gradation image video signal D2 is stored in a second region R12 of the buffer 302 corresponding to the other second pixel region R2.

[0133] FIG. 15 is a diagram schematically showing the processing in the signal synthesis unit 330 in FIG.

[0134] The storage states of the first input video signal D11 and the second input video signal D12 in the first buffer 321 and the second buffer 322 shown on the left side of Fig. 15 are the same as those in Fig. 12. The signal synthesis unit 330 acquires the gaze detection result from the control unit 201 and identifies the first region R11 and the second region R12 on the buffer 302. In Modification Example 3, due to the change in step S22 in Fig. 13, the first region R11 and the second region R12 are different from those in the right side of Fig. 12.

[0135] The signal synthesizing unit 330 synthesizes, for each pixel, the captured video signal D1 stored in the first region R11 and the first input video signal D11 stored in the region of the first buffer 321 corresponding to this first region R11. The signal synthesizing unit 330 also synthesizes, for each pixel, the captured video signal D2 stored in the second region R12 and the second input video signal D12 stored in the region of the second buffer 322 corresponding to this second region R12.

[0136] In this way, the signal combining unit 330 generates a combined video signal for one frame by combining the captured video signal and the input video signal. The signal combining unit 330 outputs the combined video signal for one line to the line memory 221 in FIG. 4, starting from the first line. As a result, the light sources 101 to 103 are driven in accordance with the combined video signal for each line. The light emitted from the light sources 101 to 103 is scanned along the scanning lines by the first scanning unit 140 and the second scanning unit 160. As a result, the frame image 20 is displayed.

[0137] The scanning method of the light emitted from each of the light sources 101 to 103 is the same as the method in Fig. 8. In Modification Example 3, the composite video signal applied when the light scans the first image region R21 is a high-gradation signal, so an image with high gradation is displayed in the first image region R21. On the other hand, the composite video signal applied when the light scans the second image region R22 is a low-gradation signal, so an image with low gradation is displayed in the second image region R22.

[0138] Therefore, according to the process of Modification Example 3, the resolution (gradation) of the image displayed in the first image region R21 including the viewpoint P20 can be maintained high, while the resolution (gradation) of the image displayed in the remaining second image region R22 can be lowered. Also, in Modification Example 3, the horizontal width of the first image region R21 set to high gradation is limited compared to Modification Example 2. Therefore, the user can clearly view the image near the line of sight, and further reduce eye fatigue.

[0139] <Other Modifications> The configurations of the image generation device 3 and the AR glasses 1 (head-mounted display) can be modified in various ways in addition to the configurations shown in the above-described embodiment and modifications.

[0140] For example, in the above embodiment and modified example, as shown in Fig. 4, the input video signal input from an external device is combined with the captured video signal of camera 13, but the input video signal may not be combined and a frame image 20 based only on the captured video signal may be displayed. In this case, input processing unit 310, first buffer 321, second buffer 322 and signal combining unit 330 are omitted from the processing in Fig. 4, and the captured video signal stored in buffer 302 is read out line by line and output to line memory 221.

[0141] In the above embodiment, the scanning speed in the direction perpendicular to the scanning lines may be changed. For example, the output frequency of the composite video signal corresponding to the second region R12 of the buffer 302 may be reduced by outputting only the lines in which the captured video signal and the second input video signal D12 are combined to the line memory 221, and the scanning speed in the direction perpendicular to the scanning lines may be increased in response to this reduction in output frequency compared to the period corresponding to the first region R11.

[0142] In the above embodiment, the signal acquisition unit 301 thins out the lines L1 by controlling the output of each line L1 in the imaging pixel region R0, but the method of thinning out the lines L1 is not limited to this. For example, the camera 13 may output pixel signals for all lines L1 in one frame to the signal acquisition unit 301, and the signal acquisition unit 301 may select pixel signals for lines to be stored in the buffer 302 from these pixel signals and store them in the buffer 302.

[0143] Furthermore, in the above embodiment and modified example, the definition (resolution, gradation) can be changed in two stages, but the definition (resolution, gradation) may be changed in three or more stages. In this case, it is sufficient if the user can select, from the three or more stages of definition, the definition (high resolution, high gradation) to be set for the first pixel region R1 and the first image region R21 and the definition (low resolution, low gradation) to be set for the second pixel region R2 and the second image region R22.

[0144] Furthermore, in the above-described embodiment and modified examples, the definition (resolution, gradation) of the area including the user's line of sight is increased compared to other areas. However, the definition of areas including the area including the user's line of sight, such as areas where movement occurs in the frame image, to which the user is likely to shift their line of sight next, may also be increased. A method for displaying a high-definition image in this area may be the same as that for the above-described embodiment and modified examples for displaying a high-definition image in an area including the line of sight. This allows the user to immediately see a high-definition image the next time they shift their line of sight to that area.

[0145] Furthermore, the resolution of the region including the user's line of sight may be changed so that it decreases as the distance to an object included in that region increases, as long as it is higher than that of other regions. In this case, the image generating device 3 may include a distance sensor such as a time-of-flight (TOF) camera whose detection range includes the field of view range of the camera 13. In this case, for example, different resolutions are set for each range of one or more thresholds for the distance to an object in the region including the user's line of sight. This allows the resolution of the region including the user's line of sight to be closer to the user's actual perception of distance, thereby providing the user with a more natural image.

[0146] Furthermore, the method of changing the definition between the area including the line of sight and other areas is not limited to the methods described in the above embodiment and modified examples, i.e., the method of changing the resolution or gradation. For example, the definition may be changed by changing both the resolution and gradation, or the definition may be changed by a method other than changing the resolution or gradation.

[0147] In the above embodiment and modified example, the viewpoint P10 is detected every frame. However, the timing of detecting the viewpoint P10 is not limited to this. For example, the viewpoint P10 may be detected every two or more predetermined frames. However, with this method, the detection of the viewpoint P10 may be delayed by a period of several frames, which is the gaze detection cycle, and the update of the area including the gaze (first pixel area R1, first image area R21) may be delayed by this period. For this reason, it is more difficult to quickly track the gaze of the area displaying a high-resolution image compared to when the viewpoint P10 is detected every frame. Therefore, in order to quickly track the gaze of the area displaying a high-resolution image, it is preferable to detect the viewpoint P10 every frame, as in the above embodiment and modified example.

[0148] In addition, in the above embodiment, the signal acquisition unit 301 reduces the resolution of the captured video signal to be stored in the second region R12 by performing a thinning process on the line L1 of the captured pixel region R0, as shown in Figure 6, but the method of storing the captured video signal in the buffer 302 is not limited to this.

[0149] For example, as shown in FIG. 16, if the number of lines L1 in the imaging pixel region R0 is half that in FIG. 6, the second region R12 of the buffer 302 may store the imaging video signal D1 and data D0 based on the pixel signals of each line L1 in the second pixel region R2 of the imaging pixel region R0, and the first region R11 of the buffer 302 may store the imaging video signal D1 based on the pixel signals of each line L1 in the first pixel region R1 of the imaging pixel region R0, and the imaging video signal D1′ generated by interpolation processing using the upper and lower imaging video signals D1.

[0150] Furthermore, in the above embodiment, it is preferable to perform the thinning process in units of two adjacent lines L1, but the thinning process is not limited to this. For example, the thinning process may be performed in units of adjacent even-numbered lines, and in this case, as in the above, the color reproducibility of the frame image 20 can be maintained at a good level. Furthermore, if the color reproducibility of the frame image 20 is not a consideration, the thinning process may be performed in units of one line, or the thinning process may be performed in units of adjacent odd-numbered lines.

[0151] In addition, in the above-mentioned modified examples 2 and 3, the input processing unit 310 reduces the gradation of the input video signal to two gradations and stores it in the second buffer 322, but this is not limited to this, and the input video signal may be reduced to a gradation other than two gradations (for example, 16 gradations) and stored in the second buffer 322.

[0152] In the above embodiment and modified example, the field of view of the camera 13 is in front of the AR glasses 1, but this is not limited thereto and may be above, below, or behind the AR glasses 1.

[0153] In the above embodiment and modified example, two sets of image generating devices 3 and mirrors 4 are provided in the AR glasses 1 to correspond to the user's pair of eyes E, but only one set may be provided in the AR glasses 1 to correspond to only one of the user's eyes E.

[0154] In the above embodiment and modified example, the light scanned by the first scanning unit 140 and the second scanning unit 160 is directed to the user's eye E via the mirror 4, but this is not limiting and the light may be directed to the user's eye E via an optical system other than a mirror (for example, a lens, etc.). In this case, the optical system may be, for example, a combination of multiple mirrors, a combination of a mirror and a lens, or a combination of multiple lenses.

[0155] In the above embodiment and modified examples, the first mirror 141 and the second mirror 161 are provided separately, but instead of the first mirror 141 and the second mirror 161, a single mirror that rotates about two axes may be provided.

[0156] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical idea defined in the claims.

[0157] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0158] (Technology 1) An image generating device comprising: a camera capable of outputting pixel signals of each pixel in an imaging pixel area line by line; a signal acquisition unit that acquires imaging video signals based on the pixel signals and stores them in a buffer; a light source that emits light to generate a frame image based on the imaging video signals stored in the buffer; a scanning unit that scans the light emitted from the light source to generate the frame image; and a detection unit that detects a user's line of sight, wherein the signal acquisition unit acquires the imaging video signals at a first resolution for a first pixel area in the imaging pixel area that includes a position corresponding to the line of sight and stores them in the buffer, and acquires the imaging video signals at a second resolution lower than the first resolution for a second pixel area in the imaging pixel area other than the first pixel area and stores them in the buffer.

[0159] This technology switches the resolution of the captured video signal stored in the buffer from the imaging pixel area according to the user's line of sight, allowing for smooth switching of image resolution between the image area near the user's line of sight and other image areas in the displayed frame image.

[0160] (Technology 2) In the image generating device described in Technology 1, the signal acquisition unit divides the imaging pixel area in a direction perpendicular to the line to set the first pixel area and the second pixel area, and sets the first resolution and the second resolution by making the frequency of the lines that do not store the imaging video signal in the buffer different between the first pixel area and the second pixel area.

[0161] According to this technology, the resolution (definition) of the first image region corresponding to the first pixel region in the frame image can be made higher than the resolution (definition) of the second image region corresponding to the second pixel region, thereby making it easier to see the first image region corresponding to the user's line of sight and reducing eye fatigue caused by the image in the second image region.

[0162] (Technology 3) In the image generating device described in Technology 2, the signal acquisition unit acquires the captured video signals from all the lines for the first pixel region and stores them in the buffer, and for the second pixel region, thins out the lines at predetermined intervals and acquires the captured video signals from the remaining lines and stores them in the buffer.

[0163] According to this technique, the resolution of the first image region and the resolution of the second image region can be made different through simple processing.

[0164] (Technology 4) In the image generating device according to Technology 3, the signal processing unit performs the thinning in units of two adjacent lines.

[0165] In an imaging element used in a camera, all colors are usually composed of two adjacent lines. Therefore, when lines are thinned out one by one, the color included in the thinned line is lost, which may result in a decrease in color reproducibility of the frame image. In contrast, according to the above-mentioned technology 4, thinning is performed in units of two adjacent lines, which prevents the loss of only one color. Therefore, the color reproducibility of the frame image can be improved.

[0166] (Technology 5) In the image generating device described in Technology 1, the signal acquisition unit sets the first resolution and the second resolution by varying a gradation that defines a luminance resolution of the captured video signal.

[0167] According to this technology, the gradation (resolution) of the first image region corresponding to the first pixel region in the frame image can be made higher than the gradation (resolution) of the second image region corresponding to the second pixel region, thereby making it easier to see the first image region corresponding to the user's line of sight and reducing eye fatigue caused by the image in the second image region.

[0168] (Technology 6) The image generating device according to any one of technologies 1 to 5, further comprising: an input processing unit that outputs a first input video signal for constructing a frame image of the first definition and a second input video signal for constructing a frame image of the second definition based on a video signal from an external device; and a signal combining unit that combines the first input video signal with the captured video signal stored in the buffer at the first definition, and combines the second input video signal with the captured video signal stored in the buffer at the second definition.

[0169] This technology allows an external image, such as a CG image, to be overlaid on an image captured by a camera. A high-resolution external image is overlaid on a first image area corresponding to a first pixel area, and a low-resolution external image is overlaid on a second image area corresponding to a second pixel area. This makes it possible to make the first image area corresponding to the user's line of sight easier to see, even when an external image is overlaid, while reducing eye strain caused by the image in the second image area.

[0170] (Technology 7) A head-mounted display comprising: the image generating device according to any one of technologies 1 to 6; and a frame that holds the image generating device.

[0171] This technique provides the same effects as techniques 1 to 6. Furthermore, by wearing the head-mounted display on the user's head, the user can grasp the scenery captured by the camera through the frame images generated by the image generating device.

[0172] REFERENCE SIGNS LIST 1 AR glasses (head mounted display) 2 Frame 3 Image generating device 4 Mirror (optical system) 12 Detection unit 13 Camera 20 Frame image 101, 102, 103 Light source 140 First scanning unit (scanning unit) 160 Second scanning unit (scanning unit) 301 Signal acquisition unit 302 Buffer 310 Input processing unit 330 Signal synthesis unit P10 Viewpoint R0 Imaging pixel area R1 First pixel area R2 Second pixel area R21 First image area R22 Second image area

Claims

1. An image generating device comprising: a camera capable of outputting pixel signals of each pixel in an imaging pixel area on a line-by-line basis; a signal acquisition unit that acquires an imaging video signal based on the pixel signals and stores it in a buffer; a light source that emits light for generating a frame image based on the imaging video signal stored in the buffer; a scanning unit that scans the light emitted from the light source to generate the frame image; and a detection unit for detecting a user's line of sight, wherein the signal acquisition unit acquires the imaging video signal at a first resolution for a first pixel area in the imaging pixel area that includes a position corresponding to the line of sight and stores it in the buffer, and acquires the imaging video signal at a second resolution lower than the first resolution for a second pixel area other than the first pixel area in the imaging pixel area and stores it in the buffer.

2. An image generating device as described in claim 1, characterized in that the signal acquisition unit divides the imaging pixel area in a direction perpendicular to the lines to set the first pixel area and the second pixel area, and sets the first resolution and the second resolution by making the frequency of the lines that do not store the imaging video signal in the buffer different between the first pixel area and the second pixel area.

3. An image generating device as described in claim 2, wherein the signal acquisition unit acquires the captured image signal from all the lines in the first pixel region and stores it in the buffer, and acquires the captured image signal from the remaining lines after thinning out the lines at a predetermined interval in the second pixel region and stores it in the buffer.

4. An image generating device according to claim 3, wherein the signal processing section performs the thinning out in units of two adjacent lines.

5. An image generating device as described in claim 1, characterized in that the signal acquisition unit sets the first resolution and the second resolution by varying the gradation that defines the luminance resolution of the captured video signal.

6. An image generating device as claimed in claim 1, comprising: an input processing unit which outputs a first input video signal for constructing a frame image of the first resolution and a second input video signal for constructing a frame image of the second resolution based on a video signal from an external device; and a signal synthesis unit which synthesizes the first input video signal with the captured video signal stored in the buffer at the first resolution, and synthesizes the second input video signal with the captured video signal stored in the buffer at the second resolution.

7. A head mounted display comprising: an image generating device according to any one of claims 1 to 6; and a frame for holding said image generating device.

Images