Image generation device and head-mounted display

Abstract

An image generation device according to the present invention is provided with: a light source that emits light for generating an image; a scanning unit that scans the light emitted from the light source; a plurality of frame buffers that individually store a plurality of types of video signals for constituting images having different definitions; a line-of-sight detection unit (12) for detecting the line of sight of a user; a distance detection unit (14) that detects the distance to an object included in a visual field range; and a control unit (201) that generates an image by controlling the light source, the scanning unit, and the plurality of frame buffers. The control unit (201) applies, to an image region including a viewpoint position corresponding to the line of sight, a video signal from a frame buffer having a lower definition as the distance to the object corresponding to the viewpoint position increases.

Description

Image generating device and head-mounted display

[0001] The present invention relates to an image generating device that generates an image by scanning light and a head-mounted display.

[0002] Conventionally, as an image generating device that generates an image by scanning light, for example, a head-mounted display such as goggles or glasses that realize AR (Augmented Reality) or VR (Virtual Reality) has been known. In these devices, for example, light based on a video signal is irradiated onto a translucent display, and the reflected light is irradiated onto a user's eyes. Alternatively, light based on the video signal is irradiated directly onto a user's eyes.

[0003] Japanese Patent Application Laid-Open No. 2006-124494 (Patent Document 1) describes an apparatus that realizes a first line density in a first portion of an image and a second line density lower than the first line density in a second portion of the image by controlling the rotation of the fast axis and the slow axis of a MEMS mirror, and determines the position of the first portion of the image based on the line of sight of the eye. As a result, the resolution of the image in the second portion that does not correspond to the line of sight is lower than the resolution of the image in the first portion that corresponds to the line of sight, thereby reducing eye fatigue for the user.

[0004] U.S. Patent No. 9,986,215

[0005] In the above configuration, a high-resolution image is displayed at the line of sight. However, when the distance to the object in front of the line of sight is far, the user will see the image near the line of sight with the sensation of looking into the distance. Normally, when a person looks into the distance, the image of an object appears less clear than when looking into the distance. Therefore, if the image near the line of sight is fixed at high resolution, when the distance to the object in front of the line of sight is far, the user will see the image at a higher resolution than normal, which may cause a sense of discomfort.

[0006] In view of these problems, the present invention aims to provide an image generating device and a head-mounted display that can bring the image resolution in the area near the user's line of sight closer to the actual resolution when viewing an object in front of the user's line of sight.

[0007] An image generating device according to a first aspect of the present invention includes a light source that emits light for generating an image, a scanning unit that scans the light emitted from the light source, multiple frame buffers that respectively store multiple types of video signals for constructing the image with different resolutions, a line-of-sight detection unit that detects a user's line of sight, a distance detection unit that detects the distance to an object included in a field of view, and a control unit that controls the light source, the scanning unit, and the multiple frame buffers to generate an image. The control unit applies the video signal from the frame buffer with lower resolution to an image area including a viewpoint position corresponding to the line of sight as the distance to the object corresponding to the viewpoint position increases.

[0008] According to the image generating device of this aspect, for an image region including a viewpoint position corresponding to the line of sight, the greater the distance to an object corresponding to this viewpoint position, the lower the resolution of the video signal applied from the frame buffer. As a result, the greater the distance to an object ahead of the line of sight, the lower the resolution at which the image is displayed near the line of sight. This allows the resolution of the image displayed in the region near the line of sight to approach the actual resolution at which the object ahead of the line of sight is viewed. Furthermore, since multiple video signals for displaying images with different resolutions are pre-stored in multiple frame buffers, the control unit need only select a video signal from one of the frame buffers in response to movement of the line of sight. This allows images to be smoothly displayed in the region near the line of sight with a resolution according to the distance.

[0009] A head-mounted display according to a second aspect of the present invention comprises an image generating device according to the first aspect, a frame for holding the image generating device, and an optical system for directing light from the image generating device to the eyes of the user wearing the head-mounted display on their head.

[0010] The head-mounted display according to this aspect has the same effects as the first aspect, and by wearing the head-mounted display on the head, the user can view the image generated by the image generating device.

[0011] As described above, the present invention provides an image generating device and a head-mounted display that can bring the image resolution in the area near the user's line of sight closer to the actual resolution when viewing an object in front of the user's line of sight.

[0012] The effects and significance of the present invention will become more apparent from the following description of the embodiments, however, the embodiments shown below are merely examples of how the present invention can be implemented, and the present invention is not limited to the embodiments described below.

[0013] FIG. 1 is a perspective view schematically showing the configuration of AR glasses according to the first embodiment. FIG. 2 is a diagram schematically showing the configuration of a projection unit according to the first embodiment. FIG. 3 is a block diagram showing the configurations of a projection unit and a detection unit according to the first embodiment. FIG. 4 is a block diagram showing the configuration of a signal processing unit according to the first embodiment. FIG. 5 is a diagram schematically showing first and second captured video signals acquired by a camera according to the first embodiment. FIG. 6 is a schematic diagram for explaining thinning processing by an input processing unit according to the first embodiment. FIG. 7( a) is a diagram schematically showing generation of a frame image according to a comparative example. FIG. 7( b) is a diagram schematically showing generation of a frame image according to the first embodiment. FIG. 8 is a flowchart showing frame image generation processing performed by an image generating device according to the first embodiment. FIG. 9 is a flowchart showing details of storage processing according to the first embodiment. FIG. 10 is a block diagram showing the configuration of a signal processing unit according to a first modification of the first embodiment. FIG. 11 is a schematic diagram for explaining thinning processing by an input processing unit according to the first modification of the first embodiment. FIG. 12 is a block diagram showing the configuration of a signal processing unit according to a second modification of the first embodiment. FIG. 13 is a diagram schematically illustrating setting of the first image area to one of five areas based on the viewpoint position according to Modification 3 of Embodiment 1. FIG. 14 is a diagram illustrating the scanning speed of the second mirror when five areas are each set as the first image area according to Modification 3 of Embodiment 1. FIG. 15 is a block diagram illustrating the configuration of a signal processing unit according to Embodiment 2. FIG. 16 is a diagram schematically illustrating generation of a frame image according to Embodiment 2. FIG. 17 is a flowchart illustrating frame image generation processing performed by the image generating device according to Embodiment 2. FIG. 18 is a flowchart illustrating processing in step S33 of FIG. 17 according to Embodiment 3. FIGS. 19(a) and 19(b) are diagrams schematically illustrating a frame image generation method according to Embodiment 3. FIG. 20 is a block diagram illustrating the configurations of a projection unit and a gaze detection unit according to Embodiment 4. FIG. 21 is a block diagram illustrating the configuration of a signal processing unit according to Embodiment 4. FIG. 22 is a flowchart illustrating frame image generation processing performed by the image generating device according to Embodiment 4.23(a) and 23(b) are diagrams schematically showing an example of operation of frame image generation processing according to embodiment 4. FIG. 24 is a block diagram showing the configuration of a signal processing unit according to embodiment 5. FIG. 25 is a flowchart showing frame image generation processing performed by an image generation device according to embodiment 5. FIG. 26 is a block diagram showing the configuration of a signal processing unit according to a modified example.

[0014] However, the drawings are for illustrative purposes only and do not limit the scope of the present invention.

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following embodiments, an example in which the present invention is applied to an image generation device for a head-mounted display is shown. Examples of head-mounted displays include AR glasses, AR goggles, VR glasses, and VR goggles. The head-mounted display in the following embodiments is AR glasses. However, the following embodiments are one embodiment of the present invention, and the present invention is not limited to the following embodiments. For example, the present invention is not limited to image generation devices for head-mounted displays, but can also be applied to image generation devices such as in-vehicle head-up displays.

[0016] First Embodiment FIG. 1 is a perspective view schematically illustrating the configuration of AR glasses 1. FIG.

[0017] 1 also shows mutually orthogonal X, Y, and Z axes in addition to the front, back, left, right, up, and down directions of the AR glasses 1. The positive X-axis direction, positive Y-axis direction, and positive Z-axis direction correspond to the rightward, backward, and upward directions of the AR glasses 1, respectively.

[0018] The AR glasses 1 include a frame 2, a pair of image generating devices 3, and a pair of mirrors 4. The AR glasses 1 are worn on the user's head, similar to ordinary eyeglasses.

[0019] The frame 2 holds a pair of image generating devices 3 and a pair of mirrors 4. The frame 2 is composed of a front portion 2a and a pair of support portions 2b. The pair of support portions 2b extend rearward from the right and left ends of the front portion 2a. When the frame 2 is worn by a user, the front portion 2a is positioned in front of a pair of eyes E of the user. The frame 2 is composed of an opaque material. The frame 2 may also be composed of a transparent material.

[0020] The pair of image generation devices 3 are symmetrical with respect to the YZ plane passing through the center of the AR glasses 1. The image generation devices 3 generate images at the eyes E of a user wearing the AR glasses 1 on their head.

[0021] The mirror 4 has a concave reflective surface and is installed on the inner surface of the front surface 2a of the frame 2. The mirror 4 almost totally reflects the light projected from the corresponding projection unit 11 and guides it to the user's eye E.

[0022] The image generating device 3 includes a projection unit 11 , a line-of-sight detection unit 12 , a camera 13 , and a distance detection unit 14 .

[0023] The projection unit 11 is installed on the inner surface of the support unit 2b. The projection unit 11 projects light modulated by a video signal onto the corresponding mirror 4. The light from the projection unit 11 reflected by the mirror 4 is irradiated onto the fovea centralis, which is located at the center of the retina in the eye E. This allows the user to visually grasp the frame image 20 (see FIG. 2) generated by the image generation device 3.

[0024] The pair of line-of-sight detection units 12 are installed on the inner surface of the front surface 2a between the pair of mirrors 4. The line-of-sight detection units 12 are used to detect the line of sight of the user.

[0025] The pair of cameras 13 are installed on the outer surface of the front surface 2a in front of the pair of mirrors 4. The cameras 13 capture an image within the field of view of the cameras 13. In this embodiment, the field of view of the cameras 13 is in front of the AR glasses 1.

[0026] The pair of distance detection units 14 are installed on the outer surface of the front surface 2a so as to be adjacent to the pair of cameras 13 in the left-right direction. The distance detection units 14 detect the distance to an object included in their own field of view. The distance detection units 14 are configured, for example, with TOF (Time Of Flight) cameras. The TOF cameras may be of a flash type or a scanning type.

[0027] The field of view of distance detection unit 14 substantially overlaps with the field of view of camera 13. Therefore, distance detection unit 14 detects the distance to an object included in the field of view of camera 13. The field of view of distance detection unit 14 may be slightly wider than the field of view of camera 13. Distance detection unit 14 detects the distance to each position within the field of view with a predetermined resolution. Therefore, the distance to an object is obtained for each position on the image captured by camera 13 based on the distance detected by distance detection unit 14.

[0028] FIG. 2 is a diagram schematically illustrating the configuration of the projection unit 11. As shown in FIG.

[0029] The projection unit 11 includes light sources 101 , 102 , and 103 , collimator lenses 111 , 112 , and 113 , apertures 121 , 122 , and 123 , a mirror 131 , dichroic mirrors 132 and 133 , a first scanning unit 140 , a relay optical system 150 , and a second scanning unit 160 .

[0030] Light sources 101, 102, and 103 are, for example, semiconductor laser light sources. Light source 101 emits laser light with a red wavelength in the range of 635 nm to 645 nm, light source 102 emits laser light with a green wavelength in the range of 510 nm to 530 nm, and light source 103 emits laser light with a blue wavelength in the range of 440 nm to 460 nm.

[0031] In the first embodiment, a color image is generated as the frame image 20 described below, and therefore the projection unit 11 includes light sources 101, 102, and 103 capable of emitting red, green, and blue laser light. When a monochromatic image is displayed as the frame image 20, the projection unit 11 may include only one light source corresponding to the color of the image. Alternatively, the projection unit 11 may include two light sources with different emission wavelengths.

[0032] The light beams emitted from the light sources 101, 102, and 103 are converted into parallel light beams by collimator lenses 111, 112, and 113, respectively. The light beams transmitted through the collimator lenses 111, 112, and 113 are shaped into approximately circular beams by apertures 121, 122, and 123, respectively.

[0033] Mirror 131 almost totally reflects the red light that has passed through aperture 121. Dichroic mirror 132 reflects the green light that has passed through aperture 122 and transmits the red light reflected by mirror 131. Dichroic mirror 133 reflects the blue light that has passed through aperture 123 and transmits the red light and green light that have passed through dichroic mirror 132. Mirror 131 and the two dichroic mirrors 132 and 133 are arranged to align the optical axes of the light of each color emitted from light sources 101, 102, and 103.

[0034] The first scanning unit 140 reflects the light that has passed through the dichroic mirror 133. The first scanning unit 140 is, for example, a MEMS (Micro Electro Mechanical System) mirror. The first scanning unit 140 is configured to rotate a first mirror 141, onto which the light that has passed through the dichroic mirror 133 is incident, around an axis 141a parallel to the Z-axis direction in response to a drive signal. The rotation of the first mirror 141 changes the reflection direction of the light. As a result, the light reflected by the first mirror 141 is scanned along a scanning line extending in the X-axis direction on the retina of the eye E, as described below.

[0035] The relay optical system 150 directs light reflected by the first scanning unit 140 toward the center of the second mirror 161 of the second scanning unit 160. That is, light incident on the first scanning unit 140 is deflected by the first mirror 141 at a predetermined deflection angle. The relay optical system 150 directs light at each deflection angle toward the center of the second mirror 161. The relay optical system 150 also has multiple mirrors, and reflects the light reflected by the first scanning unit 140 by the multiple mirrors to direct the light toward the second scanning unit 160. This makes it possible to achieve a long optical path length inside the relay optical system 150 and reduce the deflection angle of the light when viewed from the second mirror 161.

[0036] The second scanning unit 160 reflects the light that has passed through the relay optical system 150. The second scanning unit 160 is, for example, a MEMS mirror. The second scanning unit 160 has a configuration that rotates a second mirror 161, onto which the light that has passed through the relay optical system 150 is incident, around an axis 161a parallel to the XY plane in response to a drive signal. The rotation of the second mirror 161 changes the reflection direction of the light. As a result, the scanning line on the retina of the eye E along which the light is scanned by the first scanning unit 140 is changed in the Z-axis direction, as described below.

[0037] The light reflected by the second scanning unit 160, i.e., the light emitted from the projection unit 11, is reflected by the mirror 4 and forms a frame image 20 on the retina of the eye E. This allows the user to view the frame image 20. The field of view of the frame image 20 viewed by the user corresponds to the field of view of the camera 13. Therefore, the distance detection unit 14 in FIG. 1 detects the distance to an object present within the field of view of the user when viewing the frame image 20.

[0038] FIG. 3 is a block diagram showing the configuration of the projection unit 11 and the line-of-sight detection unit 12. As shown in FIG.

[0039] The gaze detection unit 12 includes a light source 12a and an image sensor 12b, and is connected to the control unit 201 of the projection unit 11. The light source 12a is, for example, an LED that emits light of an infrared wavelength. The image sensor 12b is, for example, a CMOS image sensor or a CCD image sensor. The light source 12a irradiates light onto the user's eye E in response to an instruction from the control unit 201. The image sensor 12b captures an image of the user's eye E in response to an instruction from the control unit 201, and outputs the captured image to the control unit 201.

[0040] In response to instructions from the control unit 201, the cameras 13 capture images of their field of view, generate video signals, and output the generated video signals to the signal processing unit 300 of the corresponding projection unit 11. In Fig. 1 , the left camera 13 outputs the generated video signals to the signal processing unit 300 of the left projection unit 11, and the right camera 13 outputs the generated video signals to the signal processing unit 300 of the right projection unit 11. The cameras 13 of the first embodiment output a first captured video signal for high resolution and a second captured video signal for low resolution, as will be described later.

[0041] The camera 13 also has a function of a moving object detection unit 13a as a function of its built-in processing circuit. The moving object detection unit 13a detects a moving object region in a frame image 20 using the first captured video signal or the second captured video signal. The moving object detection unit 13a identifies a region of an object moving in a frame image (a moving object region) using a plurality of consecutive frame images 20. For example, the moving object detection unit 13a calculates the difference in the captured video signal for each pixel between a plurality of consecutive frame images 20, and identifies a region of an object moving in a frame image (a moving object region) from the calculated difference. The moving object detection unit 13a sequentially outputs information indicating the position of the moving object region in the frame image 20 to the control unit 201.

[0042] The control unit 201 may have the function of the moving object detection unit 13a. In this case, the first captured video signal or the second captured video signal is output from the camera 13 to the control unit 201. The control unit 201 detects a moving object area moving on the frame image 20 from the input captured video signal.

[0043] The projection unit 11 includes a control unit 201, a first mirror driving circuit 211, a second mirror driving circuit 212, a first mirror monitor sensor 213, a second mirror monitor sensor 214, a signal processing unit 300, a line memory 221, and a laser driving circuit 222.

[0044] The control unit 201 includes an arithmetic processing unit such as a CPU or FPGA, and a memory. The control unit 201 detects the user's line of sight based on the captured image from the line of sight detection unit 12, for example, by the dark pupil method, the bright pupil method, or the corneal reflex method. The control unit 201 acquires the viewpoint position in the frame image 20 formed on the user's retina based on the detected line of sight of the user. The control unit 201 also controls the signal processing unit 300 to process video signals from the camera 13 and external devices.

[0045] The first mirror drive circuit 211 drives the first mirror 141 of the first scanning unit 140 in response to a drive signal from the control unit 201. The second mirror drive circuit 212 drives the second mirror 161 of the second scanning unit 160 in response to a drive signal from the control unit 201.

[0046] The first mirror monitor sensor 213 is installed on the first mirror 141 and outputs a detection signal corresponding to the rotation of the first mirror 141 to the control unit 201. The second mirror monitor sensor 214 is installed on the second mirror 161 and outputs a detection signal corresponding to the rotation of the second mirror 161 to the control unit 201. Based on the detection signals from the first mirror monitor sensor 213 and the second mirror monitor sensor 214, the control unit 201 outputs drive signals to the first mirror drive circuit 211 and the second mirror drive circuit 212 so that the first mirror 141 and the second mirror 161 rotate with desired drive waveforms.

[0047] The signal processing unit 300 processes the video signals from the camera 13 and external devices to generate a video signal for one line. The configuration of the signal processing unit 300 will be described later with reference to FIG.

[0048] The line memory 221 outputs the video signal for one line output from the signal processing unit 300 to the laser driving circuit 222. The laser driving circuit 222 drives the light sources 101, 102, and 103 to emit light modulated by the video signal for one line output from the line memory 221.

[0049] FIG. 4 is a block diagram showing the configuration of the signal processing unit 300.

[0050] The signal processing unit 300 includes a first buffer 301, a second buffer 302, an input processing unit 310, a first buffer 321, a second buffer 322, a signal synthesis unit 330, a first frame buffer 341, and a second frame buffer 342.

[0051] In the first embodiment, the imaging processing unit 230 is configured by the camera 13. The camera 13 captures an image of the field of view and generates a first imaging video signal for high resolution and a second imaging video signal for low resolution. The first buffer 301 is a memory that temporarily stores the first imaging video signal output from the camera 13 (imaging processing unit 230). The second buffer 302 is a memory that temporarily stores the second imaging video signal output from the camera 13 (imaging processing unit 230).

[0052] FIG. 5 is a diagram schematically showing the first and second image pickup video signals acquired by the camera 13. As shown in FIG.

[0053] The camera 13 generates a first imaging video signal in a first imaging period set within one frame, and generates a second imaging video signal in a second imaging period different from the first imaging period within one frame. For example, the first imaging period is the first half period within one frame, and the second imaging period is the second half period within one frame, and the lengths of the first imaging period and the second imaging period are the same.

[0054] During the first imaging period, the camera 13 drives all of the light receiving units (pixels) in the camera 13 to generate a first imaging video signal for high resolution. Meanwhile, during the second imaging period, the camera 13 drives every other row of the light receiving units arranged horizontally in the camera 13 to generate a second imaging video signal for low resolution.

[0055] In Fig. 5, the video signals stored in the first buffer 301 and the second buffer 302 for each line are indicated by solid and dashed lines. Of the first captured video signals stored in the first buffer 301, the odd-numbered first captured video signals from the top are indicated by solid lines, and the even-numbered first captured video signals from the top are indicated by dashed lines. For convenience, the first buffer 301 in Fig. 5 shows 17 lines of first captured video signals, but the actual number of lines is several levels greater. In the second captured video signals stored in the second buffer 302, the dashed lines of the first buffer 301 are omitted and the second captured video signals are indicated by solid lines.

[0056] 5, the second imaged video signal in the second buffer 302 is a signal in which every other line of the first imaged video signal in the first buffer 301 is thinned out by driving each light receiving unit in the camera 13. However, the way in which the lines are thinned out is not limited to this, and for example, the second imaged video signal in the second buffer 302 may be a signal in which every three or more lines of the first imaged video signal in the first buffer 301 are thinned out by one line.

[0057] Returning to Fig. 4, the video signal from the external device is, for example, a video signal related to CG (Computer Graphics) that is superimposed on a predetermined region of the frame image 20. This video signal has the same resolution as the first captured video signal output from the camera 13. The input processing unit 310 performs thinning processing on the video signal input from the external device. The first buffer 321 is a memory that temporarily stores the video signal input from the external device, i.e., the first input video signal that has not been thinned out by the input processing unit 310. The second buffer 322 is a memory that temporarily stores the second input video signal that has been thinned out by the input processing unit 310.

[0058] FIG. 6 is a schematic diagram for explaining the thinning process performed by the input processing unit 310. In FIG.

[0059] The input processing unit 310 generates a first input video signal and a second input video signal having different resolutions from a video signal from an external device.

[0060] In Fig. 6, the video signals for each line stored in the first buffer 321 and the second buffer 322 are indicated by solid and dashed lines. Of the first input video signals stored in the first buffer 321, the odd-numbered first input video signals from the top are indicated by solid lines, and the even-numbered first input video signals from the top are indicated by dashed lines. For convenience, the first buffer 321 in Fig. 6 shows 17 lines of first input video signals, but the actual number of lines is several levels greater. The second input video signal stored in the second buffer 322 is a signal obtained by thinning out the dashed lines of the first buffer 321.

[0061] 6, the second input video signal of the second buffer 322 is a signal obtained by thinning out every other line of the first input video signal of the first buffer 321. However, the method of thinning out the lines is not limited to this. For example, the second input video signal may be a signal obtained by thinning out every three or more lines of the first input video signal of the first buffer 321. Furthermore, the second input video signal may be generated by mixing adjacent lines of the first input video signal stored in the first buffer 321. In the mixing process, for example, two adjacent lines are replaced with one line calculated as the average value of the signals of these two lines.

[0062] 4 , the signal combining unit 330 combines the first captured video signal for high resolution stored in the first buffer 301 with the first input video signal for high resolution stored in the first buffer 321 to generate a first combined video signal for high resolution for one frame. The signal combining unit 330 also combines the second captured video signal for low resolution stored in the second buffer 302 with the second input video signal for low resolution stored in the second buffer 322 to generate a second combined video signal for low resolution for one frame.

[0063] The first frame buffer 341 stores one frame of the first synthesized video signal for high resolution generated by the signal synthesis unit 330. The second frame buffer 342 stores one frame of the second synthesized video signal for low resolution generated by the signal synthesis unit 330.

[0064] In response to a control signal from the control unit 201 (see FIG. 3 ), the first frame buffer 341 sequentially outputs one line of the first composite video signal out of one frame of the stored first composite video signal to the line memory 221. In response to a control signal from the control unit 201, the second frame buffer 342 sequentially outputs one line of the second composite video signal out of one frame of the stored second composite video signal to the line memory 221. The line memory 221 receives either one line of the first composite video signal from the first frame buffer 341 or one line of the second composite video signal from the second frame buffer 342.

[0065] Next, a method for generating a frame image 20 according to a comparative example and a method for generating a frame image 20 according to the first embodiment will be described in order.

[0066] FIG. 7A is a diagram schematically showing the generation of a frame image 20 according to a comparative example.

[0067] The first scanning unit 140 scans light in the X-axis direction along the scanning line, and the second scanning unit 160 changes the scanning line in the Z-axis direction, thereby generating a frame image 20 on the retina of the user's eye E. When the scanning line is changed, the scanning positions of the first scanning unit 140 and the second scanning unit 160 are moved with the light sources 101, 102, and 103 turned off, as shown by the dotted lines in FIG. 7A. When changing the scanning line, the scanning position is moved to the beginning of the scanning line of the next row when scanning of each row of scanning lines is completed, except for the bottom row of scanning lines. When scanning of the bottom row of scanning lines is completed, the scanning position is moved to the beginning of the top row of scanning lines. In the comparative example, the frame image 20 is generated based only on a high-resolution video signal.

[0068] However, if the entire frame image 20 has a high resolution as shown in Fig. 7(a), the user's eyes will become tired easily. Therefore, in the first embodiment, the resolution (number of scanning lines) of the frame image 20 is set low in the area outside a predetermined range including the user's viewpoint position P10, as shown in Fig. 7(b). This reduces the user's eye fatigue.

[0069] However, if the resolution (number of scanning lines) is to be changed within the frame image 20, scanning must be performed using a video signal that corresponds to the changed resolution (number of scanning lines) as needed. However, because the user's viewpoint position P10 can change dynamically, if video signals for each resolution (number of scanning lines) are to be generated as needed in response to changes in the viewpoint position P10, the generation of the video signals may not be able to keep up, resulting in a delay in display. If such a display delay occurs, the frame image 20 will be distorted, causing discomfort to the user.

[0070] Therefore, in the first embodiment, as described above, a first composite video signal with a large number of scanning lines (high resolution) is stored in advance in the first frame buffer 341, and a second composite video signal with a small number of scanning lines (low resolution) is stored in the second frame buffer 342. Then, the control unit 201 switches between the first composite video signal from the first frame buffer 341 and the second composite video signal from the second frame buffer 342 based on the viewpoint position P10 and an area to which the user's gaze is likely to be directed, thereby generating the frame image 20. This makes it possible to suppress the above-mentioned display delay and realize image generation that tracks the user's viewpoint position P10.

[0071] FIG. 7B is a diagram schematically showing the generation of a frame image 20 according to the first embodiment.

[0072] In the first embodiment, the control unit 201 detects the user's line of sight based on the captured image acquired by the line of sight detection unit 12, and acquires a viewpoint position P10 on the frame image 20 based on the detected line of sight. The control unit 201 controls the light sources 101, 102, 103, the first scanning unit 140, and the second scanning unit 160 so that an image is generated in a first image region R1 of a predetermined number of scanning lines that includes the viewpoint position P10 on the frame image 20 by selectively applying either a high-resolution first composite video signal from the first frame buffer 341 or a low-resolution second composite video signal from the second frame buffer 342, depending on the distance detected by the distance detection unit 14 from the viewpoint position P10.

[0073] The control unit 201 also sets a specific region R3 on the frame image 20, the specific region R3 including at least a portion of an area toward which the gaze is likely to be directed. In the first embodiment, a moving object region R10 that moves on the frame image 20 is set as an area toward which the gaze is likely to be directed. As described above, the moving object region R10 is detected by the moving object detection unit 13a and output to the control unit 201. The control unit 201 controls the light sources 101, 102, 103, the first scanning unit 140, and the second scanning unit 160 so that an image is generated by selectively applying either a high-resolution first composite video signal from the first frame buffer 341 or a low-resolution second composite video signal from the second frame buffer 342 to the specific region R3 having a predetermined number of scanning lines and including the moving object region R10 on the frame image 20, depending on the distance detected by the distance detection unit 14 with respect to the center position of the specific region R3.

[0074] In addition, the control unit 201 controls the light sources 101, 102, 103, the first scanning unit 140 and the second scanning unit 160 so that an image is generated by applying a low-resolution second composite video signal from the second frame buffer 342 to the second image area R2 other than the first image area R1 and the specific area R3 on the frame image 20.

[0075] Here, the control unit 201 sets the specific region R3 to be smaller than the first image region R1. Specifically, the control unit 201 sets the width of the specific region R3 in the Z-axis direction to be smaller than the width of the first image region R1 in the Z-axis direction. More specifically, the control unit 201 sets the number of scanning lines included in the specific region R3 to be smaller than the number of scanning lines included in the first image region R1.

[0076] The width of the first image region R1 in the Z-axis direction is predetermined. The control unit 201 sets the first image region R1 so that the viewpoint position P10 is aligned with the center of this width. The width of the first image region R1 may be set taking into account the width of the range in the Z-axis direction around the viewpoint that can normally be clearly recognized by a person. For example, the width of the first image region R1 in the Z-axis direction may be set to approximately the width of the range in the Z-axis direction around the viewpoint that can normally be clearly recognized by a person.

[0077] The width of the specific region R3 in the Z-axis direction is predetermined. The control unit 201 sets the specific region R3 so that the center of gravity of the moving object region R10 is aligned with the center of this width. If the width of the moving object region R10 in the Z-axis direction is smaller than the specified width of the specific region R3, the control unit 201 may reduce the width of the specific region R3 from the specified width to the width of the moving object region R10.

[0078] It is not necessary to set a specified width for the width of the specific region R3 in the Z-axis direction. That is, the control unit 201 may dynamically change the width of the specific region R3 according to the size of the moving object region R10 so that the width of the specific region R3 approximately matches the width of the moving object region R10.

[0079] However, in this case, if the moving object region R10 becomes larger because the moving object is closer to the camera 13, the width of the specific region R3 in the Z-axis direction becomes larger than the width of the first image region R1, and if a high-resolution image is displayed in these regions, the region in which the high-resolution image is displayed will occupy a larger proportion of the frame image 20. This may reduce the effect of alleviating eye fatigue in the user.

[0080] In contrast to this, by setting specific region R3 smaller than first image region R1 as described above, when a high-resolution image is displayed in first image region R1 and specific region R3, it is possible to primarily increase the resolution of first image region R1, which is the area the user's gaze is directed at, while subsidiarily increasing the resolution of specific region R3, which is the area the gaze is likely to be directed at next, thereby appropriately reducing the proportion of the area in which the high-resolution image is displayed in the frame image 20. This makes it possible to smoothly and quickly display a high-resolution image at the position of the user's gaze by following changes in the user's gaze while appropriately reducing eye fatigue.

[0081] When the width of the specific region R3 is dynamically changed according to the size of the moving object region R10 as described above, a predetermined upper limit may be set for the width of the specific region R3 in the Z-axis direction. For example, as with the width of the first image region R1, this upper limit may be set taking into account the width in the Z-axis direction of the range around the viewpoint that can normally be clearly recognized by a human. This prevents the specific region R3 from becoming significantly larger, thereby appropriately reducing eye fatigue in the user.

[0082] In FIG. 7(b), for convenience, about five scanning lines are shown in the first image region R1, a total of about four scanning lines are shown in the second image region R2, and about three scanning lines are shown in the specific region R3, but the actual number of scanning lines is several levels higher than this.

[0083] The number of scanning lines included in the first image region R1 may be changed as appropriate. Also, in Fig. 7B, the first image region R1 has ranges with the same number of scanning lines above and below the viewpoint position P10 as the center, but the number of scanning lines corresponding to the upper range and the number of scanning lines corresponding to the lower range may be different from each other.

[0084] In addition to the above-described resolution control, in the first embodiment, the resolution is further controlled in accordance with the distance detected by the distance detection unit 14. That is, the image resolutions in the first image region R1 and the specific region R3 are not fixed to a high resolution, but are adjusted in accordance with the distance detected by the distance detection unit 14 for these regions.

[0085] When the distance to an object in the line of sight is far, the user will see an image near the line of sight (viewpoint position P10) with the sensation of looking into the distance. Normally, when a person looks into the distance, the image of an object appears less clear than when looking into the distance. Therefore, if the image near the line of sight is fixed at high resolution, when the distance to the object in the line of sight is far, the user will see the image at a higher resolution than normal, which can cause a sense of discomfort.

[0086] To solve this problem, the control unit 201 generates an image by applying a video signal (second composite video signal) from a frame buffer (here, the second frame buffer 342) with a lower definition (lower resolution) the greater the distance to the object corresponding to the viewpoint position P10, to an image region (first image region R1) that includes the viewpoint position P10 corresponding to the line of sight. This makes it possible to bring the definition (resolution) of the image displayed in the first image region R1 closer to the actual definition (resolution) when viewing an object in front of the line of sight, thereby reducing any discomfort the user may feel when viewing the image.

[0087] Furthermore, the control unit 201 applies a video signal from a frame buffer (here, the second frame buffer 342) with a lower definition (lower resolution) to generate an image in the specific region R3, as the distance to the object corresponding to the position of the specific region R3 (for example, the position of the center of gravity of the moving object region R10) increases. This allows the definition (resolution) of the image displayed in the specific region R3 to approach the actual definition when viewing the object located in the specific region R3, thereby reducing the sense of incongruity felt when the user shifts their gaze to the specific region R3 and looks at the specific region R3.

[0088] FIG. 8 is a flowchart showing the process of generating a frame image 20 performed by the image generating device 3.

[0089] The processing in steps S11 to S21 is related to the generation of a frame image 20 corresponding to one frame.

[0090] The control unit 201 performs a storage process (S11), whereby a first composite video signal for one frame of high resolution and a second composite video signal for one frame of low resolution are stored in the first frame buffer 341 and the second frame buffer 342, respectively, based on the video signal from the camera 13 and the video signal from the external device.

[0091] Fig. 9 is a flowchart showing details of the storage process in step S11 in Fig. 8. The control unit 201 controls the image capture processing unit 230 and the signal processing unit 300 to execute the process in Fig. 9.

[0092] 5, the imaging processing unit 230 (camera 13 in the first embodiment) generates a first imaging video signal for high resolution during a first imaging period, and generates a second imaging video signal for low resolution during a second imaging period. Then, the imaging processing unit 230 stores the generated first imaging video signal and second imaging video signal in the first buffer 301 and the second buffer 302, respectively (S101).

[0093] 6, the input processing unit 310 generates a first input video signal for high resolution and a second input video signal for low resolution based on a video signal input from an external device. Then, the input processing unit 310 stores the generated first input video signal and second input video signal in the first buffer 321 and the second buffer 322, respectively (S102). Note that the processes of steps S101 and S102 are performed in parallel.

[0094] The signal synthesis unit 330 synthesizes the first captured video signal stored in the first buffer 301 with the first input video signal of the same line as the first captured video signal among the first input video signals stored in the first buffer 321, to generate a first synthesized video signal for one frame of high resolution. The signal synthesis unit 330 then stores the generated first synthesized video signal in the first frame buffer 341 (S103).

[0095] The signal synthesis unit 330 synthesizes the second captured video signal stored in the second buffer 302 with the second input video signal of the same line as the second captured video signal, among the second input video signals stored in the second buffer 322, to generate a second synthesized video signal for one frame of low resolution. The signal synthesis unit 330 then stores the generated second synthesized video signal in the second frame buffer 342 (S104). Note that the processes of steps S103 and S104 are performed in parallel.

[0096] Returning to FIG. 8, the processes in steps S12 to S20 are related to the generation of one line of an image.

[0097] In parallel with the processing of steps S12 to S20, the control unit 201 controls the first mirror driving circuit 211 so that the first mirror 141 rotates repeatedly at the same period in the processing of each line, and drives the laser driving circuit 222 based on the video signal for one line input to the line memory 221.

[0098] The control unit 201 detects the user's viewpoint position P10 based on the captured image acquired by the line-of-sight detection unit 12 (S12). The control unit 201 sets a first image region R1 and a second image region R2 based on the viewpoint position P10 detected in step S12 (S13). Furthermore, the control unit 201 sets a specific region R3 as described above based on the detection result from the moving object detection unit 13a (S14).

[0099] The control unit 201 determines whether the current scanning line is within the first image region R1 or the specific region R3 (S15).

[0100] If the current scanning line is within the first image region R1 or the specific region R3 (S15: YES), the control unit 201 refers to the detection results sequentially input from the distance detection unit 14 and determines whether the distance to the object at the viewpoint position P10 or the moving object region R10 including the current scanning line is equal to or less than a predetermined threshold (S16). The threshold is set to, for example, about several meters.

[0101] If the distance to the object corresponding to the viewpoint position P10 or the moving object region R10 is equal to or less than the threshold (S16: YES), the control unit 201 outputs a high-resolution first composite video signal from the first frame buffer 341 to the line memory 221 (S17). This generates one line of image using the first composite video signal from the first frame buffer 341. In parallel with this, the control unit 201 controls the second mirror drive circuit 212 to rotate the second mirror 161 at the first scanning speed (S18). This narrows the spacing between vertically adjacent scanning lines, as shown in the first image region R1 or specific region R3 in FIG. 7B. In this way, a high-resolution image is displayed in the first image region R1 or specific region R3.

[0102] On the other hand, if the distance to the object corresponding to the viewpoint position P10 or the moving object region R10 is greater than the threshold (S16: NO), the control unit 201 causes the second frame buffer 342 to output a low-resolution second composite video signal to the line memory 221 (S19). This generates one line of image using the second composite video signal from the second frame buffer 342. Concurrently, the control unit 201 controls the second mirror drive circuit 212 to rotate the second mirror 161 at a second scanning speed faster than the first scanning speed (S20). This widens the spacing between vertically adjacent scanning lines, as shown in the second image region R2 in FIG. 7B. Thus, a low-resolution image is displayed in the first image region R1 or the specific region R3.

[0103] If the current scanning line is in the second image region R2 (S15: NO), the control unit 201 executes the processes of steps S19 and S20, as in the case where the determination in step S16 is NO, thereby displaying a low-resolution image in the second image region R2.

[0104] In steps S17 and S19, the first composite video signal or the second composite video signal is output to the line memory 221 so that, at the start of scanning in steps S18 and S20, the first composite video signal or the second composite video signal corresponding to that scan line is held in the line memory 221. In other words, when the thinning process shown in Fig. 6 is performed, the frequency with which the first composite video signal is output to the line memory 221 is about twice the frequency with which the second composite video signal is output to the line memory 221.

[0105] The control unit 201 determines whether or not image generation for one frame has been completed (S21). If image generation for one frame has not been completed (S21: NO), the control unit 201 returns the process to step S12 and executes the processes of steps S12 to S20 again. When image generation for one frame has been completed in this manner (S21: YES), the process of Fig. 8 ends. By repeating the process of Fig. 8, frame images 20 are continuously generated.

[0106] If the specific region R3 set in step S14 overlaps the first image region R1 in part or in whole, the specific region R3 and the first image region R1 are merged into a single region. In this case, step S15 determines whether the current scanning line is within the merged region, and if the scanning line is within the merged region, the process proceeds to step S16.

[0107] <Effects of First Embodiment> According to the first embodiment, the following effects are achieved.

[0108] 4, the image generating device 3 includes a plurality of frame buffers (first frame buffer 341, second frame buffer 342) that respectively store a plurality of types of video signals (first composite video signal, second composite video signal) for constructing images (frame images 20) with different definitions (resolutions). As shown in Fig. 8, the control unit 201 applies a video signal (second composite video signal) from a low-definition frame buffer (second frame buffer 342) to an image region (first image region R1) that includes a viewpoint position P10 corresponding to the line of sight (S15: YES), the greater the distance to the object corresponding to viewpoint position P10 (S16: NO) (S19).

[0109] With this configuration, the image displayed in the vicinity of the line of sight (first image region R1) has lower definition (resolution) as the distance to the object in front of the line of sight increases, so that the definition of the image displayed in the region near the line of sight (first image region R1) can be made closer to the actual definition when viewing the object in front of the line of sight.

[0110] Furthermore, since multiple video signals (first composite video signal, second composite video signal) for displaying images with different resolutions are pre-stored in multiple frame buffers (first frame buffer 341, second frame buffer 342), the control unit 201 only needs to select a video signal from one of the frame buffers in response to the movement of the line of sight, thereby enabling images according to distance to be displayed smoothly and quickly in the area near the line of sight (first image area R1).

[0111] 4, the image generating device 3 includes an imaging processing unit 230 that captures an image of the field of view and outputs a plurality of types of captured video signals (first captured video signal, second captured video signal) for constructing frame images 20 with different definitions (resolutions). The plurality of types of video signals (first composite video signal, second composite video signal) stored in a plurality of frame buffers (first frame buffer 341, second frame buffer 342) are constructed using the plurality of types of captured video signals (first captured video signal, second captured video signal), and the image region including the viewpoint position P10 is a first image region R1 of a predetermined size on the frame image 20 that includes the viewpoint position P10.

[0112] This configuration allows the user to view an image of the field of view in front of the user. Furthermore, the definition (resolution) of the image area near the user's line of sight can be adjusted according to the distance to the object in front of the user's line of sight. Therefore, the definition (resolution) of the first image area near the line of sight on the frame image 20 can be made closer to the actual definition when viewing the object in front of the user's line of sight.

[0113] As shown in Figure 8, the control unit 201 applies a video signal (second composite video signal) from a frame buffer (second frame buffer 342) corresponding to a predetermined definition (resolution) other than the highest definition (resolution) to a second image region R2 other than the first image region R1 on the frame image 20.

[0114] According to this configuration, the definition (resolution) of the second image region R2 outside the vicinity of the line of sight is set low, thereby reducing eye fatigue of the user.

[0115] As shown in Figure 8, the control unit 201 sets a specific area R3 on the frame image 20 that includes at least a portion of an area that is likely to be looked at (S14), and applies a video signal (second composite video signal) from a low-definition frame buffer (second frame buffer 342) to the specific area R3 (S15: YES) (S19), the greater the distance to the object corresponding to the position of the specific area R3 (S16: NO).

[0116] With this configuration, the resolution of the image displayed in the specific area R3 can be made closer to the resolution when actually viewing an object located at the position of the specific area R3, thereby reducing the sense of discomfort felt when the user shifts their gaze to the specific area R3 and looks at the specific area R3.

[0117] As shown in Figure 3, the image generating device 3 further includes a moving object detection unit 13a that detects a moving object region R10 on the frame image 20, and the control unit 201 sets the detected moving object region R10 as a specific region that is likely to attract attention.

[0118] When a user sees movement in a position other than the user's line of sight, the user is likely to shift their gaze to that position. With this configuration, the specific region R3 is set to include at least a portion of the moving object region R10, so the resolution of the moving object region R10, to which the user's gaze is likely to be directed next, can be adjusted in advance according to the distance to the object. Therefore, when the user's gaze is actually directed toward the moving object region R10 in response to the object's movement, an image with a resolution close to the user's perception according to the distance can be displayed in that region.

[0119] As shown in FIG. 7B, the control unit 201 sets the specific region R3 to be smaller than the first image region R1.

[0120] As a result, when a high-resolution image is displayed in the first image region R1 and the specific region R3, the first image region R1, to which the user's gaze is directed, can be primarily made high-resolution, while the specific region R3, to which the gaze is likely to be directed next, can be made high-resolution secondarily, and the proportion of the region in which the high-resolution image is displayed relative to the frame image 20 can be appropriately reduced. Therefore, a high-resolution image can be displayed smoothly and quickly at the position of the user's gaze by following changes in the user's gaze while appropriately reducing eye fatigue.

[0121] As shown in FIG. 7B, the image definition adjusted in the first image region R1 and the specific region R3 is the image resolution determined by the scanning speed of the scanning unit (second scanning unit 160).

[0122] According to this configuration, the scanning speed of the scanning section (second scanning section 160) is controlled to adjust the density of the scanning lines, thereby making it possible to adjust the definition of the image.

[0123] As shown in FIG. 3, the moving object detector 13 a is disposed in the camera 13 that acquires frame images for display, and detects a moving object region R10 from the image signal captured by the camera 13 .

[0124] According to this configuration, the camera 13 is used both to acquire frame images for display and to detect the moving object region R10, which allows the configuration of the image generating device 3 to be simplified.

[0125] As shown in Figure 5, camera 13 outputs a first imaging video signal corresponding to high resolution during a first imaging period within one frame period, and outputs a second imaging video signal corresponding to low resolution during a second imaging period different from the first imaging period within one frame period.

[0126] According to this configuration, the first imaged video signal with high resolution and the second imaged video signal with low resolution can be generated smoothly.

[0127] Based on a video signal from an external device, the input processing unit 310 outputs a first input video signal for constructing a high-resolution (first definition) frame image 20 and a second input video signal for constructing a low-resolution (second definition) frame image 20. The signal combining unit 330 stores a first combined video signal, obtained by combining the first captured video signal and the first input video signal, in a first frame buffer 341, and stores a second combined video signal, obtained by combining the second captured video signal and the second input video signal, in a second frame buffer 342.

[0128] According to this configuration, the video signal from the imaging processing unit 230 and the video signal from the external device can be combined to generate the frame image 20. This allows a CG image or the like based on the video signal to be added to the image captured by the camera 13 and displayed, allowing the user to grasp the additional information based on the video signal.

[0129] As shown in Figure 1, the mirror 4 (optical system) guides the light scanned by the first scanning unit 140 and the second scanning unit 160 (scanning unit) to the eyes E of a user wearing AR glasses 1 (head-mounted display) on their head.

[0130] According to this configuration, by wearing the AR glasses 1 on the head, the user can grasp the scenery captured by the camera 13 through the frame image 20 generated by the image generation device 3.

[0131] <Modification 1 of Embodiment 1> In Embodiment 1, camera 13 outputs both the first captured video signal and the second captured video signal, but this is not limited to this. Camera 13 may output one type of video signal, and a processing unit located downstream of camera 13 may generate video signals with different resolutions.

[0132] FIG. 10 is a block diagram showing the configuration of a signal processing unit 300 according to this modification.

[0133] 4, the signal processing unit 300 in Fig. 10 includes an input processing unit 350 between the camera 13 and the first and second buffers 301 and 302. The imaging processing unit 230 in this modified example is made up of the camera 13 and the input processing unit 350.

[0134] The camera 13 of this modified example outputs only a high-resolution video signal similar to the first captured video signal of embodiment 1. The input processing unit 350 performs a thinning process on the video signal input from the camera 13 similar to the thinning process performed by the input processing unit 310. The first buffer 301 temporarily stores the first captured video signal output from the camera 13 that has not been thinned out by the input processing unit 350. The second buffer 302 temporarily stores the second captured video signal that has been thinned out by the input processing unit 350.

[0135] FIG. 11 is a schematic diagram for explaining the thinning process by the input processing unit 350. In FIG.

[0136] The input processing unit 350 generates, from the video signal from the camera 13, a first captured video signal and a second captured video signal having different resolutions.

[0137] In Fig. 11, the video signals for each line stored in the first buffer 301 and the second buffer 302 are indicated by solid and dashed lines. Of the first imaged video signals stored in the first buffer 301, the odd-numbered first imaged video signals from the top are indicated by solid lines, and the even-numbered first imaged video signals from the top are indicated by dashed lines. For convenience, the first buffer 301 in Fig. 11 shows 17 lines of first imaged video signals, but the actual number of lines is several levels greater. The second imaged video signal stored in the second buffer 302 is a signal obtained by thinning out the dashed lines from the first buffer 301.

[0138] 11 , the second imaged video signal in the second buffer 302 is a signal obtained by thinning out every other line of the first imaged video signal in the first buffer 301. However, the manner of thinning out the lines is not limited to this. For example, the second imaged video signal may be a signal obtained by thinning out every three or more lines of the first imaged video signal in the first buffer 301. The second imaged video signal may also be generated by mixing adjacent lines of the first imaged video signal stored in the first buffer 301. In the mixing process, for example, two adjacent lines are replaced with one line calculated as the average value of the signals of these two lines.

[0139] <Effects of Modification 1 of Embodiment 1> The camera 13 outputs a first imaging video signal corresponding to a high resolution, and the imaging processing unit 230 includes an input processing unit 350 that thins out or mixes the first imaging video signal of high resolution to generate a second imaging video signal corresponding to a low resolution.

[0140] According to this configuration, the camera 13 only needs to output one type of first imaging video signal, thereby simplifying the configuration and processing of the camera 13. Furthermore, in a configuration in which the first imaging video signal and the second imaging video signal are obtained in two different imaging periods as shown in the first embodiment, if the subject moves at high speed, the position of the subject will differ between the two types of video signals. However, according to this modified example, because the camera 13 only outputs one type of first imaging video signal, the position of the subject will be the same between the first imaging video signal and the second imaging video signal generated by the input processing unit 350, even if the subject moves at high speed. This avoids the user feeling uncomfortable due to the first imaging video signal and the second imaging video signal.

[0141] <Modification 2 of Embodiment 1> In the first embodiment, both the video signal from the camera 13 and the video signal from the external device are input to the signal processing unit 300, but as will be described below, only the video signal from the camera 13 may be input to the signal processing unit 300.

[0142] FIG. 12 is a block diagram showing the configuration of a signal processing unit 300 according to this modification.

[0143] 4, the signal processing unit 300 in Fig. 12 does not include the input processing unit 310, the first buffer 321, the second buffer 322, and the signal combining unit 330. The first captured video signal from the camera 13 is temporarily stored in the first buffer 301, and then one frame of the first captured video signal is output to the first frame buffer 341. The second captured video signal from the camera 13 is temporarily stored in the second buffer 302, and then one frame of the second captured video signal is output to the second frame buffer 342.

[0144] In this configuration, the frame image generation process is the same as that shown in FIG.

[0145] Effect of Modification 2 of First Embodiment The first frame buffer 341 stores the first captured video signal from the first buffer 301, and the second frame buffer 342 stores the second captured video signal from the second buffer 302. The process of generating the frame image 20 is similar to that shown in Fig. 8. Either the first captured video signal from the first frame buffer 341 or the second captured video signal from the second frame buffer 342 is selectively applied to the first image region R1 and the specific region R3 depending on the distance to the object, to display an image. The second captured video signal from the second frame buffer 342 is applied to display an image in the second image region R2.

[0146] With this configuration, as in the above-mentioned first embodiment, the resolution of the images displayed in the first image region R1 near the line of sight and the specific region R3 to which the line of sight is likely to be directed can be made closer to the actual resolution (resolution) when viewing objects located in these regions.

[0147] In this modified example, as in modified example 1 shown in Figure 10, one type of video signal may be output from camera 13, and input processing unit 350 may be arranged between camera 13 and first buffer 301 and second buffer 302.

[0148] <Modification 3 of Embodiment 1> In Embodiment 1, the first image area R1 is set based on the viewpoint position P10 based on the user's line of sight, but this is not limited to this, and the first image area R1 may be fixed in advance.

[0149] FIG. 13 is a diagram showing, in this modified example, a schematic diagram illustrating that the first image region R1 is set to one of five regions R11 to R15 based on the viewpoint position P10.

[0150] When viewpoint position P10 is included in viewpoint regions R01 to R05 in frame image 20, control unit 201 sets regions R11 to R15 as the first image region R1, respectively. Viewpoint regions R01 to R05 are obtained by dividing frame image 20 into five regions in the vertical direction (Z-axis direction). Regions R11 to R15 are regions set corresponding to viewpoint regions R01 to R05, and each region includes a predetermined number of scanning lines.

[0151] FIG. 14 is a diagram showing the scanning speed of the second mirror 161 when five regions R11 to R15 are each set in the first image region R1 according to this modified example.

[0152] When regions R11 to R15 are set in the first image region R1 as shown in the upper part of Fig. 14, the scanning speed of the second mirror 161 is set as shown in the graph in the lower part of Fig. 14. The graph in the lower part of Fig. 14 shows the scanning speed of the second mirror 161 for scanning one line. In the five graphs in the lower part of Fig. 14, the speed of the second mirror 161 is slower in each of the regions R11 to R15. As a result, when the five regions R11 to R15 are set in the first image region R1, respectively, the image resolution is increased in the ranges corresponding to the regions R11 to R15.

[0153] As for the specific region R3, as described above, among the multiple selection candidates for the specific region R3 set in advance, the selection candidate that includes the center of gravity of the moving object region R10 may be set as the specific region R3 for displaying a high-resolution image.

[0154] In this modified example, similarly to the first embodiment, the area other than the first image area R1 and the specific area R3 is set as the second image area R2.

[0155] <Effects of Modification 3 of Embodiment 1> The control unit 201 sets the area including the viewpoint position P10 as the first image area R1 from among multiple areas R11 to R15 that are formed by dividing the frame image 20 in advance in a direction intersecting the scanning line (Z-axis direction).

[0156] With this configuration, multiple regions R11 to R15 are prepared in advance, making it possible to smoothly set the first image region R1 that includes the viewpoint position P10. When the same process is performed on the specific region R3, it is possible to smoothly set the specific region R3 that includes the center of gravity of the moving object region R10.

[0157] In the first modification of the first embodiment, the input processing units 350 and 310 output two types of video signals, one for high resolution and one for low resolution. In contrast, in the second embodiment, the input processing units 350 and 310 output video signals for high gradation and one for low gradation.

[0158] FIG. 15 is a block diagram showing the configuration of a signal processing unit 300 according to the second embodiment.

[0159] The signal processing unit 300 in FIG. 15 is different from the first modification of the first embodiment in FIG. 10 in that the two types of video signals output from the input processing units 350 and 310 are different.

[0160] The input processing unit 350 outputs the first imaging video signal for high gradation output from the camera 13 as is to the first buffer 301, and outputs a second imaging video signal generated by reducing the gradation of the first imaging video signal for high gradation output from the camera 13 to the second buffer 302. The first imaging video signal output from the camera 13 is, for example, a video signal in which shading is expressed in 256 gradations, and the second imaging video signal generated by the gradation reduction by the input processing unit 350 is, for example, a two-gradation video signal in which the gradation is reduced to two using a threshold value. Both the first imaging video signal and the second imaging video signal are video signals for high resolution.

[0161] Input processing unit 310 outputs a first input video signal for high gradation output from an external device directly to first buffer 321, and outputs a second input video signal generated by reducing the gradation of the first input video signal for high gradation output from the external device to second buffer 322. The first input video signal output from the external device is, for example, a video signal expressed in 256 gradations, and the second input video signal generated by the gradation reduction by input processing unit 310 is, for example, a two-level video signal reduced to two gradations using a threshold value. Both the first input video signal and the second input video signal are video signals for high resolution.

[0162] The signal combining unit 330 combines the first captured video signal for high gradation stored in the first buffer 301 with the first input video signal for high gradation stored in the first buffer 321 to generate a first combined video signal for high gradation for one frame. The signal combining unit 330 also combines the second captured video signal for low gradation stored in the second buffer 302 with the second input video signal for low gradation stored in the second buffer 322 to generate a second combined video signal for low gradation for one frame.

[0163] FIG. 16 is a diagram schematically illustrating generation of a frame image 20 according to the second embodiment.

[0164] The control unit 201 sets a first image region R31 of a predetermined size that includes the viewpoint position P10. As in the first embodiment, the size of the first image region R1 may be set taking into consideration the range around the viewpoint that can usually be clearly recognized by a person.

[0165] Furthermore, the control unit 201 sets the specific region R33 so as to include at least a portion of the moving object region R10 detected by the moving object detection unit 13a. For example, the specific region R33 is set so that the center of gravity of the moving object region R10 is aligned with the center. As in the first embodiment, the specific region R33 is set smaller than the first image region R31. In the example of Fig. 16, the size of the specific region R33 is fixed to a specified size. If the moving object region R10 is smaller than the specified size of the specific region R33, the specific region R33 may be reduced to a size that inscribes the moving object region R10.

[0166] The size of the specific region R33 may also change depending on the size of the moving object region R10. In this case, as described in the first embodiment, it is preferable to set an upper limit to the change in size of the specific region R33. In this case, the upper limit may be set, for example, taking into account the range around the viewpoint that can be clearly recognized by a person.

[0167] As in the first embodiment, the control unit 201 selectively applies either the first composite video signal from the first frame buffer 341 or the second composite video signal from the second frame buffer 342 to the first image region R31 including the viewpoint position P10 and the specific region R33, depending on the distance to an object located in these regions. Furthermore, the control unit 201 applies the second composite video signal from the second frame buffer 342 to the second image region R32 other than the first image region R31 and the specific region R33.

[0168] FIG. 17 is a flowchart showing the process of generating a frame image 20 performed by the image generating device 3 according to the second embodiment.

[0169] In the process of Figure 17, steps S15 to S20 are replaced with steps S31 to S23 compared to the process of embodiment 1 shown in Figure 8. In the process of Figure 17, the second mirror 161 is rotated in the negative direction of the Z axis at a constant scanning speed. This scanning speed is the same as the scanning speed when a high-resolution image is displayed in embodiment 1. Below, the process that differs from Figure 8 will be described.

[0170] The control unit 201 determines whether the current scanning line is included only in the second image region R32 (S31).

[0171] If the current scan line is included only in the second image region R32 (S31: YES), the control unit 201 outputs the second composite video signal from the second frame buffer 342 to the line memory 221 (S32). As a result, one line of image is generated by the second composite video signal for low gradation.

[0172] On the other hand, if the current scanning line is included in the second image area R32 and / or the specific area R33 as well as the first image area R31 (S31: NO), the control unit 201 selectively outputs the first composite video signal of the first frame buffer 341 or the second composite video signal of the second frame buffer 342 to the line memory 221 depending on the positions of the first image area R31 and the specific area R33 (S33).

[0173] FIG. 18 is a flowchart showing the process of step S33 in FIG.

[0174] If the write target address on the line memory 221 is an address corresponding to the second image area R32 (S201: YES), the control unit 201 outputs the second composite video signal on the second frame buffer 342 corresponding to this address to the second frame buffer 342, and writes the second composite video signal to the write target address (S202).

[0175] Furthermore, if the write target address in line memory 221 is an address corresponding to first image region R31 (S201: NO, S202: YES), control unit 201 refers to the distance corresponding to viewpoint position P10 among the distances detected by distance detection unit 14, and determines whether this distance is equal to or less than a predetermined threshold (S203). This threshold is set in the same way as the threshold in step S16 of FIG. 8.

[0176] If the distance to the viewpoint position P10 is equal to or less than the threshold value (S203: YES), the control unit 201 causes the first composite video signal stored in the first frame buffer 341 corresponding to this address to be output to the first frame buffer 341, and writes the first composite video signal to the write target address (S205). On the other hand, if the distance to the viewpoint position P10 is greater than the threshold value (S203: NO), the control unit 201 causes the second composite video signal stored in the second frame buffer 342 corresponding to this address to be output to the second frame buffer 342, and writes the second composite video signal to the write target address (S202).

[0177] Furthermore, if the write target address in the line memory 221 is an address corresponding to the specific region R33 (S201: NO, S202: NO), the control unit 201 refers to the distance detected by the distance detection unit 14 that corresponds to the position of the specific region R3 (for example, the position of the center of gravity of the moving object region R10), and determines whether this distance is equal to or less than a predetermined threshold (S204). This threshold is also set in the same way as the threshold in step S16 of FIG. 8.

[0178] If the distance to the position of the specific region R3 is equal to or less than the threshold value (S204: YES), the control unit 201 causes the first composite video signal in the first frame buffer 341 corresponding to this address to be output to the first frame buffer 341, and writes the first composite video signal to the write target address (S205). On the other hand, if the distance to the position of the specific region R3 is greater than the threshold value (S204: NO), the control unit 201 causes the second composite video signal in the second frame buffer 342 corresponding to this address to be output to the second frame buffer 342, and writes the second composite video signal to the write target address (S202).

[0179] As a result, the image of the first image region R31 and the specific region R33 of one line of image is generated by applying the composite video signal for high gradation or low gradation depending on the distance, and the image of the second image region R32 of one line of image is generated by the second composite video signal for low gradation.

[0180] Through this processing, an image with gradation adjusted according to distance is displayed in the first image region R31 near the viewpoint position P10 and in the specific region R33 to which the gaze is likely to be directed, and a low-gradation image is displayed in the second image region R32 other than the first image region R31 and the specific region R33.

[0181] Effect of Embodiment 2 As in Embodiment 1, the image displayed near the line of sight (first image region R1) has lower resolution (lower gradation) as the distance to the object in front of the line of sight increases. Therefore, the resolution of the image displayed in the region near the line of sight (first image region R1) can be made closer to the actual resolution when viewing the object in front of the line of sight.

[0182] In addition, the resolution of the image displayed in the specific area R3 can be made to approach the resolution when actually viewing an object located in the specific area R3, thereby reducing the sense of incongruity felt when the user shifts their gaze to the specific area R3 and looks at the specific area R3.

[0183] 15 and 16, the image definition adjusted in the first image region R1 and the specific region R3 is the gradation that defines the luminance resolution of the video signal. Therefore, the image definition can be adjusted by simply adjusting the gradation while keeping the scanning speed of the scanning unit (second scanning unit 160) constant.

[0184] The camera 13 outputs a first imaging video signal corresponding to a high gradation (first gradation), and the imaging processing unit 230 has an input processing unit 350 that performs a gradation reduction process on the high-gradation first imaging video signal to generate a second imaging video signal corresponding to a low gradation (second gradation).

[0185] According to this configuration, high-gradation and low-gradation video signals can be generated smoothly.

[0186] The input processing unit 350 may be integral with the camera 13. The video signals from the external device may be a first input video signal for high gradation and a second input video signal for low gradation. In this case, the input processing unit 310 is omitted.

[0187] Also, similar to Modification 2 of Embodiment 1, the configuration for processing video signals from an external device, i.e., the input processing unit 310, the first buffer 321, the second buffer 322, and the signal synthesis unit 330, may be omitted from the configuration of Figure 15.

[0188] Third Embodiment In the first and second embodiments, the moving object region R10 is exemplified as a region to which the gaze is likely to be directed, but the region to which the gaze is likely to be directed is not limited to this.

[0189] For example, a region of interest that the user is likely to pay attention to may be set in advance as a region to which the user's gaze is likely to be directed. In this case, the control unit 201 sets a specific region to include the region of interest. The region of interest is, for example, near the bottom edge of the frame image 20. When walking while wearing the AR glasses 1, the user is particularly concerned about whether there are any obstacles or the like under their feet. For this reason, the user occasionally shifts their gaze to their feet while looking at the image of the scenery ahead.

[0190] 19(a) and 19(b) are diagrams schematically showing the generation of a frame image 20 according to embodiment 3. Fig. 19(a) shows a scanning configuration in which a specific region R3 is set near the bottom edge of the frame image 20 in the configuration of embodiment 1, and Fig. 19(b) shows a scanning configuration in which a specific region R3 is set near the bottom edge of the frame image 20 in the configuration of embodiment 2. In this case, the region near the bottom edge of the frame image 20 is the region of interest, and this region of interest is set as is as the specific region R3.

[0191] In the example of Fig. 19(a), a specific region R3 is set near the bottom edge of the frame image 20, and therefore, near this bottom edge, the resolution is adjusted according to the distance to the object, as in the above-described embodiment 1. In the example of Fig. 19(b), a specific region R33 is set near the bottom edge of the frame image 20, and therefore, near this bottom edge, the gradation is adjusted according to the distance to the object, as in the above-described embodiment 2.

[0192] Advantages of Third Embodiment As described above, in the third embodiment, a region of interest (here, near the bottom edge of the frame image 20) that the user is likely to focus their attention on is set in advance in the control unit 201, and the control unit 201 sets specific regions R3 and R33 so as to include the region of interest. This allows the user to immediately view an image in which the resolution has been adjusted according to the distance when they move their gaze to the region of interest. In the examples of Figures 19(a) and 19(b), if the user occasionally moves their gaze to the ground while looking at an image of the scenery ahead, they can immediately view an image in which the resolution of the area near the ground has been adjusted.

[0193] In the examples of FIGS. 19A and 19B, since there is no need to detect the moving object region R10, the moving object detector 13a is omitted from the configuration of FIG.

[0194] 19(a) and 19(b) , regions such as traffic lights and traffic signs may be set as regions of interest. In this case, the control unit 201 is provided with an image analysis function for extracting regions such as traffic lights and traffic signs from the video signal from the camera 13. The control unit 201 uses this function to identify these regions of interest on the frame image 20 and set specific regions R3 and R33 so as to include at least a portion of these regions. For example, the control unit 201 sets the specific regions R3 and R33 so that the centers of gravity of the identified regions of interest are aligned with the centers of the specific regions R3 and R33.

[0195] In the third embodiment, the frame image generation process is similar to the process shown in Fig. 8 or 17. In the third embodiment, the specific regions R3 and R33 are preferably set smaller than the first image region R1.

[0196] <Fourth Embodiment> In the above-described first to third embodiments, a frame image 20 captured by the camera 13 is displayed. In contrast, in the fourth embodiment, the camera 13 is omitted, and only an image based on an input video signal input from an external device is displayed. In the fourth embodiment, the frame 2 shown in Fig. 1 is made of a transparent material. The user views the scenery ahead with the naked eye through the front surface 2a of the frame 2.

[0197] As in the above embodiments 1 to 3, the control unit 201 controls the first scanning unit 140 and the second scanning unit 160 so that the entire frame is scanned, and causes the light sources 101 to 103 to emit light according to the display image during the scanning period of the region R41 of the display image.

[0198] FIG. 20 is a block diagram showing the configuration of the projection unit 11 and the line-of-sight detection unit 12 according to the fourth embodiment.

[0199] As shown in Fig. 20, in the fourth embodiment, the camera 13 is omitted from the configuration of Fig. 3. Furthermore, an input video signal input from an external device is input to the signal processing unit 300 as well as the control unit 201. The other configurations are the same as those of Fig. 3.

[0200] The control unit 201 extracts a display image area on the image area (frame) from an input video signal from an external device. As in the first embodiment, the input video signal is, for example, a video signal that displays a CG image in a predetermined area on the image area (frame) and does not display an image in the other area (does not cause the light sources 101 to 103 to emit light). In this case, the control unit 201 extracts the CG image area on the frame as the display image area.

[0201] FIG. 21 is a block diagram showing the configuration of a signal processing unit 300 according to the fourth embodiment.

[0202] 21 omits the imaging processing unit 230, first buffer 301, second buffer 302, and signal combining unit 330 from the signal processing unit 300 in FIG. 4 . The first input video signal from the input processing unit 310 is temporarily stored in the first buffer 321, and then one frame of the first input video signal is output to the first frame buffer 341. The second input video signal from the input processing unit 310 is temporarily stored in the second buffer 322, and then one frame of the second input video signal is output to the second frame buffer 342.

[0203] FIG. 22 is a flowchart showing a frame image generation process performed by the image generation device 3 according to the fourth embodiment.

[0204] 22 differs from the processing of embodiment 1 shown in FIG. 8 in that steps S13 to S16 are replaced with steps S41 to S44. Furthermore, since the first frame buffer 341 and the second frame buffer 342 store the first input video signal and the second input video signal, respectively, the signals targeted in steps S11, S17, and S19 are the first input video signal and the second input video signal. The processing that differs from FIG. 8 will be described below.

[0205] The control unit 201 determines whether the area of ​​the display image, such as a CG image, includes the viewpoint position P10 (S41). If the area of ​​the display image does not include the viewpoint position P10 (S41: NO), the control unit 201 proceeds to step S19. If the area of ​​the display image includes the viewpoint position P10 (S41: YES), the control unit 201 determines whether the distance detected by the distance detection unit 14 for the viewpoint position P10, i.e., the distance corresponding to the area of ​​the display image, is equal to or less than a predetermined threshold (S42). The threshold is set to approximately several meters, as described above. If the distance corresponding to the area of ​​the display image is greater than the predetermined threshold (S42: NO), the control unit 201 proceeds to step S19.

[0206] If the distance corresponding to the region of the display image is equal to or smaller than a predetermined threshold (S42: NO), the control unit 201 sets a region that includes at least a portion of the region of the display image and has a predetermined width in the Z-axis direction as the high-definition region (S43). The high-definition region is set so that its midpoint in the Z-axis direction coincides with the center of gravity of the region of the display image. As in Modification 3 of the first embodiment, the high-definition region may be selectively set from a pre-fixed region. The width of the high-definition region in the Z-axis direction may usually be set taking into account the width of the Z-axis range around the viewpoint that can be clearly recognized by a human.

[0207] The control unit 201 determines whether the current scanning line is within the high-definition area (S44).

[0208] If the current scanning line is within the high-definition region (S44: YES), the control unit 201 proceeds to step S17. If the current scanning line is not within the high-definition region (S44: NO), the control unit 201 proceeds to step S19. When the process proceeds to step S17, one line of image is generated using the first input video signal from the first frame buffer 341. In parallel with this, the control unit 201 controls the second mirror drive circuit 212 so that the second mirror 161 rotates at the first scanning speed (S18). This results in a high-resolution image being displayed.

[0209] On the other hand, when the process proceeds to step S19, one line of image is generated by the second video signal from the second frame buffer 342. In parallel with this, the control unit 201 controls the second mirror drive circuit 212 so that the second mirror 161 rotates at a second scanning speed that is faster than the first scanning speed (S20). As a result, a low-resolution image is displayed.

[0210] 23A and 23B are diagrams showing an example of the operation of the frame image generation process.

[0211] 23A, if region R41 of the display image does not include viewpoint P10, no high-resolution region is set. In this case, the spacing between scanning lines is widened in all regions, including region R41 of the display image. As a result, a low-resolution image is displayed in region R41 of the display image.

[0212] 23B, when region R41 of the display image includes viewpoint P10 and the distance acquired for viewpoint P10 is equal to or less than a threshold, high-definition region R42 is set. In this case, the spacing between scanning lines is narrowed in high-definition region R42. As a result, a high-resolution image is displayed in region R41 of the display image.

[0213] If region R41 of the display image includes viewpoint P10, but the distance acquired for viewpoint P10 is greater than the threshold, no high-resolution region is set, as in FIG. 23A. In this case, the spacing between scanning lines is also increased in all regions, including region R41 of the display image. As a result, a low-resolution image is displayed in region R41 of the display image.

[0214] 23B, the control unit 201 applies a video signal (second input video signal) from a frame buffer (second frame buffer 342) with a lower definition (lower resolution) to a region R41 of the display image that includes a viewpoint position P10 corresponding to the line of sight, the greater the distance to the object corresponding to the viewpoint position P10. This makes it possible to bring the definition of the image displayed in the region near the line of sight (region R41 of the display image) closer to the actual definition when viewing an object in front of the line of sight.

[0215] <Embodiment 5> In embodiment 4, the level of detail that is adjusted according to the distance to an object in the line of sight is the resolution, but in embodiment 5, the level of detail that is adjusted according to the distance to an object in the line of sight is the gradation.

[0216] FIG. 24 is a block diagram showing the configuration of a signal processing unit 300 according to the fifth embodiment.

[0217] 24, the image pickup processing unit 230, the first buffer 301, the second buffer 302, and the signal synthesis unit 330 are omitted from the configuration of FIG.

[0218] Input processing unit 310 outputs the first input video signal with a high gradation to first buffer 321, and outputs the second input video signal with a low gradation to second buffer 322. First frame buffer 341 stores one frame's worth of the first input video signal with a high gradation at high resolution, and second frame buffer 342 stores one frame's worth of the 21 input video signals with a low gradation at high resolution.

[0219] FIG. 25 is a flowchart showing a frame image generation process performed by the image generation device 3 according to the fifth embodiment.

[0220] In the process of Fig. 25, steps S13, S14, and S31 are replaced with steps S51 to S53, as in the process of Fig. 22. Furthermore, the signals targeted in steps S11, S32, and S33 are the first input video signal and the second input video signal. The processes of steps S51 and S52 are the same as steps S41 and S42 of Fig. 22.

[0221] If the area of ​​the display image includes the viewpoint position P10 (S51: YES) and the distance detected by the distance detection unit 14 for the viewpoint position P10 is equal to or less than a predetermined threshold (S52: YES), the control unit 201 determines whether the scanning line is within the area of ​​the display image (S53). If the scanning line is within the area of ​​the display image (S53: YES), the control unit 201 proceeds to step S33. If the scanning line is not within the area of ​​the display image (S53: NO), the control unit 201 proceeds to step S32. If the determination in step S51 or step S52 is NO, the control unit 201 also proceeds to step S32.

[0222] In step S33, a high-gradation first input video signal is output from the first frame buffer 341 to a memory address in the line memory 221 corresponding to the area of ​​the display image, and a low-gradation second input video signal is output from the second frame buffer 342 to other memory addresses in the line memory 221.

[0223] 25, if region R41 of the display image does not include viewpoint P10 (S51: NO), all regions of the display image, including region R41, are set to low gradation. As a result, a low gradation image is displayed in region R41 of the display image.

[0224] Furthermore, if region R41 of the display image includes viewpoint P10 (S51: YES) and the distance acquired for viewpoint P10 is equal to or less than the threshold (S52: YES), region R41 of the display image is set to high gradation, and the other regions are set to low gradation, thereby displaying a high gradation image in region R41 of the display image.

[0225] If region R41 of the display image includes viewpoint P10 (YES at S51), but the distance acquired for viewpoint P10 is greater than the threshold (NO at S52), the entire region is set to low gradation, and a low-resolution image is displayed in region R41 of the display image.

[0226] Effect of Fifth Embodiment The control unit 201 applies an input video signal (second input video signal) from a frame buffer (second frame buffer 342) with a lower resolution (lower gradation) to a region R41 of the display image that includes a viewpoint position P10 corresponding to the line of sight, the greater the distance to the object corresponding to the viewpoint position P10. As a result, when the region R41 of the display image is near the line of sight, the resolution of the image of the input video signal displayed in the region of the display image can be made closer to the actual resolution when viewing an object in front of the line of sight.

[0227] <Other Modifications> In the above-described first and second embodiments, the camera 13 used to create the frame image for display has the functionality of the moving object detection unit 13a, but a camera having the functionality of the moving object detection unit 13a may be provided separately from the camera 13.

[0228] In this case, the other camera is placed close to camera 13 so that its field of view substantially overlaps the field of view of camera 13, and is pointed in the same direction as camera 13. As a result, the position on the frame image of the moving object area detected by the moving object detection unit of the other camera becomes substantially the same as the position of the moving object area on the frame image of camera 13, and the position of the former can be used as the position of the latter. If there is a discrepancy between these two positions, the former position may be converted into the latter position, and the latter position may be obtained from the former position.

[0229] The control unit 201 acquires a moving object region R10 on the frame image 20 of the camera 13 from the moving object region detected by the other camera. Then, the control unit 201 sets specific regions R3, R33 so as to include at least a portion of the moving object region R10, as in the above-described first and second embodiments. The frame image generation process is similar to the process in FIG. 8 or FIG. 17 .

[0230] According to this modification, the function of the moving object detection unit is assigned to another camera, so that the camera 13 only needs to capture an image for display and output it to the signal processing unit 300. Therefore, the video signal for display can be smoothly output to the signal processing unit 300.

[0231] Furthermore, in the above embodiments 1 to 5, the first frame buffer 341 for high definition and the second frame buffer 342 for low definition are arranged in the signal processing unit 300, but the number of frame buffers that hold video signals of different definitions is not limited to this, and three or more frame buffers may be arranged in the signal processing unit 300.

[0232] For example, as shown in FIG. 26, three frame buffers 371 to 373 may be arranged in the signal processing unit 300, each of which holds three types of composite video signals: high definition, medium definition, and low definition.

[0233] The input processing unit 350 outputs three types of captured video signals: high definition, medium definition, and low definition.

[0234] When definition is defined by resolution as in the first embodiment, for example, high definition (high resolution) is the resolution with the number of lines on the left side of Fig. 11, medium definition (medium resolution) is the resolution with the number of lines on the right side of Fig. 11, and low definition (low resolution) is the resolution obtained by further thinning out every other line from the resolution on the right side of Fig. 11. The medium resolution here corresponds to the low resolution in the case of Fig. 10.

[0235] Furthermore, when the definition is defined by the gradation as in the second embodiment, for example, high definition (high gradation) is 256 gradations, which are the gradations of the video signal output from the camera 13, medium definition (medium gradation) is 64 gradations, which is ¼ of 256 gradations, and low definition (low gradation) is 2 gradations. The low gradation here corresponds to the low gradation in the second embodiment.

[0236] Three buffers 351 to 353 for holding the image pickup video signals of the three different resolutions are arranged in front of the signal synthesis unit 330 .

[0237] The input processing unit 310 outputs three types of input video signals: high definition, medium definition, and low definition. These three types of definition are the same as the three types of captured video signals output by the input processing unit 350. Three buffers 361 to 363 that hold the input video signals of these definitions, respectively, are arranged before the signal combining unit 330.

[0238] The signal synthesis unit 330 acquires the captured video signal and the input video signal of the same resolution from the corresponding buffers, synthesizes the acquired video signals, and outputs the generated synthesized video signal to the frame buffer of the corresponding resolution among the three subsequent frame buffers 371 to 373.

[0239] The control unit 201 applies a lower-resolution video signal from the frame buffer to the first image region R1 and the specific region R3 as the distance detected by the distance detection unit 14 increases. In this case, two thresholds are set. When the distance detected by the distance detection unit 14 for the first image region R1 and the specific region R3 is equal to or less than a smaller first threshold, the control unit 201 applies the composite video signal from the frame buffer 371 to the first image region R1 and the specific region R3. When this distance is greater than the first threshold and equal to or less than a larger second threshold, the control unit 201 applies the composite video signal from the frame buffer 372 to the first image region R1 and the specific region R3. When this distance is greater than the second threshold, the control unit 201 applies the composite video signal from the frame buffer 373 to the first image region R1 and the specific region R3.

[0240] This configuration allows for more precise adjustment of the resolution depending on the distance, so that the resolution of the images displayed in the first image region R1 and the specific region R3 can be made closer to the actual resolution when viewing an object beyond these regions.

[0241] 26, the processing units for displaying an image based on an input video signal, that is, input processing unit 310, buffers 361 to 363, and signal synthesis unit 330, may be omitted, as in the configuration of Fig. 12. Furthermore, similar configurations and processes may be applied to the configurations of embodiments 4 and 5.

[0242] When four or more frame buffers are used, thresholds are set that are one less than the number of frame buffers, and the frame buffer that outputs the video signal to the line buffer is selectively determined based on the relationship between the distance and each threshold.

[0243] Furthermore, in the above-described first and second embodiments, the region of a moving object moving in the vertical and horizontal directions (directions parallel to the X-Y plane) on the frame image 20 is detected as the moving object region R10, but the region of a moving object moving in the front-to-back direction (Z-axis direction) may also be detected as the moving object region R10. In this case, the region of a moving object moving in the front-to-back direction (Z-axis direction) may be extracted based on the change in distance detected by the distance detection unit 14.

[0244] In the above-described second embodiment, the gradation of the composite video signal is changed between the first image region R1 and the specific region R3 and the second image region R2, but the resolution may be changed between the first image region R1 and the specific region R3 and the second image region R2, as in the first embodiment. Furthermore, the configuration of the third embodiment may be added to the configurations of the first and second embodiments to adjust the definition of both the specific region including the moving object region and the specific region including the region of interest.

[0245] In the first modification of the first embodiment, the input processing unit 350 performs thinning and mixing processes on the input first captured video signal for high resolution to generate the second captured video signal for low resolution, as shown in Fig. 11. However, when the second captured video signal for low resolution is input from the camera 13, the input processing unit 350 may perform interpolation processes on the input second captured video signal for low resolution to generate the first captured video signal for high resolution. However, because the interpolation process requires a higher processing load than the thinning and mixing processes, it is preferable that the video signal input to the input processing unit 350 be the first captured video signal for high resolution.

[0246] In the second embodiment, the first image region R31 and the second image region R32 are set based on the viewpoint position P10, but this is not limiting, and the first image region R1 and the second image region R2 may be set based on the viewpoint position P10, as in the first embodiment shown in Fig. 7B. Furthermore, in the second embodiment, a plurality of regions R11 to R15 prepared in advance may be set based on the viewpoint position P10, as in the third modification of the first embodiment.

[0247] In the second embodiment, the input processing units 350 and 310 performed processing to reduce the number of gradations of the video signal to two gradations, but this is not limited to this, and processing to reduce the video signal to a gradation other than two gradations (for example, 16 gradations) may also be performed.

[0248] In the fourth and fifth embodiments, the entire frame is scanned, and light sources 101 to 103 emit light in accordance with the display image during the scanning period of region R41 of the display image, but only region R41 of the display image may be scanned. In this case, the resolution may be adjusted in accordance with the distance obtained for region R41 of the display image.

[0249] Furthermore, the distance detection unit 14 only needs to have its field of view set so that, at least when the gaze is directed at any position on the image generated by the image generation unit 3, it can detect the distance to an object in the line of sight at these positions.

[0250] In the above embodiment and modified example, the field of view of the camera 13 is in front of the AR glasses 1, but this is not limited thereto and may be above, below, or behind the AR glasses 1.

[0251] In the above embodiment and modified example, two sets of image generating devices 3 and mirrors 4 are provided in the AR glasses 1 to correspond to the user's pair of eyes E, but only one set may be provided in the AR glasses 1 to correspond to only one of the user's eyes E.

[0252] In the above embodiment and modified examples, the light scanned by the first scanning unit 140 and the second scanning unit 160 is directed to the user's eye E via the mirror 4, but this is not limiting and the light may be directed to the user's eye E via an optical system other than a mirror (for example, a lens, etc.). In this case, the optical system may be, for example, a combination of multiple mirrors, a combination of a mirror and a lens, or a combination of multiple lenses.

[0253] In the above embodiment and modified examples, the first mirror 141 and the second mirror 161 are provided separately, but instead of the first mirror 141 and the second mirror 161, a single mirror that rotates about two axes may be provided.

[0254] Furthermore, the configurations of the image generating device 3 and the AR glasses 1 (head-mounted display) can be modified in various ways in addition to the configurations shown in the above-described embodiment and modified examples. Various modifications are possible as appropriate within the scope of the technical ideas shown in the scope of claims.

[0255] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0256] (Technology 1) An image generating device comprising: a light source that emits light for generating an image; a scanning unit that scans the light emitted from the light source; a plurality of frame buffers that respectively store a plurality of types of video signals for composing the image with different resolutions; a gaze detection unit that detects the line of sight of a user; a distance detection unit that detects the distance to an object included in the field of view; and a control unit that controls the light source, the scanning unit, and the plurality of frame buffers to generate an image, wherein the control unit applies the video signal from the frame buffer with lower resolution to an image area including a viewpoint position corresponding to the gaze, the greater the distance to the object corresponding to the viewpoint position.

[0257] According to this technology, a video signal from a frame buffer with a lower resolution is applied to an image area including a viewpoint position corresponding to the line of sight, the greater the distance to an object corresponding to this viewpoint position. As a result, an image is displayed near the line of sight with a lower resolution the greater the distance to an object ahead of the line of sight. This allows the resolution of the image displayed in the area near the line of sight to approach the actual resolution when viewing the object ahead of the line of sight. Furthermore, since multiple video signals for displaying images with different resolutions are pre-stored in multiple frame buffers, the control unit only needs to select a video signal from one of the frame buffers in response to movement of the line of sight. This allows images to be smoothly displayed in the area near the line of sight with a resolution according to the distance.

[0258] (Technology 2) The image generating device according to Technology 1 further comprises an imaging processing unit that images the field of view and outputs a plurality of types of imaged video signals for constructing frame images with different resolutions, the plurality of types of video signals being constructed using the plurality of types of imaged video signals, and the image area being a first image area of ​​a predetermined size on the frame image that includes the viewpoint position.

[0259] This technology allows a user to view an image of the field of view in front of the user. The definition (resolution) of the image area near the user's line of sight can be adjusted according to the distance to the object in front of the user's line of sight. Therefore, the definition of the first image area near the line of sight on the frame image can be made closer to the actual resolution when viewing the object in front of the user's line of sight.

[0260] (Technology 3) In the image generating device described in Technology 2, the control unit applies the video signal from the frame buffer corresponding to a predetermined resolution other than the highest resolution to a second image area other than the first image area on the frame image.

[0261] According to this technique, the definition of the second image area other than the area near the line of sight is set low, thereby reducing eye fatigue of the user.

[0262] (Technology 4) In the image generating device described in Technology 2 or 3, the control unit sets a specific area on the frame image that includes at least a part of an area to which the line of sight is likely to be directed, and applies to the specific area the video signal from the frame buffer with a lower resolution the greater the distance to the object corresponding to the position of the specific area.

[0263] This technology makes it possible to bring the resolution of the image displayed in a specific area closer to the resolution when actually viewing an object located in the specific area, thereby reducing the sense of discomfort felt when the user shifts their gaze to the specific area and looks at it.

[0264] (Technology 5) The image generating device according to Technology 4, further comprising a moving object detection unit that detects a moving object area on the frame image, and the control unit sets the specific area as an area to which the gaze is likely to be directed, based on the detected moving object area.

[0265] When a user sees movement in a position other than the user's line of sight, the user is likely to shift their gaze to that position. According to Technology 5, the specific region is set to include at least a portion of the moving object region, so the resolution of the moving object region to which the user is likely to be next directed can be adjusted in advance according to the distance to the object. Therefore, when the user's gaze is actually directed to the moving object region in response to the object's movement, an image with a resolution close to the user's perception according to the distance can be displayed in that region.

[0266] (Technology 6) In the image generating device described in Technology 4, an area of ​​interest that a user is likely to pay attention to is set in advance, and the control unit sets the specific area so as to include the area of ​​interest.

[0267] This technology allows a user to immediately see an image with definition adjusted according to distance when they move their line of sight to a region of interest.

[0268] (Technology 7) In the image generating device according to any one of Techniques 4 to 6, the control unit sets the specific area to be smaller than the first image area.

[0269] According to this technology, when a high-resolution image is displayed in the first image area and the specific area, the first image area, which the user's gaze is directed at, is primarily made high-resolution, while the specific area, which the gaze is likely to be directed at next, is secondarily made high-resolution, and the proportion of the area in which the high-resolution image is displayed relative to the frame image can be appropriately reduced. Therefore, the high-resolution image can be displayed smoothly and quickly at the position of the user's gaze by following changes in the user's gaze while appropriately reducing eye fatigue.

[0270] (Technology 8) The image generating device according to Technology 1 further comprises an input processing unit that outputs a plurality of types of input video signals for constructing display images with different resolutions based on video signals from an external device, the plurality of types of video signals being constructed using the plurality of types of input video signals, and the image area being an area of ​​the display image.

[0271] According to this technology, for an area of ​​a display image including a viewpoint position corresponding to the line of sight, the greater the distance to an object corresponding to the viewpoint position, the lower the resolution of the input video signal from the frame buffer that is applied. As a result, when there is an area of ​​the display image near the line of sight, the resolution of the image of the input video signal displayed in the area of ​​the display image can be made closer to the actual resolution when viewing an object in front of the line of sight.

[0272] (Technology 9) In the image generating device according to any one of techniques 1 to 8, the definition is a resolution of the image that is determined by a scanning speed of the scanning unit.

[0273] According to this technique, the definition of the image can be adjusted by controlling the scanning speed of the scanning unit to adjust the density of the scanning lines.

[0274] (Technology 10) In the image generating device according to any one of techniques 1 to 9, the definition is a gradation that defines a luminance resolution of a video signal.

[0275] According to this technology, the image definition can be adjusted simply by adjusting the gradation while keeping the scanning speed of the scanning unit constant.

[0276] (Technology 11) A head-mounted display comprising: an image generating device according to any one of technologies 1 to 10; a frame for holding the image generating device; and an optical system for directing light from the image generating device to the eyes of the user wearing the head-mounted display on their head.

[0277] This technique provides the same effects as technique 1. Furthermore, by wearing the head-mounted display on the user's head, the user can view the image generated by the image generation device.

[0278] 1 AR glasses (head mounted display) 2 Frame 3 Image generating device 4 Mirror (optical system) 12 Line of sight detection unit 13 Camera 13a Moving object detection unit 14 Distance detection unit 20 Frame image 101, 102, 103 Light source 140 First scanning unit (scanning unit) 160 Second scanning unit (scanning unit) 201 Control unit 230 Image capturing processing unit 341 First frame buffer 342 Second frame buffer 371 to 373 Frame buffer P10 Viewpoint position R10 Moving object area R1, R31 First image area R11 to R15 Area (first image area) R2, R32 Second image area R3, R33 Specific area R41 Area of ​​displayed image

Claims

1. An image generating device comprising: a light source that emits light for generating an image; a scanning unit that scans the light emitted from the light source; a plurality of frame buffers that each store a plurality of types of video signals for composing the image with different resolutions; a gaze detection unit that detects the user's gaze; a distance detection unit that detects the distance to an object included in the field of view; and a control unit that controls the light source, the scanning unit and the plurality of frame buffers to generate an image, wherein the control unit applies the video signal from the frame buffer with a lower resolution to an image area including a viewpoint position corresponding to the gaze, the greater the distance to the object corresponding to the viewpoint position.

2. An image generating device as described in claim 1, further comprising an imaging processing unit which images the field of view and outputs a plurality of types of captured video signals for constructing frame images with different resolutions, the plurality of types of video signals being constructed using the plurality of types of captured video signals, and the image area being a first image area of ​​a predetermined size on the frame image that includes the viewpoint position.

3. An image generating device as described in claim 2, characterized in that the control unit applies the video signal from the frame buffer corresponding to a predetermined resolution other than the highest resolution to a second image area other than the first image area on the frame image.

4. An image generating device as described in claim 2, characterized in that the control unit sets a specific area on the frame image including at least a part of the area toward which the line of sight is likely to be directed, and applies to the specific area the video signal from the frame buffer with a lower resolution the greater the distance to the object corresponding to the position of the specific area.

5. An image generating device as described in claim 4, further comprising a motion detection unit that detects a moving object area moving on the frame image, and the control unit sets the specific area as an area to which the gaze is likely to be directed, the detected moving object area.

6. An image generating device according to claim 4, characterized in that an area of ​​interest that a user is likely to focus on is set in advance, and the control unit sets the specific area so as to include the area of ​​interest.

7. An image generating device according to claim 4, wherein the control unit sets the specific area to be smaller than the first image area.

8. An image generating device according to claim 1, further comprising an input processing section which outputs a plurality of types of input video signals for constructing display images with different definitions based on a video signal from an external device, the plurality of types of video signals being constructed using the plurality of types of input video signals, and the image area being an area of ​​the display image.

9. An image generating device according to claim 1, characterized in that the definition is a resolution of the image defined by the scanning speed of the scanning section.

10. An image generating device according to claim 1, wherein the definition is a gradation that defines the luminance resolution of a video signal.

11. A head-mounted display comprising: an image generating device as claimed in any one of claims 1 to 9; a frame for holding the image generating device; and an optical system for directing light scanned by the scanning unit to the eyes of a user wearing the head-mounted display on his or her head.

Images