Image processing apparatus, method for controlling image processing apparatus, and program

Abstract

To perform visual correction in an image processing apparatus that transmits outside light.SOLUTION: An image processing apparatus can transmit outside light and allow a user to visually recognize the light, and comprises: imaging means that picks up an image of the outside; area extraction means that extracts a specific area from an outside image acquired through the imaging performed by the imaging means; virtual image creation means that creates a virtual image for correcting the user's vision in the specific area; and virtual image display means that displays the virtual image superimposed on an area of the transmitted outside light corresponding to the specific area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device that transmits external light. [Background technology]

[0002] There have been various devices to assist vision in the past. In recent years, Augmented Reality (hereinafter referred to as AR) technology, which displays a virtual space superimposed on real space, has been attracting attention. Some AR devices are realized by optical see-through head-mounted displays (hereinafter referred to as HMDs), which transmit external light.

[0003] Furthermore, as a device for assisting vision, for example, color vision correction glasses are available for users with color vision disabilities. Color vision correction glasses correct color vision by blocking light other than that for which the user is less sensitive (e.g., green and blue) to the extent that light is similar to that for which the user is less sensitive (e.g., red). Patent Document 1 discloses a display device that receives information about the user's color weakness characteristics and controls light-emitting elements using generated correction data. Furthermore, even users who are not color vision disabled may have difficulty visually recognizing things at night when there is no light source. There are night vision devices that collect and display ambient light such as infrared light in dark environments such as at night when there is no light source. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-57621 [Patent Document 2] Japanese Patent Application Laid-Open No. 2016-156934 [Non-patent literature]

[0005] [Non-Patent Document 1] S. Haykin, “Neural Networks A Comprehensive Foundation 2nd Edition”, Prentice Hall, pp.156-255, July 1998 Summary of the Invention [Problem to be solved by the invention]

[0006] However, color vision correction glasses block light according to the colors to which the user has low sensitivity, and depending on the low sensitivity, the entire glasses may be strongly blocked, resulting in a darkened field of view. Furthermore, the display device in Patent Document 1 is not a transmissive display device, and the technology disclosed in Patent Document 1 cannot be applied to optical see-through HMDs. Furthermore, while night vision devices can provide a certain field of view, they do not observe visible light, so color information may be lost, and in situations where light and dark are mixed, parts of the field of view may be blown out, resulting in reduced discrimination. Even in optical see-through HMDs that transmit external light, it is necessary to address the color vision of individuals with unusual color vision characteristics and the reduced discrimination caused by the loss of color information in night vision.

[0007] The present invention aims to perform vision correction in an image processing device that transmits external light. [Means for solving the problem]

[0008] In order to solve the above problems, the image processing device of the present invention is an image processing device that can transmit external light to allow a user to see it, and is equipped with an imaging means for imaging the external world, an area extraction means for extracting a specific area from an external world image acquired by imaging with the imaging means, a virtual image generation means for generating a virtual image to correct the user's vision of the specific area, and a virtual image display means for displaying the virtual image by superimposing it on an area of ​​the transmitted external light that corresponds to the specific area. [Effects of the Invention]

[0009] According to the present invention, vision correction can be performed in an image processing device that transmits external light. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating a configuration of an image processing device 100. [Figure 2] 2A to 2C are diagrams illustrating the configuration of an in-eye unit 101 in the first embodiment. [Figure 3] FIG. 2 is a diagram showing a pixel array of an imaging unit 209. [Figure 4] FIG. 2 is a diagram showing the configuration of a control unit 105. [Figure 5] FIG. 4 is a diagram illustrating the overall configuration of a CNN in an object detection unit 405. [Figure 6] FIG. 4 is a diagram illustrating a partial configuration of a CNN in an object detection unit 405. [Figure 7] 10 is a flowchart showing a light adjustment process. [Figure 8] 4A to 4C are diagrams illustrating an example of virtual image information generation in the first embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of application of color vision correction in the first embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of virtual image information generation in the second embodiment. [Figure 11] 10A and 10B are diagrams illustrating an example of application of night vision correction in the second embodiment. [Figure 12] FIG. 11 is a diagram illustrating an example of the overall configuration of a CNN in a subject detection unit 405 according to the third embodiment. [Figure 13] FIG. 10 is a diagram showing the configuration of an in-eye unit 101 in a third embodiment. [Figure 14] FIG. 12 is a diagram showing the configuration of an imaging unit 1206 in the third embodiment. [Figure 15] FIG. 11 is a diagram illustrating calculation of the amount of parallax in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] (First embodiment) In the first embodiment, as vision correction, dimming processing by color vision correction in an image processing device that transmits external light will be described. In the image processing device of this embodiment, three optical axes, namely, the optical axis for capturing external light, the optical axis for visualizing external light, and the optical axis for outputting virtual image light, are equally arranged. FIG. 1 is a diagram showing the configuration of an image processing device 100. The image processing device 100 is an optical see-through head-mounted display (HMD) that is worn on the head of a human body and transmits external light. The image processing device 100 has an optical see-through configuration that allows the user to view image light of a virtual image without interfering with the visibility of external light. Note that the image processing device 100 may be smart glasses that support the rendering of virtual objects in AR.

[0012] The image processing device 100 includes a front-of-eye unit 101, a shield 102, a frame 103, and a control unit 105. The image processing device 100 may also include an operation unit 104. The front-of-eye unit 101 includes a front-of-eye unit 101a and a front-of-eye unit 101b. The front-of-eye unit 101a and the front-of-eye unit 101b are respectively placed in front of the right eye and the left eye of the human body, allowing the user to have binocular vision. The front-of-eye units 101a and 101b have the same configuration except that they are placed differently for the right eye and the left eye. Details of the configuration of the front-of-eye units 101a and 101b will be described later using FIG. 2.

[0013] The shield 102 is a protective member that holds the in-eye units 101a and 101b at the front of the image processing device 100 and prevents damage and dirt to the units. The shield 102 is made of a transparent material. The frame 103 is a member for attaching the image processing device 100 to the user's head. The frame 103 is, for example, a frame-shaped head-attached portion, but the shape of the frame 103 is not important as long as it can be attached to the user's head in an appropriate manner.

[0014] The control unit 105 is a unit that controls the entire image processing device 100. While Fig. 1 shows an example in which the control unit 105 is enclosed inside the frame 103, the present invention is not limited to this. For example, the control unit 105 may be arranged outside the frame 103 and connected to each component via a cable or the like. Details of the configuration of the control unit 105 will be described later with reference to Fig. 4.

[0015] The operation unit 104 is a component that accepts operations from the user. The operation unit 104 is composed of multiple components arranged on the frame 103. The operation unit 104 may be configured in any way as long as it can properly accept operations from the user in various processes described below. For example, the operation unit 104 may be four physical selection buttons (up, down, left, and right) and one physical decision button that the user can operate with their fingers. Alternatively, the image processing device 100 may not be provided with the operation unit 104, and another terminal having the function of the operation unit 104 may communicate with and control the image processing device 100. The user can change various parameters of the entire image processing device 100 by operating the operation unit 104 while looking at menus, indicators, etc. displayed on the eye-front units 101a and 101b.

[0016] Next, the detailed configuration of the in-eye unit 101 (in-eye unit 101a and in-eye unit 101b) will be described. FIG. 2 is a diagram illustrating the configuration of the in-eye unit 101 in the first embodiment. FIG. 2 is a schematic diagram of a cross section of the in-eye unit 101 as seen from the temporal side of the human body, and shows the eyeball of the user of the image processing device 100. Eyeball 200 is the eyeball of the user. Iris and lens 201 are the iris and lens of the user. Retina 202 is the retina of the user.

[0017] The eye unit 101 includes a virtual image display unit 203, a virtual image adjustment unit 204, a light guide unit 205, a light path control unit 206, a light amount adjustment unit 207, an optical system 208, and an imaging unit 209. The virtual image display unit 203 is an output unit (light emitting unit) that emits virtual image light 211. The virtual image display unit 203 is, for example, a liquid crystal LCD. The luminous flux of the virtual image light 211 is linearly polarized and emitted from the virtual image display unit 203. The virtual image display unit 203 adjusts the light intensity by emitting light of at least one wavelength. The virtual image display unit 203 can emit virtual image light with sufficient intensity for performing vision correction, described below, within a range that is safe for the user's eyes. The virtual image adjustment unit 204 performs focus adjustment control of the luminous flux of the virtual image light 211 emitted from the virtual image display unit 203. The virtual image adjustment unit 204 is, for example, a focus lens.

[0018] The light guiding unit 205 is a light guide path having a plurality of decentered reflecting surfaces with a plurality of decentered curvatures. The light guiding unit 205 has a prism body that utilizes a plurality of internal reflections. The light path control unit 206 is disposed inside the light guiding unit 205 and has a decentered curvature. The reflecting surface of the light path control unit 206 facing the outside world is a half mirror with predetermined reflectance and transmittance. The reflecting surface of the light path control unit 206 facing the user's eyeball is a full mirror with high reflectance.

[0019] The light amount adjustment unit 207, the optical system 208, and the image capture unit 209 constitute an image capture unit for capturing an image of external light. The light amount adjustment unit 207 is a diaphragm that adjusts the amount of light incident on the image capture unit 209. The light amount adjustment unit 207 adjusts the amount of light that is incident on the image capture unit 209 via the optical system 208, of the external light that has been divided by the optical path control unit 206 and guided through the light guiding unit 205. The optical system 208 performs focus adjustment control when the external light forms an image on the image capture unit 209. The optical system 208 is, for example, a focus lens.

[0020] The imaging unit 209 is an imaging means for capturing an image of external light. The imaging unit 209 is, for example, a CMOS sensor. The imaging unit 209 has photoelectric conversion elements (light receiving elements) that photoelectrically convert the external light imaged by the optical system 208 into an electrical signal. The imaging unit 209 has, for example, m horizontal pixels and n vertical pixels of light receiving elements. The imaging unit 209 also serves as a measuring means as well as an imaging means. Two photoelectric conversion elements (light receiving regions) are arranged in each light receiving element of each pixel of the imaging unit 209. An image formed on the imaging unit 209 and photoelectrically converted is processed as an image signal (image data) by an image processing unit 401 (described later). An image of the imaging surface (captured image) can be obtained by adding the outputs of the two photoelectric conversion elements. Furthermore, two images with parallax (parallax images) corresponding to the outputs of the two photoelectric conversion elements can be obtained. In the following description of this embodiment, the captured image obtained by adding the outputs of the two photoelectric conversion elements will be referred to as image A+B, and the parallax images that are the outputs of the two photoelectric conversion elements will be referred to as image A and image B. Furthermore, it is desirable that the imaging unit 209 has high sensitivity and low noise so that it can capture images that enable subject detection by the subject detection unit 405 (described later) even in low-light conditions where it is difficult to identify a subject with the naked eye.

[0021] Here, the optical paths of the external light 210 and the virtual image light 211 controlled by the eye-front unit 101 will be described. First, the optical path of the external light 210 will be described taking as an example a case where the external light 210 enters the eye-front unit 101 via the optical path 212. After passing through the optical path 212, the external light 210 enters the light guiding unit 205 of the eye-front unit 101. The external light that entered the light guiding unit 205 is split into two beams, with a portion reflected and a portion transmitted by the optical path control unit 206, which is a half mirror. The external light reflected by the optical path control unit 206 is emitted from the light guiding unit 205 via the optical path 213. The external light emitted from the light guiding unit 205 has its light amount adjusted by the light amount adjustment unit 207, is refracted by the optical system 208, and forms an image on the imaging unit 209. Meanwhile, the external light that has passed through the optical path control unit 206 enters the user's eyeball 200 via the optical path 215. The user can see external light by refracting external light incident on the user's eyeball 200 through the iris and lens 201 and then forming an image on the retina 202. In this way, part of the external light is guided to the imaging section 209 in the front-of-eye unit 101 and imaged, and part of the external light passes through the front-of-eye unit 101 and is seen by the user.

[0022] Next, the optical path of virtual image light 211, which is dimmed to perform partial vision correction, will be described. Virtual image light 211 emitted by virtual image display unit 203 is subjected to focus adjustment control by virtual image adjustment unit 204 and then enters light guiding unit 205. The light that entered light guiding unit 205 passes through optical path 214, is reflected by optical path control unit 206, and enters user's eyeball 200 through optical path 215. At this time, the virtual image light reflected by optical path control unit 206 has the same optical axis as external light that has passed through optical path control unit 206, and enters user's eyeball 200 through optical path 215. The light that entered user's eyeball 200 is refracted by iris and crystalline lens 201 and then forms an image on retina 214, allowing the user to view a virtual image.

[0023] In this way, the external light 210 visible to the naked eye 200, the external light 210 captured by the imaging unit 209, and the virtual image light 211 displayed by the virtual image display unit 203 all have the same optical axis, resulting in a parallax-free structure. Therefore, this configuration is suitable for calculating the correction area, which will be described later. Note that the configuration shown in FIG. 2 described in this embodiment is only an example, and any optical see-through device may be used as long as it is capable of superimposing the external light and the virtual image light with high precision, and is not limited to the above-described configuration.

[0024] Next, the pixel structure of the imaging unit 209 will be described. FIG. 3 is a diagram showing the pixel structure of the imaging unit 209. FIG. 3(A) is a diagram showing the pixel array of the imaging unit 209. The imaging unit 209 has pixels 310 arranged two-dimensionally. FIG. 3(A) shows a pixel range of 4 rows x 4 columns of the pixels 310 of the imaging unit 209. The array of the pixels 310 is, for example, a Bayer array. In the Bayer array, pixel 310G having a spectral sensitivity of G (green) is arranged as two pixels in the diagonal direction. Furthermore, pixel 310R having a spectral sensitivity of R (red) and pixel 310B having a spectral sensitivity of B (blue) are arranged as the other two pixels. In FIG. 3(A), pixel 310G is shown in white, pixel 310R is shown in gray, and pixel 310B is shown with diagonal lines.

[0025] Each pixel 310, represented by a square, has sub-pixels (sub-pixels 311a and 311b) corresponding to two photoelectric conversion elements for pupil division, represented by rectangles. The sub-pixel 311a is a first pixel that receives a light beam that has passed through a first pupil-intensive region of the imaging optical system. The sub-pixel 311b is a second pixel that receives a light beam that has passed through a second pupil region of the imaging optical system. The first pixel corresponds to image A. The second pixel corresponds to image B. Each pixel functions as an imaging pixel and a focus detection pixel.

[0026] In Figure 3(A), the plane parallel to the paper surface is the XY plane, and the axis perpendicular to the paper surface is the Z axis. The pixels 310 are arranged two-dimensionally on the XY plane. The Z axis is parallel to the imaging optical axis of the imaging unit 209, and the direction toward the sky on the paper surface is the positive direction. The sub-pixels 311a and 311b are arranged along the X axis direction.

[0027] FIG. 3B is a cross-sectional view of pixel 310G taken along the ZX plane. In FIG. 3B, the XZ plane is parallel to the paper, and the Y axis is perpendicular to the paper. Each of subpixels 311a and 311b has an independent pn junction photodiode, consisting of a p-type layer 320 and two divided n-type layers. If necessary, an intrinsic layer may be sandwiched between them to form a pin structure photodiode. A microlens 322 is disposed at a predetermined distance from the light receiving surface 311 in the positive direction of the Z axis. The microlens 322 is formed on a color filter 323. One microlens 322 is disposed on the front side (light incident side) of the two photoelectric conversion units (subpixels 311a and 311b) that make up pixel 310. The area sharing one microlens 322 constitutes one pixel.

[0028] Light incident on pixel 310 is collected by microlens 322 and further dispersed by color filter 323, after which it is received by subpixel 311a and subpixel 311b, respectively. In each subpixel, pairs of electrons and holes (positive holes) are generated according to the amount of light received, and after they are separated by a depletion layer, the electrons are accumulated. Meanwhile, the holes are discharged to the outside of the image sensor through a p-type layer connected to a constant voltage source. The electrons accumulated in subpixel 311a and subpixel 311b are transferred to a capacitance section (FD) via a transfer gate and converted into a voltage signal.

[0029] In this embodiment, pupil-division sub-pixels 311a and 311b are provided in all pixels 310R, 310G, and 310B of the imaging unit 209. The sub-pixels 311a and 311b are used as focus detection pixels. However, this embodiment is not limited to this, and focus detection pixels capable of pupil division may be provided in only some of the pixels. Furthermore, although this embodiment shows an example of a configuration in which two photodiodes are arranged for one microlens, a configuration in which three or more (four, nine, etc.) photodiodes are arranged for one microlens may also be used. For example, the present invention is also applicable to a configuration in which multiple photodiodes are arranged in the vertical or horizontal direction for one microlens.

[0030] Next, the configuration of the control unit 105 will be described with reference to Fig. 4. The control unit 105 includes a virtual image generation unit 400, an image processing unit 401, a control unit 402, a temporary recording unit 403, a recording unit 404, a subject detection unit 405, a power management unit 406, a power supply unit 407, and a bus 408. The virtual image generation unit 400 is a generation unit that generates an image to be output by the virtual image display unit 203 and displayed on the in-eye unit 101. The images generated by the virtual image generation unit 400 include an image to be superimposed on an image of the external world to correct vision, and a virtual image. The virtual image is, for example, a menu or indicator that assists operation using the operation unit 104, as well as information indicating the status of the external world obtained from the imaging unit 209 and the status of the image processing device 100. A method for generating an image to be superimposed on an image of the external world generated by the virtual image generation unit 400 of this embodiment to correct vision will be described later.

[0031] The image processing unit 401 performs various image processing on the input digital image signal. The image processing performed by the image processing unit 401 includes gamma correction, white balance processing, noise removal, demosaicing, aberration correction, color correction, and the like. The image processing performed by the image processing unit 401 also includes region extraction processing for extracting a specific region. The image processing unit 401 that performs region extraction processing functions as a region extraction unit. The image processing performed by the image processing unit 401 also includes processing for generating a captured image from a parallax image.

[0032] The image signal input to the image processing unit 401 is image data captured and output by the imaging unit 209. The image processing unit 401 can acquire two images with different parallax (parallax image, image A and image B) by separately processing the outputs of two photoelectric conversion elements. The image processing unit 401 can also acquire an image of the imaging plane (captured image, image A+B) by adding the outputs of the two photoelectric conversion elements. The image processing unit 401 also generates a defocus map based on the parallax image. The defocus map may be generated using a known method such as a split-pupil phase difference detection method. For example, the image processing unit 401 generates multiple defocus maps with different microblocks for performing correlation calculations using the method disclosed in Patent Document 2. The defocus map is a map that has a defocus amount for each pixel, and the defocus amount is expressed in units of Fδ. Based on the defocus map, it is possible to measure the distance from the imaging element to a specific area.

[0033] The control unit 402 executes various programs and controls the overall processing of the image processing device 100. The control unit 402 is, for example, a CPU (Central Processing Unit). The control unit 402 corresponds to the determining means in the claims. The temporary recording unit 403 records data that needs to be temporarily recorded in association with the overall control of the image processing device 100. The temporary recording unit 403 is, for example, a RAM (Random Access Memory). The temporarily recorded data is, for example, image data output from the imaging unit 209.

[0034] The recording unit 404 is a recording means for recording data that requires long-term recording in connection with the overall control of the image processing device 100. The recording unit 404 is, for example, a flash memory. Data recorded in the recording unit 404 includes, for example, a control program required for controlling the image processing device 100, parameters used in the operation of each unit, color vision characteristic information, and an ML dictionary (Machine Learning dictionary) applicable to the subject detection unit 405. When the image processing device 100 is started by a user's power operation, the control program and parameters stored in the recording unit 404 are loaded into the temporary recording unit 403. The control unit 402 controls the operation of the image processing device 100 in accordance with the control program and constants loaded into the temporary recording unit 403. The recording unit 404 records the color vision characteristic information for each user. In addition, the ML dictionary stored in the recording unit 404 is applied to the subject detection unit 405.

[0035] Here, the color vision characteristic information will be described. The color vision characteristic information includes the user's sensitivity to each element of the color space handled by the image processing device 100. For example, if the color space handled by the image processing device 100 is RGB, the color vision characteristic information includes the sensitivity of the user's eyes to each of RGB. The sensitivity is expressed, for example, as a value between 1 and 0. A sensitivity of 1 for each of RGB indicates normal color vision. A sensitivity less than 1 indicates low sensitivity to the corresponding color element. The closer the sensitivity value is to 0, the lower the sensitivity. For example, if the sensitivity to G and B is 1 and the sensitivity to R is 0.5, this indicates low sensitivity to red, which is 0.5 times the sensitivity to green and blue. Note that expressing the sensitivity to each of RGB as a value between 0 and 1 in the color vision characteristic information is merely an example and is not limited to this. The color vision characteristic information may be any value that indicates the user's sensitivity to each element of the color space. In this embodiment, the color vision characteristic information is recorded in the recording unit 404 for each user.

[0036] The subject detection unit 405 detects subjects and extracts areas where specific subjects exist. The subject detection unit 405 applies the ML dictionary recorded in the recording unit 404 to determine subject areas where subjects classified into a predetermined class exist. In the second and fourth embodiments, the subject detection unit 405 corresponds to an area extraction unit. Note that only subject detection may be performed by the subject detection unit 405, and the process of extracting areas corresponding to the subjects may be performed by the image processing unit 401. It is preferable to set subjects of particular importance in use cases in which the user is expected to use the image processing device 100 as the predetermined classes. For example, in use cases such as walking or driving a vehicle, subjects classified into the predetermined classes include pedestrians, bicycles, motorcycles, and automobiles. The subject detection process in the subject detection unit 405 is realized, for example, by feature extraction using DNN (Deep Neural Networks). The configuration of the subject detection unit 405 will be described in detail below with reference to FIGS. 5 and 6.

[0037] The power supply management unit 406 manages the power supply unit 407. The power supply unit 407 is managed by the power supply management unit 406 and supplies power to the entire image processing device 100. The bus 408 is a bus that connects each unit inside the control unit 105 with each unit outside the control unit 105.

[0038] Next, the configuration of the subject detection unit 405 will be described with reference to Fig. 5 and Fig. 6. In this embodiment, an example in which the subject detection unit 405 is configured using CNN (Convolutional Neural Networks) will be described, but the present invention is not limited to this. The subject detection method used by the subject detection unit 405 may be any method that can accurately detect subjects of a predetermined classification.

[0039] FIG. 5 is a diagram illustrating an example of the overall configuration of the CNN (convolutional neural network) in the object detection unit 405. The CNN is configured hierarchically, with a set of two layers called a feature detection layer (S layer) and a feature integration layer (C layer). In object detection in the object detection unit 405, an object is detected from input 2D image data. In FIG. 5, the flow of the object detection process is such that the input is at the left end and the processing progresses to the right. An input image 501, which is 2D image data, is input to the object detection unit 405.

[0040] In Figure 5, input image 501 indicates image data input to the object detection unit 405, S layer 502 indicates the feature detection layer, C layer 503 indicates the feature integration layer, feature detection cell plane 504 indicates the cell plane of the S layer, and feature integration cell plane 505 indicates the cell plane of the C layer. In CNN, the feature detection layer, S layer, first detects the next feature based on the features detected in the previous layer. The features detected in the S layer are then integrated in C layer and sent to the next layer as the detection result for that layer. For example, S layer 502a, the first S layer to which input image 501 is input, detects features from the input image 501 and outputs them to C layer 503a. C layer 503a integrates the features detected in S layer 502a and sends them to the next layer, S layer 502b. S layer 502b detects features based on the features detected in C layer 503a and outputs them to C layer 503b. Similar processing is repeated at each layer, and the features detected at the S layer 502c at the (n-1)th layer are integrated at the C layer 503c, and the features detected at the S layer 502d at the nth layer, which is the output layer, become the subject detection result.

[0041] The S layer has feature detection cell planes 504, and each feature detection cell plane detects a different feature. The C layer has feature integration cell planes 505, and pools the detection results from the previous feature detection cell planes 504. Hereinafter, unless there is a particular need to distinguish between them, the feature detection cell planes 504 and the feature integration cell planes 505 will be collectively referred to as feature planes. In this embodiment, the output layer, which is the final stage layer, does not use the C layer and is composed only of the S layer.

[0042] The feature detection processing at the feature detection cell plane and the feature integration processing at the feature integration cell plane will be described in detail with reference to Fig. 6. Fig. 6 is a diagram illustrating an example of the partial configuration of the CNN in the object detection unit 405. The feature detection cell plane 504 is composed of multiple feature detection neurons. The feature detection neurons are connected to the C layer 503 of the previous layer in a predetermined structure. The feature integration cell plane 505 is composed of multiple feature integration neurons. The feature integration neurons are connected to the S layer 502 of the same layer in a predetermined structure.

[0043] FIG. 6 shows, as an example of layers, a C layer 503e, which is an L-1th layer feature integration layer, an S layer 502f, which is an L-th layer feature detection layer, and a feature integration layer 503f, which is an L-th layer feature integration layer. As part of the feature integration cell surface 505 of the C layer 503e, which is the L-1th layer, the n-1th to n+1th feature integration cell surfaces 505 are shown. Feature integration cell surface 505a is the n-1th feature integration cell surface of the C layer 503e. Feature integration cell surface 505b is the nth feature integration cell surface of the C layer 503e. Feature integration cell surface 505c is the n+1th feature integration cell surface of the C layer 503e. As part of the feature detection cell surface 504 of the S layer 502f, which is the Lth layer, the M-1th to M+1th feature detection cell surfaces 504 are shown. Feature detection cell surface 504f is the Mth feature detection cell surface of the S layer 502f. The M-1th to M+1th feature integrated cell surfaces 505 are shown as part of the feature integrated cell surface 505 of the C layer 503f in the Lth hierarchical layer. The feature integrated cell surface 505f is the Mth feature integrated cell surface of the C layer 503f.

[0044] In the feature detection cell plane 504f of the S layer 502f shown in FIG. 6, the output value of the feature detection neuron at the position (ξ, ζ) is expressed as y M LS (ξ, ζ), in the feature integration cell plane 505f of the C layer 503f, the output value of the feature integration neuron at the position (ξ, ζ) is expressed as y M LC (ξ, ζ). Then, the coupling coefficient of each neuron is expressed as w M LS (n, u, v), w M LCAssuming that the output values ​​are (u, v), each output value can be expressed by the following formulas (1) and (2).

number

number

[0045] In Equation (1), f is an activation function, and can be any sigmoid function such as a logistic function or a hyperbolic tangent function, and can be realized by, for example, a tanh function. M LS (ξ, ζ) is the internal state of the feature detection neuron at position (ξ, ζ) on the feature detection cell surface 504f of the S layer 502f. Equation (2) uses a simple linear sum without using an activation function. When an activation function is not used as in Equation (2), the internal state u M LC (ξ, ζ) and output value y M LC (ξ, ζ) are equal. Also, y in equation (1) n (L-1C) (ξ+u, ζ+v) is the output value of the feature detection neuron, and y in Eq. (2) M LS (ξ+u, ζ+v) is called the output value of the feature integration neuron.

[0046] The following explains ξ, ζ, u, v, and n in Equation (1) and Equation (2). The position (ξ, ζ) corresponds to the position coordinates in the input image. For example, y M LSA high output value of (ξ, ζ) indicates a high probability that the feature to be detected in the feature detection cell plane 504f of the S layer 502f exists at pixel position (ξ, ζ) of the input image. Alternatively, in formula (2), means the feature integration cell plane 505b of the C layer 503e, and is called the integration target feature number. Basically, in the S layer 502f, which is the Lth layer, a product-sum operation is performed on all cell planes in the C layer 503e, which is the L-1th layer. (u, v) is the relative position coordinate of the coupling coefficient, and the product-sum operation is performed within a finite range (u, v) depending on the size of the feature to be detected. This finite range (u, v) is called the receptive field. The size of the receptive field is called the receptive field size and is expressed as the number of horizontal pixels x the number of vertical pixels in the combined range.

[0047] In equation (1), L=1, that is, the first S layer, y n (L-1C) (ξ+u,ζ+v) is the input image y in_image (ξ+u, +v) or input position map y in_posi_map (ξ+u,ζ+v). Note that the distribution of neurons and pixels is discrete, and the connection destination feature numbers are also discrete, so ξ, ζ, u, v, and n are not continuous variables but take discrete values. Here, ξ and ζ are non-negative integers, n is a natural number, and u and v are integers, all of which have a finite range.

[0048] w in formula (1) M LS (n, u, v) is the distribution of coupling coefficients for detecting a given feature. M LS By adjusting (n, u, v) to appropriate values, it becomes possible to detect specific features. Adjusting the distribution of connection coefficients is called learning. In building a CNN, various test patterns are presented to learn the y M LS The coupling coefficients are adjusted by repeatedly correcting them gradually so that (ξ, ζ) becomes an appropriate output value.

[0049] w in formula (2) M LC(u, v) uses a two-dimensional Gaussian function and can be expressed by the following equation (3).

number

[0050] Here again, since is a finite range, as in the explanation of feature detection neurons, this finite range is called the receptive field, and the size of the range is called the receptive field size. The receptive field size can be set to an appropriate value depending on the size of the feature in feature detection cell surface 504f of S layer 502f, which is the Lth layer. σ in equation (3) is a feature size factor, which can be set to an appropriate constant depending on the receptive field size. Specifically, it should be set to a value such that the outermost value of the receptive field can be considered to be approximately 0. The above calculation is repeated at each layer, and object detection is performed in S layer 502d, the final S layer of object detection unit 405, thereby performing object detection processing in object detection unit 405.

[0051] Next, a specific learning method of the object detection unit 405 will be described. In this embodiment, the connection coefficients are adjusted by supervised learning. Supervised learning is a learning method that calculates an optimal model (coefficients) from input data and correct output data. In supervised learning in this embodiment, a test pattern is given to actually find the output value of a neuron, and the connection coefficient w is calculated from the relationship between that output value and a teacher signal (the desired output value that the neuron should output). M LS Correct (n, u, v).

[0052] In the learning of this embodiment, the feature detection layer in the final layer uses the least squares method, and the feature detection layer in the intermediate layer uses the backpropagation method to correct the coupling coefficients. Methods for correcting coupling coefficients, such as the least squares method and the backpropagation method, are described in Non-Patent Document 1. In the object detection unit 405, a large number of specific patterns to be detected and patterns that should not be detected are prepared as test patterns for learning. Each test pattern is a set of an image and a teacher signal. When a tanh function is used as the activation function, when a specific pattern to be detected is presented, a teacher signal is given to neurons in the area where the specific pattern exists on the feature detection cell surface in the final layer so that the output becomes 1. Conversely, when a pattern that should not be detected is presented, a teacher signal is given to neurons in the area of ​​the pattern that should not be detected so that the output becomes -1. In actual object detection, the coupling coefficients w constructed by learning are M LS (n, u, v) and performs a calculation, and if the neuron output on the feature detection cell plane of the final layer is equal to or greater than a predetermined value, it is determined that an object exists there. In this way, the object detection unit 405 is constructed so that it can detect objects from two-dimensional images.

[0053] Next, the light adjustment process for color vision correction in the first embodiment will be described with reference to FIG. 7. In this embodiment, color vision correction is performed by superimposing a virtual image generated by the image processing device 100 on transmitted external light, thereby correcting the color vision of a user of the image processing device 100 who has unusual color vision characteristics. FIG. 7 is a flowchart showing the light adjustment process. The processing in each step is executed by the CPU of the control unit 105 in accordance with a program stored in the recording unit 404, which is a memory. This processing may be started, for example, when the image processing device 100 detects that the user has issued an instruction to start color vision correction via the operation unit 104, or when the image processing device 100 detects that the user has worn the image processing device 100.

[0054] First, in S700, the outside world is photographed by the in-eye unit 101. Specifically, the imaging units 209 of the in-eye unit 101a and the in-eye unit 101b capture images of the outside world using external light, and generate image signals, which are output to the image processing unit 401. The image processing unit 401 then performs image processing on the image signals to generate an outside world image, which is then recorded in the temporary recording unit 403.

[0055] In S701, the control unit 402 determines whether or not the dimming conditions are met. In this embodiment, the control unit 402 determines the following three conditions as dimming conditions for performing color vision correction. (1) Color vision characteristic information corresponding to the user is stored in advance in the recording unit 404. (2) In the color vision characteristics information, there are colors for which color vision sensitivity is low. (3) The user has previously instructed via the operation unit 104 to perform color vision correction. If all three conditions are met, the control unit 402 determines that the dimming conditions are met and performs the process of S702. On the other hand, if any of the three conditions is not met, the control unit 402 determines that the dimming conditions are not met and performs the process of S706.

[0056] The processing in each step from S702 to S705 is performed for each external image captured by the imaging unit 209 of each of the front-of-eye unit 101a and the front-of-eye unit 101b. In S702, the image processing unit 401 extracts a specific region from the external image. Here, the specific region is a region having a luminance value equal to or greater than a predetermined value for colors to which the user has low color vision sensitivity (color information that is difficult to see) based on color vision characteristics information. In this embodiment, partial dimming is performed on the specific region, which is a region to which the user has low color vision sensitivity. Therefore, the image processing unit 401 extracts a region having color information that is difficult to see from the external image as the specific region.

[0057] In S703, the virtual image generation unit 400 calculates a correction area corresponding to the extraction area. Here, the correction area is an area in the coordinate system of the virtual image display unit 203 that corresponds to the extraction area in the external world image extracted in S702. In this embodiment, the three optical axes, namely the optical axis of the eyeball 200, the optical axis of the virtual image display unit 203, and the optical axis of the imaging unit 209, are aligned, resulting in a configuration without parallax. Therefore, if the image magnification and pixel ratio of the virtual image display unit 203 and the imaging unit 209 are set to be equal, pixels with the same coordinates correspond to those in the coordinate systems of the external world image and the virtual image display unit 203. Even if the image magnification and pixel ratio are different, the correction area can be easily calculated by multiplying the distance from the center pixel to the pixel of interest by the image magnification and pixel ratio.

[0058] In S704, the virtual image generation unit 400 generates virtual image information to be displayed on the virtual image display unit 203. The virtual image information is pixel information of a virtual image for a display element of the virtual image display unit 203. A specific method for generating virtual image information will be described later. Next, in S705, the virtual image display unit 203 corrects the user's vision by displaying a virtual image and adjusting the light intensity of the vision correction area. Specifically, the virtual image display unit 203 corrects the user's vision by displaying a virtual image based on the correction area calculated in S703 and the virtual image information generated in S704.

[0059] Next, in S706, the control unit 402 determines whether or not the user has issued an instruction to end color vision correction via the operation unit 104. If an instruction to end color vision correction has been received from the user, the series of light adjustment processes ends. On the other hand, if an instruction to end light adjustment has not been received from the user, the process returns to S700 and light adjustment processes continue.

[0060] Here, we will explain the method for generating virtual image information for color vision correction performed by the virtual image generator 400 in S704. In this embodiment, for each pixel in the correction area calculated in S703, virtual image information is generated using pixel information of a specific area extracted from the corresponding external image and the user's color vision characteristic information recorded in the recording unit 404. The virtual image information can be calculated using the following formula (4).

number

[0061] I I (x, y) is the pixel information at coordinates (x, y) in the correction area of ​​the virtual image. R (i, j) is pixel information at coordinates (i, j) in a specific area in the external image that corresponds to the coordinates (x, y) of the correction area. n and m are the number of pixels in the i direction and j direction in the specific area in the external image. In addition, the coordinates (x, y) in the correction area in the virtual image correspond to the area in the i direction from 0 to n and the area in the j direction from 0 to m in the specific area in the external image. T is the user's color vision characteristic information.

[0062] Here, an example of virtual image information generation for color vision correction will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating an example of virtual image information generation in the first embodiment. First, a case will be described in which the image magnification and pixel ratio of the virtual image display unit 203 and the imaging unit 209 are equal. FIG. 8(A) shows an example in which the image processing device 100 is configured so that the coordinate systems of the external world image and the virtual image are one-to-one. When the coordinate systems of the external world image and the virtual image are one-to-one, both n and m are 1. The specific region 800a is the region extracted from the external world image in S702. The correction region 801a is a correction region corresponding to the specific region 800a. The pixel 802a is one pixel in the specific region 800a. The pixel 803a is one pixel in the correction region 801a and corresponds to the pixel 802a.

[0063] Here, the color vision characteristic information T is assumed to have R of 0.5, G of 1, and B of 1, indicating low sensitivity to red. If the user has low sensitivity to red, that is, if the color information that is difficult for the user to see is red, the color vision of red is corrected to make it easier to see red. The RGB pixel values ​​of pixel 802a are set to R=200, G=100, and B=100, respectively. Equation (4) is applied to the RGB pixel values ​​of pixel 802a to calculate pixel 803a. The RGB pixel values ​​of pixel 803a are set to R=100, G=0, and B=0, respectively. By similarly performing the calculation process using Equation (4) based on the user's color vision characteristic information for all pixels in specific region 800a, all pixel information of correction region 801a, which is virtual image information, is generated.

[0064] Next, a case where the image magnification and pixel ratio of the virtual image display unit 203 and the image capture unit 209 are different will be described. Even if the image magnification and pixel ratio of the virtual image display unit 203 and the image capture unit 209 are different, the correction area can be easily calculated by multiplying the distance from the center pixel to the pixel of interest by the image magnification and pixel ratio. Fig. 8(B) shows an example where the image processing device 100 is configured so that the coordinate systems of the external image and the virtual image have a 2:1 ratio. When the coordinate systems of the external image and the virtual image have a 2:1 ratio, both n and m are 2.

[0065] The specific region 800b is a region extracted from the external image in S702. The correction region 801b is a correction region corresponding to the specific region 800b. Pixels 802b, 802c, 802d, and 802e are each one pixel in the specific region 800b. The bottom side of pixel 802b is adjacent to the top side of pixel 802d, the right side of pixel 802b is adjacent to the left side of pixel 802c, the right side of pixel 802d is adjacent to the left side of pixel 802e, and the bottom side of pixel 802b is adjacent to the top side of pixel 802e. Pixel 803b is one pixel in the correction region 801b and corresponds to four pixels: pixel 802b, pixel 802c, pixel 802d, and pixel 802e.

[0066] Here, the color vision characteristic information T is assumed to be R=1, G=0.8, and B=1, indicating low sensitivity to green. If the user's sensitivity to green is low, the green color vision is corrected to make green more visible. The RGB pixel values ​​of pixel 802b are set to R=100, G=210, and B=100, respectively. The RGB pixel values ​​of pixel 802c are set to R=100, G=220, and B=100, respectively. The RGB pixel values ​​of pixel 802d are set to R=100, G=220, and B=100, respectively. The RGB pixel values ​​of pixel 802e are set to R=100, G=230, and B=100, respectively. Pixel 803b is calculated by applying formula (4) to the RGB pixel values ​​of pixels 802b, 802c, 802d, and 802e. The RGB pixel values ​​of pixel 803b are R = 0, G = 55, and B = 0. By performing the calculation process using equation (4) on all pixels of specific area 800b based on the user's color vision characteristic information in the same manner, all pixel information of correction area 801b, which is virtual image information, is generated.

[0067] In Equation (4), the pixel values ​​of the corresponding correction area are calculated using the average value of n × m pixel values ​​in the specific area of ​​the external image, but this is not limited to this. For example, known scaling methods such as the nearest neighbor method, the bicubic method, or pixel interpolation technology using deep learning may be used to transform the specific area into the correction area to generate virtual image information.

[0068] Finally, an example of application of color vision correction in this embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of application of color vision correction in the first embodiment. FIG. 9(A) shows an external image. Here, an area 900 corresponding to the jacket worn by the human subject is an area of ​​a color (e.g., red) to which the user has low sensitivity according to the user's color vision characteristic information. FIG. 9(B) shows an image perceived by the user with only the naked eye. In an area 901 corresponding to area 900, the user perceives a color different from that of area 900 in the external image.

[0069] 9(C) shows a virtual image for color vision correction provided by the image processing device 100. The image processing device 100 performs light adjustment processing and generates virtual image information for correcting an area 901, which is a vision correction area (S704). The image processing device 100 then performs light adjustment and vision correction by displaying a virtual image of an area 902 corresponding to the area 901 on the virtual image display units 203 of the in-eye unit 101a and the in-eye unit 101b (S705).

[0070] Fig. 9(D) shows a color vision-corrected image seen by a user using the image processing device 100 that performs light adjustment processing. By visually superimposing a virtual image of region 902 generated by the image processing device 100 partially on an image of region 900 of the transmitted outside world, the user can see an image in which region 903 has been color vision-corrected to correspond to the outside world image. That is, part of the outside world light shown in Fig. 9(A) is transmitted through the image processing device 100, and the virtual image shown in Fig. 9(C) is output from the virtual image display unit 203 and superimposed on the outside world light, and is seen by the user.

[0071] As described above, according to this embodiment, color vision can be corrected by generating a virtual image to compensate for colors to which the user has low sensitivity, illuminating the image processing device 100, which transmits external light, and superimposing the external light and the virtual image. As a result, even a user with unusual color vision characteristics can have their color vision corrected to the same level as a user with normal color vision characteristics, and can view an image without parallax between the external world and the virtual image.

[0072] (Second embodiment) In the first embodiment, light adjustment processing for color vision correction was described. In the second embodiment, as vision correction, light adjustment processing for night vision correction in an image processing device that transmits external light is described. Note that, as in the first embodiment, in the image processing device of the second embodiment, three optical axes, namely, the optical axis for capturing external light, the optical axis for visualizing external light, and the optical axis for outputting virtual image light, are assumed to be equally arranged. Furthermore, the overall configuration of the imaging device, the configuration of the in-eye unit, the optical paths of external light and virtual image light, the configuration of the imaging unit, the configuration of the control unit, the configuration of the subject detection unit, and the learning method of the subject detection unit in this embodiment are the same as those in the first embodiment, and therefore will not be described again. Below, differences from the first embodiment will be described.

[0073] The flow of light adjustment processing for night vision correction of the image processing device 100 in the second embodiment will be described using FIG. 7. In this embodiment, night vision correction is performed by superimposing a virtual image generated by the image processing device 100 on transmitted external light, thereby improving the discernibility of important subject images expected depending on the usage environment even under night vision conditions. The processing in each step of FIG. 7 is executed by the CPU of the control unit 105 in accordance with a program stored in the recording unit 404, which is a memory. This processing may be started, for example, when the image processing device 100 detects that the user has issued an instruction to start night vision correction via the operation unit 104, or when the image processing device 100 detects that the user is wearing the image processing device 100.

[0074] The process of capturing an outside image in S700 is the same as S700 in the first embodiment. In S701, the control unit 402 determines whether or not the dimming conditions for performing night vision correction are met. In this embodiment, the control unit 402 determines whether or not the dimming conditions are met based on the following two conditions: (1) The environment is low in discernibility (visually difficult for the user) based on the image information of the external image. (2) Has the user previously instructed via the operation unit 104 to perform night vision correction? An environment with low discrimination, i.e., an environment where the user has difficulty seeing, indicates that the environment in which the image processing device 100 is used is under night vision. Whether or not the environment has low discrimination and is difficult to see is determined, for example, by evaluating the pixel values ​​(brightness values) of the entire external world image. Specifically, if the sum of the pixel values ​​of the entire external world image is less than a predetermined value, it is determined that the environment is difficult to see. Furthermore, the image processing device 100 may be equipped with an illuminance meter (not shown), and if the illuminance measured by the illuminance meter is less than a predetermined illuminance, it may be determined that the environment is difficult to see. If both of the two conditions are met, the control unit 402 determines that the dimming condition is met and performs processing of S702. On the other hand, if any of the two conditions is not met, the control unit 402 determines that the dimming condition is not met and performs processing of S706.

[0075] The processing in each step from S702 to S705 is performed for each external image captured by the imaging unit 209 of each of the front-of-sight unit 101a and the front-of-sight unit 101b. In S702, the subject detection unit 405 extracts a specific region from the external image. Here, the specific region is a subject region corresponding to a predetermined subject. The predetermined subject is a subject that has been set in advance as a subject of particularly high importance in a use case. For example, when the use case is while walking or while driving a vehicle, pedestrians, bicycles, motorcycles, and automobiles are set as the predetermined subjects. In this embodiment, in a low-visibility environment such as night vision, the region corresponding to a subject of particularly high importance is displayed with partial dimming to improve its visibility.

[0076] In S703, the virtual image generation unit 400 calculates a correction area corresponding to the extracted area. Here, the correction area is an area in the coordinate system of the virtual image display unit 203 that corresponds to the specific area in the external world image extracted in S702. In this embodiment, the three optical axes, namely the optical axis of the eyeball 200, the optical axis of the virtual image display unit 203, and the optical axis of the imaging unit 209, are aligned, resulting in a configuration without parallax. Therefore, if the image magnification and pixel ratio of the virtual image display unit 203 and the imaging unit 209 are set to be equal, pixels with the same coordinates will correspond in the coordinate systems of the external world image and the virtual image display unit 203. Even if the image magnification and pixel ratio are different, the correction area can be easily calculated by multiplying the distance from the center pixel to the pixel of interest by the image magnification and pixel ratio.

[0077] In step S704, the virtual image generation unit 400 generates virtual image information to be displayed on the virtual image display unit 203. The virtual image information is pixel information of a virtual image for the display element of the virtual image display unit 203. A method for generating virtual image information for night vision correction in S704 of the second embodiment will be described. In this embodiment, for each pixel of the correction area calculated in S703, virtual image information is generated using pixel information of a specific area extracted from the corresponding external image and a night vision correction coefficient calculated by the image processing unit 401. The virtual image information can be calculated using the following formula (5). Note that the pixel values ​​of the virtual image information are rounded down to the nearest integer.

number

number

[0078] β is the maximum night vision correction rate recorded in the recording unit 404. The maximum night vision correction rate β indicates the percentage of the maximum amount of light that the virtual image display unit 203 can output when performing night vision correction. The maximum night vision correction rate β is expressed as a value between 1 and 0, with 1 indicating that night vision correction is performed to the maximum, and 0 indicating that night vision correction is not performed. Rout_MAX is the maximum pixel value of any of RGB that can be output by the virtual image display unit 203. Rin_MAX is the maximum pixel value of any of RGB among the pixel values ​​of a specific region in the external image. Rin_MAX By setting in this way, the pixel values ​​of all color elements of all pixels in the correction area are controlled so as not to exceed the pixel value determined by the maximum night vision correction rate. Also, for the entire correction area, a common night vision correction coefficient α is used to calculate the pixel I I By calculating (x, y), the distribution tendency of pixel information in a specific area is maintained.

[0079] Here, an example of virtual image information generation for night vision correction will be described with reference to FIG. 10. FIG. 10 is a diagram illustrating an example of virtual image information generation in the second embodiment. First, a case will be described in which the image magnification and pixel ratio of the virtual image display unit 203 and the imaging unit 209 are equal. FIG. 10(A) shows an example in which the image processing device 100 is configured so that the coordinate systems of the external image and the virtual image are one-to-one. The specific area 1000a is an area extracted from the external image in S702. The correction area 1001a is a correction area corresponding to the specific area 1000a. The pixel 1002a is one pixel in the specific area 1000a. The pixel 1003a is one pixel in the correction area 1001a and corresponds to the pixel 1002a.

[0080] To generate a virtual image correction area from a specific area of ​​the external image, Equation (5) and Equation (6) are applied to the RGB pixel values ​​of pixel 1002a. For example, the RGB pixel values ​​of pixel 1002a are set to R=200, G=100, and B=100, respectively. The maximum night vision correction rate β is set to 0.5, I Rout_MAX 255, I Rin_MAXis set to 10 as the pixel value of the color element R of pixel 1002a. The night vision correction coefficient α is calculated as α = 0.5 × 255 / 10 = 12.75 using equation (6). When equation (5) is applied to each of the RGB pixel values ​​shown in pixel 1002a, the RGB pixel values ​​of pixel 1003a are R = 127, G = 12, and B = 12. Here, the pixel values ​​are truncated to the nearest integer. By performing the same process on all pixels in specific area 1000a, all pixel information for correction area 1001a is generated.

[0081] Next, a case where the image magnification and pixel ratio of the virtual image display unit 203 and the image capturing unit 209 are different will be described. Even if the image magnification and pixel ratio of the virtual image display unit 203 and the image capturing unit 209 are different, the correction area can be easily calculated by multiplying the distance from the center pixel to the pixel of interest by the image magnification and pixel ratio. Fig. 10(B) shows an example where the image processing device 100 is configured so that the coordinate systems of the external image and the virtual image have a 2:1 ratio.

[0082] The specific region 1000b is the region extracted from the external image in S702. The correction region 1001b is the correction region corresponding to the specific region 1000b. The pixels 1002b, 1002c, 1002d, and 1002e are each one pixel in the specific region 1000b. The pixel 1003b is one pixel in the correction region 1001b and corresponds to four pixels: the pixel 1002b, the pixel 1002c, the pixel 1002d, and the pixel 1002e.

[0083] The RGB pixel values ​​of pixel 1002b are R=1, G=20, and B=3, respectively. The RGB pixel values ​​of pixel 1002c are R=3, G=20, and B=1, respectively. The RGB pixel values ​​of pixel 1002d are R=3, G=20, and B=1, respectively. The RGB pixel values ​​of pixel 1002e are R=1, G=10, and B=3, respectively. β and I Rout_MAX is the same as in Figure 8(A), and I Rin_MAXis set to the pixel value of the color element G of pixel 1002b, which is 30. The night vision correction coefficient α is calculated as α = 0.5 × 255 / 30 = 4.25 using equation (6). Pixel 1003b is calculated by applying equation (5) to the RGB pixel values ​​of pixels 1002b, 1002c, 1002d, and 1002e. The RGB pixel values ​​of pixel 1003b are R = 8, G = 85, and B = 8, respectively. Here, the pixel values ​​are truncated to the nearest integer. By performing the same process on all pixels of extraction area 1000b, all pixel information for correction area 1001b is generated.

[0084] In Equation (5), the pixel values ​​of the corresponding correction area are calculated using the average value of the n × m pixel values ​​in the extracted area of ​​the external image, but this is not limited to this. For example, known scaling methods such as the nearest neighbor method, the bicubic method, or pixel interpolation technology using deep learning may be used to transform the specific area into the correction area to generate virtual image information.

[0085] In S705, the virtual image display unit 203 corrects the user's vision by displaying a virtual image and adjusting the brightness of the vision correction area. Specifically, the virtual image display unit 203 corrects the user's vision by displaying a virtual image based on the correction area calculated in S703 and the virtual image information generated in S704. In S706, the control unit 402 determines whether the user has issued an instruction to end night vision correction via the operation unit 104. If an instruction to end night vision correction has been received from the user, the control unit 402 ends the series of light adjustment processes. On the other hand, if an instruction to end light adjustment for night vision correction has not been received from the user, the control unit 402 returns to S700 and continues the light adjustment process for night vision correction.

[0086] An application example of night vision correction in this embodiment will be described with reference to FIG. 11. FIG. 11 is a diagram illustrating an example of virtual image information generation in the second embodiment. FIG. 11(A) shows an external image. Area 1100 is the subject area (specific area) detected by the subject detection unit 405 in S702. Because it is under night vision, the entire external image including area 1100 is difficult to distinguish with the naked eye.

[0087] 11(B) shows a virtual image for night vision correction provided by the image processing device 100. The image processing device 100 performs light adjustment processing for night vision correction of the subject area and generates virtual image information of the area 1101 (S704). Then, the image processing device 100 adjusts light by displaying a virtual image of the area 1101 corresponding to the area 1100, which is the subject area to be corrected, on the virtual image display units 203 of the front-of-eye unit 101a and the front-of-eye unit 101b, thereby performing vision correction (S705).

[0088] Fig. 11(C) shows a night-vision corrected image seen by a user using the image processing device 100 that performs light adjustment processing. By superimposing a virtual image of area 1101 generated by the image processing device 100 on an image of area 1100 of the transmitted outside world, the user can see an image in which area 1102 has been night-vision corrected. That is, part of the outside world light shown in Fig. 11(A) is transmitted through the image processing device 100, and the virtual image shown in Fig. 11(B) is output from the virtual image display unit 203 and superimposed on the outside world light, and is seen by the user.

[0089] As described above, according to this embodiment, night vision correction can be performed by generating a virtual image for improving the distinguishability of important subject images in night vision, illuminating the image processing device 100, which transmits external light, and superimposing the external light and the virtual image. This improves the distinguishability of important subject images expected depending on the usage environment even in night vision, and allows the user of the image processing device to see an image without parallax between the external world and the virtual image.

[0090] (Third embodiment) In the first embodiment, a dimming process using color vision correction was described in a case where three optical axes, namely, an optical axis for capturing external light, an optical axis for visualizing external light, and an optical axis for outputting virtual image light, are equally arranged. In the third embodiment, a dimming process using color vision correction is described in a case where two optical axes, namely, an optical axis for capturing external light and an optical axis for outputting virtual image light, are equally arranged. That is, in this embodiment, a case where an optical axis for capturing external light (an optical axis of external light corresponding to the imaging unit 1206) differs from an optical axis for visualizing external light (an optical axis of the user's eyeball) and an optical axis for outputting virtual image light (an optical axis of virtual image light output by the virtual image display unit 1303) is described. The color vision characteristic information, the configuration of the subject detection unit, the learning method of the subject detection unit, the method for generating virtual image information in color vision correction, and specific examples of color vision correction are the same as those in the first embodiment, and therefore will not be described again. Differences from the first embodiment will be described below.

[0091] 12 is a diagram showing the configuration of an image processing device 1200 in the third embodiment. The image processing device 1200 includes a front-of-eye unit 1201, a shield 1202, a frame 1203, a control unit 1205, and an imaging unit 1206. The image processing device 100 may also include an operation unit 1204. The configurations of the shield 1202, the frame 1203, the operation unit 1204, and the control unit 1205 are similar to those of the shield 102, the frame 103, the operation unit 104, and the control unit 105 in the first embodiment, and therefore a description thereof will be omitted. The detailed configuration of the control unit 1205 is also similar to that of the control unit 105 in the first embodiment shown in FIG. 4, and therefore a description thereof will be omitted.

[0092] The imaging unit 1206 is a unit used to capture external world images for both left and right use in order to generate virtual images for the in-eye unit 1201 (in-eye unit 1201a and in-eye unit 1201b). A detailed configuration of the imaging unit 1206 will be described later using FIG. 14. In this embodiment, the imaging performed by the imaging unit 209 of the in-eye unit 101 in the first embodiment is performed by the imaging unit 1206. Therefore, the in-eye unit 1201 of this embodiment does not have the imaging unit 209 or a configuration for guiding light to the imaging unit 209. Details of the in-eye unit 1201 of this embodiment will be described using FIG. 13.

[0093] FIG. 13 is a diagram illustrating the configuration of the in-eye unit 1201 in the third embodiment. FIG. 13 shows the in-eye unit 1201a as viewed from the top of the head of a human body. Here, the configuration will be described using the in-eye unit 1201a as an example, but the in-eye unit 1201b has a similar configuration. The in-eye unit 1201a includes a virtual image display unit 1303, a virtual image adjustment unit 1304, and an optical path control unit 1306. The virtual image display unit 1303 and the virtual image adjustment unit 1304 have the same configurations as the virtual image display unit 203 and the virtual image adjustment unit 204 in the first embodiment, respectively, and therefore their description will be omitted. The eyeball 1300 is the user's eyeball. The iris and lens 1301 are the user's iris and lens. The retina 1302 is the user's retina.

[0094] The light path control unit 1306 is a half mirror with predetermined reflectance and transmittance. The light path control unit 1306 reflects the virtual image output from the virtual image display unit 1303. The light path control unit 1306 also transmits external light 1310 that enters the in-eye unit 1201a via an optical path 1312. The virtual image reflected by the light path control unit 1306 has the same optical axis as the external light that has passed through the light path control unit 1306, and enters the user's eyeball 1300 via an optical path 1315.

[0095] The optical paths of external light 1310 and virtual image light 1311 controlled by the front-of-eye unit 1201a will be described. The external light 1310 passes through optical path 1312, enters the front-of-eye unit 1201a, and passes through optical path control section 1306. The light that has passed through optical path control section 1306 enters the user's eyeball 1300 via optical path 1315. The light that has entered the user's eyeball 1300 is refracted by the iris and crystalline lens 1301, and then forms an image on the retina 1302, allowing the user to view the external light.

[0096] Virtual image light 1311 emitted by virtual image display unit 1303 is subjected to focus adjustment control by virtual image adjustment unit 1304, then reflected by optical path control unit 1306, has the same optical axis as external light that has passed through optical path control unit 1306, and enters user's eyeball 1300 via optical path 1315. The light that has entered user's eyeball 1300 is refracted by iris and crystalline lens 1301 and then forms an image on retina 1314, allowing the user to view a virtual image.

[0097] The external light 1310 visually perceived by the naked eye 1300 via the image processing device 1200 of this embodiment and the virtual image light 1311 displayed by the virtual image display unit 1303 have the same optical axis and a parallax-free structure. Therefore, this is a suitable configuration for calculating the correction area. Note that the configuration of the eye-front unit 1201 described here is an example, and is not limited to this. Any optical see-through device that can superimpose external light and virtual image light with high precision may be used.

[0098] The configuration of the imaging unit 1206 will be described using Fig. 14. Fig. 14 is a diagram illustrating the configuration of the imaging unit 1206 in the third embodiment. The imaging unit 1206 captures external light and outputs the acquired image data to the control unit 1205. The imaging unit 1206 has a light amount adjustment unit 1407, an optical system 1408, and an imaging unit 1409. The light amount adjustment unit 1407, the optical system 1408, and the imaging unit 1409 have the same configurations as the light amount adjustment unit 207, the optical system 208, and the imaging unit 209 in the first embodiment, respectively, and therefore detailed descriptions thereof will be omitted. Furthermore, the pixel structure of the imaging unit 1409 is the same as that in the first embodiment (Fig. 3), and therefore detailed descriptions thereof will be omitted.

[0099] Next, the light adjustment process for color vision correction in the third embodiment will be described with reference to Fig. 7. In this embodiment, color vision correction is performed by superimposing a virtual image generated by the image processing device 100 on transmitted external light, thereby correcting the color vision of a user of the image processing device 100 who has unusual color vision characteristics. The processing in each step is executed by the CPU of the control unit 105 in accordance with a program stored in the recording unit 404, which is a memory.

[0100] First, in S700, the imaging unit 1206 captures an image of the outside world. Specifically, the imaging section 1409 of the imaging unit 1206 captures an image of the outside world and generates an image signal, which is output to the image processing section 401. The image processing section 401 then performs image processing on the image signal to generate an outside world image, which is recorded in the temporary recording section 403. In this embodiment, the image processing section 401 also generates a defocus map corresponding to the outside world image, and records it in the temporary recording section 403.

[0101] The processing in steps S701 and S702 is the same as that in the first embodiment, and therefore a detailed description thereof will be omitted. In step S701, if three dimming conditions for performing dimming processing for color vision correction are satisfied, a specific region is extracted from the external image in step S702. The specific region is a region that has a luminance value equal to or greater than a predetermined value for colors to which the user has low color vision sensitivity based on color vision characteristics information.

[0102] In S703, the virtual image generation unit 400 calculates a correction area corresponding to the extraction area. Here, the correction area is an area in the coordinate system of the virtual image display unit 203 that corresponds to the extraction area in the external world image extracted in S702. In this embodiment, the correction area for the virtual image display unit 1303 provided in each of the front-of-eye unit 1201a and the front-of-eye unit 1201b is calculated based on the external world image captured by the imaging unit 1409 provided in the imaging unit 1206. Therefore, in S703 of this embodiment, the correction area for each virtual image display unit 1303 is calculated. Similarly, in each process of S703 to S705 of this embodiment, processing for each virtual image display unit 1303 is performed.

[0103] Furthermore, in this embodiment, although the optical axis of the eyeball 1300 and the optical axis of the virtual image display unit 1303 are aligned, the optical axis of the imaging unit 1409 is different, resulting in a configuration with parallax. To correct the parallax, the amount of parallax and the corresponding number of parallax pixels are calculated, and corresponding pixels in the correction area are derived by shifting from the pixel center based on the calculated number of parallax pixels. The method for calculating the amount of parallax will be described later with reference to FIG. 15 . If the image magnification and pixel ratio of the virtual image display unit 1303 and the imaging unit 1409 are set equal, the coordinate systems of the external image and the virtual image display unit 1303 correspond to pixels shifted by the number of parallax pixels from the same coordinates. Even if the image magnification and pixel ratio are different, the corresponding pixels can be easily derived by multiplying the distance from a pixel shifted by the number of parallax pixels from the center pixel to the pixel of interest by the image magnification and pixel ratio.

[0104] In S704, the virtual image generation unit 400 generates virtual image information to be displayed on the virtual image display unit 203. The virtual image information is pixel information of a virtual image for a display element of the virtual image display unit 203. The method of generating the virtual image information is the same as in the first embodiment, so a detailed description will be omitted. Next, in S705, the virtual image display unit 203 corrects the user's vision by displaying a virtual image and adjusting the light intensity of the vision correction area. Specifically, the virtual image display unit 203 corrects the user's vision by displaying a virtual image based on the correction area calculated in S703 and the virtual image information generated in S704. In S706, the control unit 402 determines whether the user has issued an instruction to end color vision correction via the operation unit 104. If an instruction to end color vision correction has been received from the user, the series of light adjustment processes is terminated. On the other hand, if an instruction to end light adjustment has not been received from the user, the process returns to S700 and the light adjustment process continues.

[0105] Here, the method for calculating the amount of parallax in S703 will be described with reference to FIG. 15. FIG. 15 is a diagram for explaining the calculation of the amount of parallax. FIG. 15 schematically shows the top of the user's head, object 1500, imaging unit 1206, virtual image display unit 1303, virtual image adjustment unit 1304, and user's eyeball 1300. Object 1500 is an object present in a specific area of ​​the external image, and eyeball 1300 is the user's left eye. In front of the user's left eye wearing image processing device 1200 is preocular unit 1201b. Iris and crystalline lens 1301 and retina 1302 are the iris, crystalline lens, and retina of eyeball 1300, respectively. Light amount adjustment unit 1407, optical system 1408, and imaging unit 1409 are components of imaging unit 1206. Also, it is assumed here that the virtual image output from the virtual image display unit 1303 is not reflected by the optical path control unit 1306 but travels straight ahead and enters the user's eyeball 1300, and the virtual image display unit 1303 and the virtual image adjustment unit 1304 are illustrated in positions that maintain their optical positional relationship.

[0106] In this embodiment, two optical axes, the visual optical axis of external light and the optical axis of the output of virtual image light, are arranged coplanar. Optical axis 1501 is the optical axis center of the imaging unit 1409. Optical axis 1503 is the optical axis center of the eyeball 1300 and the optical axis center of the virtual image display unit 1303. Light incident at a certain point on the subject 1500 travels along optical axis 1501 and reaches the optical axis center of the imaging element 1409. On the other hand, light reflected at the same point on the subject 1500 travels along optical path 1502 and reaches the retina 1302 shifted from the optical axis center (optical axis 1503) by a parallax amount p. A correction area in the coordinate system of the virtual image display unit 203, which corresponds to the extraction area of ​​the external image captured by the imaging unit 1409, has a parallax amount p. In this case, the parallax amount p can be calculated using the following equation (7).

number

[0107] def is distance information from the image capturing unit 1409 to the subject 1500, obtained from a defocus map corresponding to the coordinates of interest in the external world image. D is the distance between the optical axis center of the image capturing unit 1409 and the optical axis center of the virtual image display unit 1303. Since D is determined by the configuration of the image processing device 100 and is known, it is recorded in advance in the recording unit 404 and read out for use. l is the length from the user's iris and crystalline lens 1301 to the retina 1302. l is a statistical value of the human body that is recorded in advance in the recording unit 404 and read out for use. When the parallax amount p is the distance on the virtual image display unit 1303, it becomes the number of parallax pixels P. The number of parallax pixels P can be calculated using the following formula (8).

number

[0108] S is the number of pixels of the virtual image display unit 1303. h is the size of the virtual image display unit 1303. Since S and h are determined and known from the configuration of the image processing device 100, they are recorded in advance in the recording unit 404 and read out for use. From the above, the parallax amount p and the number of parallax pixels P can be calculated. When calculating a correction area in the coordinate system of the virtual image display unit 203 that corresponds to the extracted area of ​​the external world image, the virtual image generation unit 400 derives corresponding pixels by shifting them from the pixel center based on the number of parallax pixels P.

[0109] As described above, according to this embodiment, even when the optical axis of the captured image of external light differs from the optical axis of the visual sense of the external light and the optical axis of the output of virtual image light, a virtual image whose position has been corrected due to the misalignment of the optical axis can be superimposed and displayed on the external light passing through the image processing device 100. Then, a virtual image is generated to compensate for colors to which the user has low sensitivity, and is projected onto the image processing device 100 that transmits the external light, thereby superimposing the external light and the virtual image, thereby correcting color vision. This allows even a user with unusual color vision characteristics to have their color vision corrected to the same level as a user with normal color vision characteristics, and to view an image without parallax between the external world and the virtual image.

[0110] (Fourth embodiment) In the second embodiment, a dimming process using night vision correction was described when three optical axes, namely, the optical axis for capturing external light, the optical axis for visualizing external light, and the optical axis for outputting virtual image light, are equally spaced. In the third embodiment, a dimming process using night vision correction was described when two optical axes, namely, the optical axis for visualizing external light and the optical axis for outputting virtual image light, are equally spaced. That is, in this embodiment, a case where the optical axis for capturing external light is different from the optical axis for visualizing external light and the optical axis for outputting virtual image light, is described. The overall configuration of the imaging device, the configuration of the in-eye unit, the optical paths of external light and virtual image light, the configuration of the imaging unit and imaging section, the configuration of the control section, the configuration of the subject detection section, the learning method of the subject detection section, and the parallax correction method are the same as in the third embodiment, and therefore their description will be omitted. Furthermore, the method for generating virtual image information in night vision correction and specific examples of night vision correction are the same as in the second embodiment, and therefore their description will be omitted.

[0111] The light control process for night vision correction in the fourth embodiment will be described with reference to Fig. 7. In this embodiment, night vision correction is performed by superimposing a virtual image generated by the image processing device 100 on transmitted external light, thereby improving the discernibility of important subject images expected depending on the usage environment even under night vision. The process in each step of Fig. 7 is executed by the CPU of the control unit 105 according to a program stored in the recording unit 404, which is a memory.

[0112] In S700, the imaging unit 1206 captures an image of the outside world. The processing of S700 in this embodiment is the same as the processing of S700 in the third embodiment. In S701, the control unit 402 determines whether or not the light control conditions are met. The processing of S701 in this embodiment is the same as the processing of S701 in the second embodiment. The processing of S702 in this embodiment is the same as the processing of S703 in the second embodiment. The processing of each of S703 to S706 in this embodiment is the same as the processing of each of S703 to S706 in the third embodiment.

[0113] As described above, according to this embodiment, even when the optical axis of the captured image of external light differs from the visual optical axis of the external light and the optical axis of the output of virtual image light, it is possible to superimpose and display a virtual image whose position due to the misalignment of the optical axis is corrected on the external light passing through the image processing device 100. Furthermore, night vision correction can be performed by generating a virtual image for improving the distinguishability of an important subject image under night vision conditions, irradiating it on the image processing device 100 that transmits external light, and superimposing the external light and the virtual image. This improves the distinguishability of an important subject image expected depending on the usage environment, even under night vision conditions, and allows the user of the image processing device to see an image without parallax between the external world and the virtual image.

[0114] In the third and fourth embodiments, the optical axis of the captured external light is different from the visual optical axis of the external light and the optical axis of the output virtual image light. However, this is not limiting. It is sufficient that two of the optical axes of the imaging unit, the virtual image display means, and the user's eyeball are aligned. In this case, it is possible to correct parallax based on the position information of at least two of the imaging unit, the virtual image display means, and the user's eyeball, and to determine the position where the virtual image is displayed.

[0115] The disclosure of this embodiment includes the following configuration of an image processing device. (Configuration 1) An image processing device that transmits external light and allows a user to see, An imaging means for imaging the outside world; an area extraction means for extracting a specific area from an external image captured by the imaging means; a virtual image generating means for generating a virtual image for correcting the user's vision with respect to the specific area; and a virtual image display means for displaying the virtual image by superimposing it on an area corresponding to the specific area of ​​the transmitted external light. (Configuration 2) further comprising a recording means for recording the color vision characteristics of the user; the region extraction means extracts, as a specific region, a region having color information to which the user has low color vision sensitivity from the external image based on the color vision characteristics; The image processing device according to configuration 1, wherein the virtual image generating means generates a virtual image for correcting color information for which the user has low color vision sensitivity based on pixel information of the specific region and the color vision characteristics. (Configuration 3) the region extraction means detects a subject region corresponding to a predetermined subject from the external image and extracts the subject region as a specific region; 3. The image processing device according to claim 1, wherein the virtual image generating means generates a virtual image for correcting the discernibility of the specified subject based on pixel information of the specific region and a night vision correction coefficient. (Configuration 4) a recording means for recording a maximum night vision correction rate indicating a rate at which night vision correction is performed with respect to a maximum amount of light that can be output by the virtual image display means, the region extraction means extracts the subject region as a specific region when the discriminability of the outside world is low, i.e., when a sum of pixel values ​​of the outside world image is less than a predetermined value; The image processing device according to configuration 3, wherein the virtual image generating means calculates a night vision correction coefficient based on the maximum night vision correction rate, the maximum pixel value that can be output by the virtual image display means, and the maximum pixel value of the specific region. (Configuration 5) an optical path control unit including an external world side that is a half mirror that transmits a part of the external world light to make it incident on the eyeball of the user and reflects a part of the external world light to make it incident on the imaging means, and an eyeball side that is a full mirror that reflects a virtual image irradiated by a virtual image display means to make it incident on the eyeball; 5. The image processing device according to any one of configurations 1 to 4, characterized in that the image processing device is arranged so that the optical axis of external light corresponding to the imaging means, the optical axis of virtual image light output by the virtual image display means, and the optical axis of the user's eyeball are aligned. (Configuration 6) the optical axis of external light corresponding to the imaging means, the optical axis of virtual image light output by the virtual image display means, and the optical axis of the user's eyeball are arranged to be aligned with each other; 5. The image processing device according to any one of configurations 1 to 4, wherein the position at which the virtual image is displayed is determined based on position information of at least two or more of the imaging means, the virtual image display means, and the user's eyeball. (Configuration 7) the imaging means receives light beams that have passed through different pupil regions of an imaging optical system, and outputs a plurality of images with parallax; 7. The image processing device according to configuration 6, wherein when determining the position where the virtual image is to be displayed, a distance from the imaging means to the specific area measured from a plurality of images with parallax is used. (Configuration 8) 8. The image processing device according to any one of configurations 1 to 7, wherein the virtual image display means displays the virtual image by emitting light of at least one wavelength. (Configuration 9) 8. The image processing device according to any one of configurations 1 to 7, wherein the image processing device is a head-mounted display.

[0116] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0117] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0118] 100 Image processing device 101 Front Unit 105 Control Unit 203 Virtual image display unit 209 Imaging unit 400 Virtual image generation unit 402 Control Unit 405 Subject detection unit

Claims

1. An image processing device that transmits external light and allows a user to see, An imaging means for imaging the outside world; an area extraction means for extracting a specific area from an external image captured by the imaging means; a virtual image generating means for generating a virtual image for correcting the user's vision with respect to the specific area; and a virtual image display means for displaying the virtual image by superimposing it on an area corresponding to the specific area of ​​the transmitted external light.

2. further comprising a recording means for recording the color vision characteristics of the user; the region extraction means extracts, as a specific region, a region having color information to which the user has low color vision sensitivity from the external image based on the color vision characteristics; The image processing device according to claim 1 , wherein the virtual image generating means generates a virtual image for correcting color information for which the user has low color vision sensitivity, based on the pixel information of the specific region and the color vision characteristics.

3. the region extraction means detects a subject region corresponding to a predetermined subject from the external image and extracts the subject region as a specific region; 2. The image processing device according to claim 1, wherein the virtual image generating means generates a virtual image for correcting the discernibility of the predetermined subject based on pixel information of the specific area and a night vision correction coefficient.

4. a recording means for recording a maximum night vision correction rate indicating a rate at which night vision correction is performed with respect to a maximum amount of light that can be output by the virtual image display means, the region extraction means extracts the subject region as a specific region when the discriminability of the outside world is low, i.e., when a sum of pixel values ​​of the outside world image is less than a predetermined value; 4. The image processing device according to claim 3, wherein the virtual image generating means calculates a night vision correction coefficient based on the maximum night vision correction rate, the maximum pixel value that can be output by the virtual image display means, and the maximum pixel value of the specific region.

5. an optical path control unit including an external world side that is a half mirror that transmits a part of the external world light to make it incident on the eyeball of the user and reflects a part of the external world light to make it incident on the imaging means, and an eyeball side that is a full mirror that reflects a virtual image irradiated by a virtual image display means to make it incident on the eyeball; 2. The image processing device according to claim 1, wherein the optical axis of the external light corresponding to the imaging means, the optical axis of the virtual image light output by the virtual image display means, and the optical axis of the user's eyeball are aligned.

6. the optical axis of external light corresponding to the imaging means, the optical axis of virtual image light output by the virtual image display means, and the optical axis of the user's eyeball are arranged to be aligned with each other; 2. The image processing device according to claim 1, wherein the position at which the virtual image is displayed is determined based on position information of at least two of the imaging means, the virtual image display means, and the user's eyeball.

7. the imaging means receives light beams that have passed through different pupil regions of an imaging optical system, and outputs a plurality of images with parallax; 7. The image processing device according to claim 6, wherein when determining the position where the virtual image is to be displayed, a distance from the imaging means to the specific area measured from a plurality of images with parallax is used.

8. 2. The image processing apparatus according to claim 1, wherein the virtual image display means displays the virtual image by emitting light of at least one wavelength.

9. The image processing device according to claim 1 , wherein the image processing device is a head-mounted display.

10. A control method for an image processing device that transmits external light and allows a user to see, comprising: an extraction step of extracting a specific region from an external image acquired from an imaging means for imaging the external world; a virtual image generating step of generating a virtual image for correcting the user's vision with respect to the specific area; a virtual image display step of superimposing and displaying the virtual image on an area corresponding to the specific area of ​​the transmitted external light.

11. A program that causes a computer of an image processing apparatus to execute each step according to claim 10.

Images