Image correction device for augmented reality with images

Abstract

The present application relates to a kind of image correction devices of augmented reality image, provide a kind of image correction devices of augmented reality image, it is combined with augmented reality optical device, the augmented reality optical device has the image emission unit of the augmented reality image light corresponding to the emission of augmented reality image, at least a part of the actual matter image light as the image light emitted from actual matter is made to the pupil of user by the optical mechanism, and multiple reflection parts are arranged in the inside of the optical mechanism to reflect the augmented reality image light emitted by image emission unit and be transmitted to the pupil of user, the image correction devices of the augmented reality image of the present application is characterized in that, including: compensation function decision part, when the original augmented reality image is emitted by the image emission unit, the compensation function for correcting the luminance information of the observation image observed by the user through the augmented reality optical device is determined;And correction image information generation unit, based on the compensation function determined by the compensation function decision part and the original augmented reality image information, generate correction augmented reality image information, the image emission unit emits the correction augmented reality image light corresponding to the correction augmented reality image information generated by the correction image information generation unit, and the multiple reflection parts reflect the correction augmented reality image light emitted by the image emission unit and be transmitted to the pupil of user.

Description

TECHNICAL FIELD

[0001] The present application relates to an image correction device for augmented reality images, and more particularly, to an image correction device for augmented reality images capable of providing augmented reality images having uniform brightness distribution. BACKGROUND

[0002] As is well known, augmented reality (AR) refers to providing a virtual image or image provided by a computer or the like superimposed on an actual image of the real world.

[0003] In order to realize such augmented reality, an optical system capable of providing a virtual image or image generated by a device such as a computer superimposed on an image of the real world is required. As such an optical system, a technique using an optical mechanism such as a prism that reflects or refracts a virtual image using an HMD (Head Mounted Display) or a glasses-type device is known.

[0004] However, these devices using the conventional optical system have a problem in that, due to a complicated structure, the weight and volume are relatively large, it is inconvenient for a user to wear, and the manufacturing process is also complicated, so that the manufacturing cost is high.

[0005] In addition, the limitation of the conventional device is that when the user changes the focal distance while gazing at the real world, the virtual image is out of focus. In order to solve this problem, a structure such as using a prism capable of adjusting the focal distance of the virtual image, or a technique of electrically controlling a variable focus lens according to the change of the focal distance is proposed. However, this technique also has a problem in that the user needs to perform a separate operation to adjust the focal distance, or a separate processor and the like hardware and software for controlling the focal distance are required.

[0006] In order to solve the problem of such a conventional technique, as described in Patent Document 1, the present applicant has developed a device capable of realizing augmented reality by projecting a virtual image, i.e., an augmented reality image, through a pupil to a retina by using a reflecting portion having a size smaller than a pupil of a person.

[0007] Figure 1 FIG. 1 is a view showing an augmented reality optical device 100 as disclosed in Patent Document 1 applied by the present applicant.

[0008] Figure 1 The augmented reality optical device 100 of FIG. 1 includes an optical mechanism 10, a reflecting portion 20, an image emission portion 30, and a frame portion 40.

[0009] The optical mechanism 10 is a mechanism that transmits at least a part of the actual object image light as image light emitted from an actual object, and for example, can be a spectacle lens in which the reflection portion 30 is embedded. Further, the optical mechanism 10 also functions to transmit the augmented reality image light reflected from the reflection portion 20 in a manner to be delivered to the pupil.

[0010] The frame portion 40 is a mechanism that fixes and supports the image emission portion 30 and the optical mechanism 10, and for example, can be a spectacle frame or the like.

[0011] The image emission portion 30 is a mechanism that emits augmented reality image light as image light corresponding to a virtual image, that is, an augmented reality image, and for example, can have a small display device that displays an augmented reality image on a screen to radiate augmented reality image light, and a collimator for collimating the image light radiated from the display device into parallel light.

[0012] The reflection portion 20 provides an augmented reality image by reflecting the augmented reality image light emitted from the image emission portion 30 toward the pupil 50 of the user.

[0013] Figure 1 The reflection portion 20 of the present embodiment is formed to a size smaller than the size of a general pupil of a person, that is, 8 mm or less, and when the reflection portion 20 is formed to be smaller than the pupil, the depth of field of the light incident to the pupil through the reflection portion 20 can be made close to infinity, that is, the depth of field can be made very deep.

[0014] The depth of field refers to a range recognized as in focus, and the deepening of the depth of field means that the focal distance of the augmented reality image also deepens. Therefore, when the depth of field deepens, even if the user changes the focal distance of the actual world while gazing at the actual world, the focus of the augmented reality image is always recognized as in focus regardless of this. This can be seen as a pinhole effect. Therefore, even if the user changes the focal distance while gazing at an actual object existing in the actual world, for the augmented reality image, a clear virtual image can always be seen.

[0015] Figure 2 and Figure 3 An augmented reality optical device 200 in which a plurality of reflection portions 20 are arranged is shown, Figure 2 is a side view of the augmented reality optical device 200, Figure 3 is a front view of the augmented reality optical device 200.

[0016] Figure 2 and Figure 3 is substantially the same as the augmented reality optical device 100 of Figure 1 , except that the reflection portion 20 is arranged in a plurality of arrays so as to be able to provide a wider angle of view.

[0017] Right now, Figure 2 and Figure 3 The reflective elements 20 in the augmented reality optical device 200 are arranged in a 4×9 array, such as... Figure 2 As shown, when the augmented reality optical device 200 is viewed from the side, the reflective portions 20 constituting the lower row are configured to be located closer to the pupil 50 so that each reflective portion 20 can receive the augmented reality image emitted from the image emission portion 30.

[0018] The image emission unit 30 receives augmented reality image information from an image playback device (not shown) such as a computer or smartphone, which allows the augmented reality image to be displayed on the image emission unit 30, and emits augmented reality image light corresponding to the augmented reality image information and transmits it to multiple reflective units 20.

[0019] Multiple reflectors 20 provide the user with an augmented reality image corresponding to the augmented reality image light by reflecting the augmented reality image light emitted from the image emission unit 30 to the pupil 50 of the user's eye.

[0020] The augmented reality optical device 200 with such multiple reflective parts 20 has the advantage of providing a wider viewing angle, but there is a problem of uneven brightness distribution in the augmented reality image provided to the user.

[0021] Figure 4 It is used to explain how users access... Figure 2 and Figure 3 The figure shows the problem of uneven brightness distribution in augmented reality images observed by an augmented reality optical device 200.

[0022] Figure 4 (A) shows a screen displaying an augmented reality image in the image emission unit 30. Figure 4 (B) shows the observation by the user through the augmented reality optical device 200. Figure 4 The augmented reality image of (A) is an observed image. It should be noted that, in Figure 4 In order to illustrate the principle of the present invention, the screen contrast is presented in an exaggerated manner.

[0023] like Figure 4 As shown in (A), an augmented reality image with uniform brightness distribution is displayed on the display section of the image emitting section 30. However, when such an augmented reality image is transmitted to the pupil 50 via multiple reflectors 20, as... Figure 4 As shown in (B), the user will observe an image with uneven brightness distribution.

[0024] Figure 1 The portion indicated by white in (B) is a region that appears brighter than the surrounding region due to light reflected from the adjacent reflecting portion 20.

[0025] Such unevenness of the luminance distribution is related to the shape, size, and arrangement structure of the reflecting portion 20, and the pupil diameter, the eye relief, the focal position of the eyeball, the horizontal position of the eyeball, and the like. Therefore, a problem that can occur in the augmented reality optical device 200 using a plurality of reflecting portions 20 is that the augmented reality image that the user finally observes is unclear or not clear due to the unevenness of the luminance distribution.

[0026] [Patent Document 1]

[0027] Korean Patent Publication No. 10-1660519 (Published on September 29, 2016) SUMMARY

[0028] PROBLEMS TO BE SOLVED BY THE INVENTION

[0029] To solve the problems as described above, the present application aims to provide an augmented reality optical device that provides an augmented reality image using a plurality of reflecting portions, that is, an image correction device for an augmented reality image that can provide an augmented reality image having a uniform luminance distribution by previously correcting the luminance distribution information of the augmented reality image, so as to be able to solve the problem of unevenness of the luminance distribution.

[0030] TECHNICAL SOLUTION

[0031] To solve the problems as described above, the present application provides an image correction device for augmented reality, which is combined with an optical device for augmented reality having an image emission section that emits augmented reality image light corresponding to an augmented reality image, an optical mechanism that transmits at least a part of actual object image light, which is image light emitted from an actual object, toward a pupil of a user, and a plurality of reflection sections that are arranged inside the optical mechanism to reflect the augmented reality image light emitted from the image emission section to be transmitted to the pupil of the user, characterized by comprising: a compensation function decision section that decides a compensation function for correcting luminance information of an observation image observed by the user through the optical device for augmented reality when an original augmented reality image is emitted from the image emission section; and a corrected image information generation section that generates corrected augmented reality image information based on the compensation function decided by the compensation function decision section and the original augmented reality image information, the image emission section emitting corrected augmented reality image light corresponding to the corrected augmented reality image information generated by the corrected image information generation section, and the plurality of reflection sections reflecting the corrected augmented reality image light emitted from the image emission section to be transmitted to the pupil of the user.

[0032] The compensation function decision section can decide the compensation function by a formula where α is a scaling factor, M(x, y) is a shape function of the reflection section, P(x, y) is an arrangement function of the reflection section, PSF(x, y) is an eye point spread function (PSF), and * is a convolution operator.

[0033] Further, the shape function M(x, y) of the reflection section can be decided by where x and y are coordinate values when a plane on which the reflection sections are arranged is expressed in a two-dimensional coordinate system, and r is a radius of the reflection section. 1 in a case where x and y are smaller than r, and 0 in other cases.

[0034] Further, the arrangement function P(x, y) of the reflection section can be a function that represents positions of centers of the plurality of reflection sections.

[0035] Further, the arrangement function P(x, y) of the reflection section can be decided by a formula where h is the number of reflection sections in the horizontal direction, w is the number of reflection sections in the vertical direction, w h is a pitch of the reflection sections in the horizontal direction, and w vis a pitch of the reflection portions in the longitudinal direction, δ is a Dirac delta function, and x and y are coordinate values of the reflection portions when a plane on which the reflection portions are arranged is expressed in a two-dimensional coordinate system.

[0036] Further, the arrangement function P(x, y) of the reflection portions can be determined by the equation where h is the number of reflection portions in the lateral direction, w is the number of reflection portions in the longitudinal direction, w h is a pitch of the reflection portions in the lateral direction, w v is a pitch of the reflection portions in the longitudinal direction, δ is a Dirac delta function, and x and y are coordinate values of the reflection portions when a plane on which the reflection portions are arranged is expressed in a two-dimensional coordinate system.

[0037] Further, the eye point spread function can be determined based on a diameter of a pupil, a reference image distance, and an exit pupil distance.

[0038] Further, the eye point spread function can be determined by where c is a blur diameter,

[0039] Further, the corrected image information generating section can generate the corrected augmented reality image information by the equation: corrected augmented reality image information = original augmented reality image information x compensation function.

[0040] Further, the image correction device can further include a depth camera that measures the reference image distance, and the compensation function determining section receives the reference image distance from the depth camera to calculate the eye point spread function.

[0041] Further, the image correction device can further include an eye tracking device that measures the diameter of the pupil and the exit pupil distance, and the compensation function determining section receives the diameter of the pupil and the exit pupil distance from the eye tracking device to calculate the eye point spread function.

[0042] Further, the compensation function determining section can receive relative position information of the pupil from the eye tracking device to calculate an offset value of the arrangement function of the reflection portions, and determine the arrangement function P(x, y) of the reflection portions based on the offset value.

[0043] Effects of the Invention

[0044] According to the present application, an augmented reality optical device using a plurality of reflection portions to provide an augmented reality image can be provided, that is, by previously correcting luminance distribution information of an augmented reality image, an augmented reality image having a uniform luminance distribution can be provided, so that an image correction device for an augmented reality image capable of solving a problem of non-uniform luminance distribution can be provided. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 2 is a diagram showing an augmented reality optical device 100 as disclosed in Patent Literature 1 filed by the present applicant.

[0046] Figure 3 and Figure 2 shows an augmented reality optical device 200 in which a plurality of reflection portions 20 are arranged, Figure 3 is a side view of the augmented reality optical device 200, Figure 4 is a front view of the augmented reality optical device 200.

[0047] Figure 2 is a diagram for explaining that a user observes an augmented reality image through the augmented reality optical device 200. Figure 3 and Figure 5 is a diagram showing a problem of non-uniform luminance distribution of an augmented reality image observed by the user through the augmented reality optical device 200.

[0048] Figure 6 is a diagram showing an image correction device 300 for an augmented reality image and the augmented reality optical device 200 according to an embodiment of the present application.

[0049] Figure 7 shows another example of an arrangement function of the reflection portion 20.

[0050] Figure 8 is a diagram showing an example of an eye point spread function.

[0051] Figure 9 is a diagram for explaining c, d a , d o , d m of the eye point spread function.

[0052] Figure 10 shows another example of the eye point spread function.

[0053] Figure 11 shows a principle of expressing a luminance distribution of an observation image observed by the user through a convolution operation of a shape function of the reflection portion 20, an arrangement function of the reflection portion 20, and the eye point spread function.

[0054] Figure 12 is a diagram for explaining a principle of the present application capable of obtaining an observation image having a uniform luminance distribution by previously correcting an observation image observed by the user.

[0055] Figure 5 is a view showing an observation image in a case where the image correction device 300 of the present application is applied, compared with an observation image in a case where the image correction device 300 is not applied. DETAILED DESCRIPTION

[0056] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0057] Figure 5 is a view showing the image correction device 300 of an embodiment of the present application and the augmented reality optical device 200.

[0058] Figure 2 The image correction device 300 of an embodiment of the present application (hereinafter referred to as "image correction device 300") generates corrected augmented reality image information based on original augmented reality image information, and transmits the generated corrected augmented reality image information to the image emission section 30 of the augmented reality optical device 200, thereby performing a function of providing a user with the corrected augmented reality image having a uniform brightness distribution through the augmented reality optical device 200.

[0059] The image correction device 300 includes a compensation function determination section 310 and a corrected image information generation section 320, which will be described later, and first, the structure and operation of the augmented reality optical device 200 to which the image correction device 300 of the present application is applied will be described schematically.

[0060] As described in the foregoing with reference to Figure 3 and Figure 5 The augmented reality optical device 200 has the optical mechanism 10, the plurality of reflection sections 20, and the image emission section 30, as described in the foregoing in the background art.

[0061] The optical mechanism 10 is a mechanism that transmits at least a part of the actual object image light, which is image light emitted from an actual object, toward the pupil 50 of the user's eye. The plurality of reflection sections 20 are mechanisms that are disposed inside the optical mechanism 10 to reflect the corrected augmented reality image light emitted by the image emission section 30 and transmit it to the pupil 50 of the user.

[0062] Among them, as described in the foregoing in the background art, the plurality of reflection sections 20 are each formed to have a size smaller than the average size of the pupil of a person, i.e., 8 mm or less, and more preferably 4 mm or less, so as to be able to obtain a pinhole effect by deepening the depth of field.

[0063] The image emission section 30 is a mechanism that receives the corrected augmented reality image information from the image correction device 300 and emits the corrected augmented reality image light corresponding thereto. The image emission section 30 can be constituted, for example, by a display device such as a small LCD and a collimator. Since such an image emission section 30 is not a direct object of the present application and is known in the art, detailed description is omitted.

[0064] In Figure 6 In the above, the plurality of reflecting sections 20 are illustrated as receiving the corrected augmented reality image light directly from the image emission section 30, but this is exemplary, and obviously, it can also be constituted so that the corrected augmented reality image light from the image emission section 30 is transmitted to the reflecting sections 20 by being totally reflected at least once on the inner face of the optical mechanism 10.

[0065] Further, as described above, the plurality of reflecting sections 20 are configured so that the reflecting sections 20 constituting the lower row when viewed from the side of the optical mechanism 10 are located closer to the pupil 50, but this is also exemplary, and of course, other configuration structures can also be had.

[0066] That is, the augmented reality optical device 200 to which the image correction device 300 of the present application is applied is not particularly limited in the specific configuration structure of the reflecting sections 20 or the optical path between the image emission section 30 and the pupil 50, and can be applied to all augmented reality optical devices 200 that transmit augmented reality image light to the pupil 50 of the user using a plurality of reflecting sections 20.

[0067] Next, the image correction device 300 will be described.

[0068] The image correction device 300 includes a compensation function decision section 310 and a corrected image information generation section 320.

[0069] The compensation function decision section 310 decides a compensation function for correcting the luminance information of the observation image that the user observes through the augmented reality optical device 200 when the original augmented reality image is emitted by the image emission section 30, and the corrected image information generation section 320 performs a function of generating corrected augmented reality image information based on the compensation function decided by the compensation function decision section 310.

[0070] First, the compensation function decision section 310 receives the original augmented reality image information, for example, from an image playback device (not illustrated) included in a device such as a computer or a smartphone, and decides a compensation function for correcting the luminance information of the observation image that the user observes when the original augmented reality image is emitted by the image emission section 30 based on this and transmits it to the corrected image information generation section 320.

[0071] Among them, the augmented reality image information is information that allows the augmented reality image to be emitted from the image emission part 30 as augmented reality image light, and is information for displaying the augmented reality image on a display part (not shown) included in the image emission part 30.

[0072] Generally, as for the information for displaying an image on a display part, when the image is composed of m x n pixels, a three-dimensional array of m x n x 3 composed of intensity values of each color of R, G, B of each pixel is composed. Among them, the intensity value of each color of R, G, B of each pixel corresponds to the brightness of each color of R, G, B of the corresponding pixel. The range of the intensity value is usually expressed in 8 bits in a manner of taking a value between 0 and 255, and by the combination of each color, 256 x 256 x 256 kinds of combinations can be had.

[0073] The augmented reality image information is also composed of a three-dimensional array of m x n x 3 composed of intensity values of each color of R, G, B of each pixel in a case where the corresponding augmented reality image is composed of m x n pixels, in a manner that the augmented reality image can be displayed on a display part included in the image emission part 30, and this can be labeled with a matrix.

[0074] On the other hand, in the present application, the "original" augmented reality image information refers to the image information of the augmented reality image in a state where the brightness information is not corrected by the video correction device 300 of the present application, and the "pre-compensated" augmented reality image information refers to the image information of the corrected augmented reality image having the brightness information corrected by the video correction device 300 of the present application.

[0075] The compensation function for correcting the brightness information of the observation video observed by the user when the original augmented reality image is emitted through the image emission part 30 can be determined by the following formula.

[0076]

[0077] Among them, α is a scaling factor, and is a value for normalizing the brightness of the corrected augmented reality image so as to correspond to the range that the display part of the image emission part 30 can present when the brightness of the corrected augmented reality image exceeds the range that the display part can present. This can vary depending on the maximum value of the corrected augmented reality image.

[0078] Further, M(x, y) is a shape function of the reflection portion 20, P(x, y) is an arrangement function of the reflection portion 20, PSF(x, y) is an eye point spread function (PSF), and * is a convolution operator.

[0079] Further, x and y are coordinate values of an x-axis and a y-axis when a plane on which the reflection portion 20 is arranged or a retina is expressed in a two-dimensional coordinate system.

[0080] The shape function M(x, y) of the reflection portion 20 is a function that mathematically represents the shape of the reflection portion 20. For example, when the reflection portion 20 is circular, it can be defined as follows.

[0081]

[0082] where x and y are coordinate values when a plane on which the reflection portion 20 is arranged is expressed in a two-dimensional coordinate system, and r is the radius of the reflection portion 20.

[0083] Further, 1 in a case of being smaller than r, and 0 in other cases.

[0084] The arrangement function P(x, y) of the reflection portion 20 is a function that describes the arrangement structure of the reflection portion 20 on a plane on which the reflection portion 20 is arranged, and is a function that represents the position of the center of each reflection portion 20 in a two-dimensional coordinate system of the plane on which the reflection portion 20 is arranged.

[0085] For example, when it is assumed that a plurality of reflection portions 20 are arranged in a two-dimensional grid shape, the arrangement function P(x, y) of the reflection portion 20 that represents the position of the center of each reflection portion 20 can be defined as follows.

[0086]

[0087] where h is the number of reflection portions 20 in the horizontal direction, w is the number of reflection portions 20 in the vertical direction, w h is the horizontal pitch of the reflection portions 20, w v is the vertical pitch of the reflection portions 20, and δ is a Dirac delta function. Further, x and y are coordinate values of each reflection portion 20 when a plane on which the reflection portion 20 is arranged is expressed in a two-dimensional coordinate system.

[0088] Figure 6 Another example of the arrangement function P(x, y) of the reflection portion 20 is shown.

[0089] Referring to Figure 7 , a plurality of reflection portions 20 are arranged in a two-dimensional grid shape of hexagons, and are arranged at intervals of w veach column interval w h / 2.

[0090] In this case, the arrangement function of the reflection portion 20 can be defined as follows.

[0091]

[0092] In the above formula, the sigma operation on the left side indicates the arrangement of the reflection portion 20 of the odd-numbered column from the left, and the sigma operation on the right side indicates the arrangement of the reflection portion 20 of the even-numbered column.

[0093] where h is the number of the reflection portion 20 in the lateral direction, w is the number of the reflection portion 20 in the longitudinal direction, w h is the interval of the reflection portion 20 in the lateral direction, w v is the interval of the reflection portion 20 in the longitudinal direction, and δ is the Dirac delta function. Further, x, y are coordinate values of each reflection portion 20 when a plane on which the reflection portion 20 is arranged is expressed in a two-dimensional coordinate system.

[0094] The eye point spread function (PSF(x, y)) is a two-dimensional spatial distribution function on the retinal plane indicating the intensity value of an image imaged on the retina when a point light source at a distance d m from the pupil 50 is observed.

[0095] Figure 7 is a graph showing an example of the eye point spread function.

[0096] Figure 8 The eye point spread function of the above formula can be defined by the following Gaussian distribution.

[0097]

[0098] where c is the blur diameter defined by

[0099] Further, d a is the diameter of the pupil 50, d o is the reference image distance (focal position of an object included in an image for augmented reality), d m is the eye relief (distance from the pupil 50 to the reflection portion 20), and x, y are coordinate values when the retina is expressed in a two-dimensional coordinate system.

[0100] Figure 8 is a graph for explaining the c, d a , d​o , d m of the figure.

[0101] Referring to Figure 8 , d a is the diameter of the pupil 50, d m is the eye relief (the distance from the pupil 50 to the reflecting portion 20).

[0102] Further, d o is the reference image distance, which refers to the distance between the position of the virtual object included in the image for augmented reality, i.e., the focal point position, and the pupil 50.

[0103] On the other hand, c is the blur diameter, which refers to the diameter of the circle of confusion when a point light source is projected onto the retina through the reflecting portion 20.

[0104] The formula of the above can be derived from the proportion of the triangle as shown in Figure 9 .

[0105] Figure 9 Another example of the ocular point spread function is shown.

[0106] It can be seen that the ocular point spread function of Figure 9 presents a distribution in which the luminance is highest at the position corresponding to the radius of the pupil 50 from the origin, slightly lower at the position corresponding to the center of the pupil 50, and sharply decreases at the positions outside the radius of the pupil 50.

[0107] Figure 7 The ocular point spread function of can be generalized to , where the subscripts of d a , d o , d m mean that the ocular point spread function varies depending on these values.

[0108] Such an ocular point spread function is a function determined by the characteristics of the human eye, and the detailed form of the ocular point spread function of each person can vary depending on the individual's visual acuity, age, eye condition, etc., but in general, the central portion has the form of a bright concentric circle, and the ocular point spread function of each person varies depending on the pupil diameter, the reference image distance, and the eye relief, but has the same value under the same conditions.

[0109] Figure 9 On the other hand, it is to be noted that, although Figure 7The ocular point spread function in the diagram is expressed as a one-dimensional function for ease of explanation, but it actually corresponds to the distribution function of the retinal surface, and is therefore a two-dimensional function that varies according to the position (x, y) of the retina.

[0110] also, Figure 9 and Figure 10 The eye point spread function shown is exemplary, but other eye point spread functions can certainly be used.

[0111] The compensation function can be determined by the shape function (M(x,y)) of the reflective part 20, the arrangement function (P(x,y)) of the reflective part 20, the dot spread function (PSF(x,y)) of the eyeball, and the scaling factor (α), based on the following principle.

[0112] The image observed by the user is an image emitted by the image emission unit 30, reflected by the reflection unit 20, and projected onto the retina through the pupil 50. Therefore, the brightness of the image observed by the user can be represented by the result of multiplying the value of the original augmented reality image by the convolution operation of the shape function (M(x,y)) of the reflection unit 20, the arrangement function (P(x,y)) of the reflection unit 20, and the dot spread function (PSF(x,y)).

[0113] Figure 10 This demonstrates the principle of expressing the brightness distribution of the observed image by the user through the convolution operation of the shape function of the reflective part 20, the arrangement function of the reflective part 20, and the eyeball dot spread function.

[0114] like Figure 10 As shown in (A), the original image has the same brightness distribution regardless of position, and the reflective part 20 is as follows: Figure 10 The width and spacing configuration shown in (B) is assumed to have the following characteristics: Figure 10 When the eye point spread function is shown in (C), it can be seen that the brightness of the observed image will be as follows: Figure 10 The value shown in (D) is the result of the convolution operation of the shape function (M(x,y)) of the reflector 20, the arrangement function (P(x,y)) of the reflector 20, and the eye point spread function (PSF, PSF(x,y)) multiplied by the original image.

[0115] wherein the multiplication of the original image by the result of the convolution operation of the shape function of the reflection portion 20, the arrangement function of the reflection portion 20, and the eye point spread function means that, when the original image is composed of m x n pixels as described above, a multiplication operation is performed on the result of the m x n x 3 pixel matrix composed of the intensity values of each color of R, G, and B of each pixel by M(x, y) * P(x, y) * PSF(x, y) with respect to the components of the row and column.

[0116] That is, the luminance distribution of the observation image can be obtained by the following equation.

[0117] Luminance distribution of observation image = Original image x M(x, y) * P(x, y) * PSF(x, y)

[0118] On the other hand, unlike the luminance distribution of the original image in (A) of Figure 10 , the luminance distribution is uniform by the convolution operation in (B) and (C) of Figure 10 , that is, M(x, y) * P(x, y) * PSF(x, y). That is, as shown in (D) of Figure 11 , the final observation image exhibits a non-uniform luminance distribution depending on the position.

[0119] In view of such a principle, if the augmented reality image information transmitted to the image emission portion 30 can be corrected in advance so that the luminance distribution of the final observation image can be made uniform, a final observation image having a uniform luminance distribution can be obtained.

[0120] Figure 11 The present application is explained for the principle that, by correcting the observation image observed by the user in advance, an observation image having a uniform luminance distribution can be obtained.

[0121] Figure 10 (B) and (C) of Figure 10 (B) and (C) of Figure 11 , the difference from Figure 11 (A) is that a pre-compensated image (Pre-compensated image) having a luminance distribution as shown in (A) of

[0122] As described in Figure 11 , since the luminance distribution of the observation image can be obtained by the pre-compensated image x M(x, y) * P(x, y) * PSF(x, y), the final observation image of the pre-compensated image having a luminance distribution as shown in (A) of Figure 11 will have a uniform luminance distribution as shown in (D) of Figure 11 .

[0123] Based on such a principle, the compensation function determination portion 310 determines the compensation function by the equation The compensation function is determined.

[0124] On the other hand, the shape and arrangement of the reflection section 20 are determined in advance at the time of manufacturing the optical device for augmented reality 200, and thus the shape function and the arrangement function of the reflection section 20 are values that can be known in advance, and the eye point spread function is a value that is determined by the diameter of the pupil 50, i.e., d a , d o , and d m are values that are set in advance at the time of designing the optical device for augmented reality 200 and are not changed separately.

[0125] Therefore, the compensation function can be calculated by measuring only the diameter of the pupil 50 of the user of the optical device for augmented reality 200.

[0126] The diameter of the pupil 50 of the user varies according to the depth information of the main object displayed in the augmented reality image, the brightness information of the original augmented reality image, and the like, and since the depth information of the main object and the brightness information of the original augmented reality image are values that can be known in advance, if the average value of the diameter of the pupil 50 of the user corresponding to these values is obtained in advance through experiments, the compensation function can be easily obtained.

[0127] Next, the corrected image information generating section 320 will be described.

[0128] The corrected image information generating section 320 functions to generate corrected augmented reality image information based on the compensation function determined by the compensation function determining section 310 and the original augmented reality image information and to transfer the corrected augmented reality image information to the image projecting section 30.

[0129] Here, the corrected augmented reality image information can be obtained by the following equation.

[0130] Corrected augmented reality image information = Original augmented reality image information x Compensation function

[0131] As described above, it means that when the original augmented reality image is composed of m x n pixels, the multiplication operation is performed on the m x n x 3 pixel matrix composed of the intensity values of each color of R, G, and B of each pixel and the compensation function by the components of the row and column.

[0132] The brightness distribution of the corrected augmented reality image based on the corrected augmented reality image information thus generated is the same as that represented by (A) of Figure 11 .

[0133] The image projecting section 30 that receives such corrected augmented reality image information projects the corrected augmented reality image light corresponding thereto, and the reflection section 20 reflects the corrected augmented reality image light to be transferred to the pupil 50 to provide the corrected augmented reality image to the user, and since this is the same as Figure 12the reflection part 20 in (B), (C) and the convolution operation of the eye point spread function, the corrected augmented reality image that the user finally observes has a uniform brightness distribution as shown in (D) of Figure 12

[0134] Figure 4 is a diagram showing the observation image in the case where the image correction device 300 according to the present application is applied, compared with the observation image in the case where the image correction device 300 is not applied.

[0135] In (A) of Figure 10 , (A) shows the state where the original augmented reality image is displayed on the image emission part 30, (B) shows the observation image that the user observes when the original augmented reality image is transmitted to the pupil 50 through the augmented reality optical device 200 without applying the image correction device 300. As can be seen, the observation image has a non-uniform brightness distribution as shown in (B) of Figure 12 and Figure 12 (D).

[0136] On the other hand, Figure 12 (C) shows the state where the corrected augmented reality image corresponding to the corrected augmented reality image information generated by the image correction device 300 is displayed on the image emission part 30, (D) shows the final observation image that the user observes when the corrected augmented reality image is transmitted to the pupil 50 through the augmented reality optical device 200. As can be seen in (D) of ​ , the observation image of the corrected augmented reality image that corrects the brightness information by the image correction device 300 has a uniform brightness distribution. Therefore, the user can be provided with a clearer and more distinct augmented reality image compared with the case of (B) of ​ .

[0137] On the other hand, as mentioned above, the eye point spread function changes with the diameter d a of the pupil 50, the exit pupil distance d m , and the reference image distance d o , wherein when the reference image distance d o changes, in order to measure it, a depth camera can be used to dynamically measure the distance between the position of the virtual object and the pupil 50.

[0138] In addition, when an eye tracking device is used, the diameter d a of the pupil 50 and the exit pupil distance d m can be dynamically measured.

[0139] ​Further, in the above-described embodiments, the arrangement function (P(x, y)) of the reflection portion 20 is described with respect to a fixed position of the pupil 50, however, when the position of the pupil 50 moves, the arrangement function (P(x, y)) of the reflection portion 20 can be shifted. At this time, using an eye tracking device, the relative position of the pupil 50 can be known in real time, and thus the shift of the arrangement function (P(x, y)) of the reflection portion 20 can be corrected, which can be applied to any arrangement function (P(x, y)) of the reflection portion 20.

[0140] In this case, the arrangement function of the reflection portion 20 representing the position of the center of the reflection portion 20 as described above can be modified as follows.

[0141]

[0142] where x o , y o are the offset values in the x-axis direction and the y-axis direction, respectively.

[0143] Further, when a depth camera and an eye tracking device are used together, the diameter d a of the pupil 50, the exit pupil distance d m , and the reference image distance d o , the relative position of the pupil 50 can all be measured in real time.

[0144] Therefore, for example, in the case where the augmented reality image is a video and the reference image distance varies, and in the case where the diameter of the pupil frequently changes due to a change in brightness, the present application can also be easily applied.

[0145] Since such a depth camera and an eye tracking device are known in the art, detailed descriptions thereof are omitted here.

[0146] Although the present application has been described above with reference to preferred embodiments thereof, the present application is not limited to the above-described embodiments, and various modifications and changes can of course be made thereto.

Claims

1. An image correction apparatus for augmented reality images, combined with an augmented reality optical device, the augmented reality optical device comprising an image emitting section that emits augmented reality image light corresponding to the augmented reality image, an optical mechanism that allows at least a portion of the actual object image light, which is image light emitted from an actual object, to pass through a user's pupil, and a plurality of reflective sections disposed inside the optical mechanism to reflect the augmented reality image light emitted from the image emitting section and transmit it to the user's pupil, the image correction apparatus for augmented reality images being characterized in that it includes: The compensation function determination unit determines a compensation function for correcting the brightness information of the observed image viewed by the user through the augmented reality optical device when the original augmented reality image is emitted by the image emission unit; and The image information generation unit generates corrected augmented reality image information based on the compensation function determined by the compensation function determination unit and the original augmented reality image information. The compensation function determination part is determined by mathematical formulas. The compensation function is determined, where (x, y) are the coordinate values ​​of the plane with the reflector in a two-dimensional coordinate system, and α is a scaling factor. It is the shape function of the reflector. It is the arrangement function of the reflectors. It is the eyeball spread function, and * is the convolution operator. The image emission unit emits corrected augmented reality image light corresponding to the corrected augmented reality image information generated by the corrected image information generation unit, and the plurality of reflective units reflect the corrected augmented reality image light emitted by the image emission unit and transmit it to the user's pupil.

2. The image correction apparatus for augmented reality images according to claim 1, characterized in that, The arrangement function of the reflective part It is a function representing the position of the center of multiple reflectors.

3. The image correction apparatus for augmented reality images according to claim 2, characterized in that, The arrangement function of the reflective part From the formula The determination is made, where h is the number of transverse reflectors and w is the number of longitudinal reflectors. h It is the lateral spacing of the reflector, w v δ is the longitudinal spacing of the reflectors, δ is the Dirac function, and x and y are the coordinate values ​​of each reflector when the plane containing the reflectors is represented in a two-dimensional coordinate system.

4. The image correction apparatus for augmented reality images according to claim 2, characterized in that, The arrangement function of the reflective part From the formula The determination is made, where h is the number of transverse reflectors and w is the number of longitudinal reflectors. h It is the lateral spacing of the reflector, w v δ is the longitudinal spacing of the reflectors, δ is the Dirac function, and x and y are the coordinate values ​​of each reflector when the plane containing the reflectors is represented in a two-dimensional coordinate system.

5. The image correction apparatus for augmented reality images according to claim 1, characterized in that, The eye dot spread function is determined based on the pupil diameter, the reference image distance, and the exit pupil distance.

6. The image correction apparatus for augmented reality images according to claim 5, characterized in that, The eyeball diffusion function is composed of The decision is made, where c is the fuzzy diameter. , where d a It is the diameter of the pupil, d o The reference image distance, i.e., the distance between the focal position of an object included in the augmented reality image and the pupil, d m It is the exit pupil distance, that is, the distance between the pupil and the reflex area. x and y are the coordinate values ​​of the retina in a two-dimensional coordinate system.

7. The image correction apparatus for augmented reality images according to claim 1, characterized in that, The corrected image information generation unit generates corrected and enhanced practical image information using the following formula: Corrected augmented reality image information = Original augmented reality image information × Compensation function.

8. The image correction apparatus for augmented reality images according to claim 5, characterized in that, The image correction device also includes a depth camera that measures the distance to a reference image. The compensation function determination unit receives a reference image distance from the depth camera to calculate the eye point spread function.

9. The image correction apparatus for augmented reality images according to claim 5 or 8, characterized in that, The image correction device also includes an eye-tracking device that measures the diameter of the pupil and the exit pupil distance. The compensation function determination unit receives the pupil diameter and exit pupil distance from the eye-tracking device to calculate the eye point spread function.

10. The image correction apparatus for augmented reality images according to claim 9, characterized in that, The compensation function determination unit receives the relative position information of the pupil from the eye-tracking device to calculate the offset value of the arrangement function of the reflector, and determines the arrangement function of the reflector based on this. .

Images