Compensation Method for Distortion of HUD Image Caused by Windshield

Abstract

One aspect of the present invention relates to a method and apparatus for compensating for distortion of a head-up display image (HUD image) to display a correct 3D image on a 3D HUD by compensating for distortion caused by a windshield. A step of preparing a marker image equipped with a marker for detection at a predetermined location on a 3D display panel; a step of preparing a view image in which subpixels are represented and assigned a setting value to have a specific period; a step of obtaining an image through an optical simulation or an actual camera by reflecting the marker image or / and the view image onto a windshield; and It may be configured to include one or more steps of obtaining parameters of a regression equation for calculating warping or / and view mapping from the obtained image. According to the present invention, a HUD device capable of generating a correct 3D HUD image without distortion caused by the windshield can be provided by performing a warping process that compensates for shape distortion of a virtual object entering the driver's field of vision through the windshield in real time and / or a view mapping process that compensates for path distortion of light entering the driver's field of vision through the windshield.

Description

Technology Field

[0001] One aspect of the present invention relates to a method and apparatus for compensating for distortion of a head-up display image (HUD image) to display a correct 3D image in a 3D HUD by compensating for distortion caused by a windshield. Background Technology

[0002] The content described in this section merely provides background information regarding embodiments of the present invention and does not constitute prior art.

[0003] Head-Up Displays (HUDs) have been commercialized in 2D and are built into many vehicles. With the use of a large-screen 3D HUD, drivers can view the instrument panel, road markings, and navigation more comfortably and safely.

[0004] While 3D HUDs can be implemented in various ways, one approach is to consider a form where a 3D display is reflected onto the windshield. The method of playing images on existing 3D displays (Lightfield rendering, which allows different images to be viewed depending on the direction) is well known.

[0005] However, when viewing a 3D display reflected off a windshield, because the windshield is curved, not only are the shapes of objects within the video being played on the 3D display distorted, but the path of light is also bent in a direction different from the original, resulting in a problem where the correct 3D image cannot be viewed.

[0006] Therefore, in order to view correct 3D images, it is urgent to develop a distortion compensation method that compensates for shape distortion of objects within the image and also compensates for the path of light.

[0007] The aforementioned background technology is technical information that the inventor possessed or acquired during the process of deriving the embodiments of the present invention, and it cannot be considered as prior art disclosed to the general public prior to the filing of the embodiments of the present invention. The problem to be solved

[0008] Accordingly, one aspect of the present invention is proposed to solve the aforementioned problems, and the objective of the present invention is to provide a distortion compensation method that compensates for the shape distortion of an object in an image and also compensates for the path of light so that a correct 3D image can be viewed without distortion caused by a windshield.

[0009] The distortion compensation method presented in the present invention can be applied to windshields of various vehicle types and is a method for displaying a correct 3D image without distortion in a HUD using a 3D display capable of lightfield rendering with a parallax barrier, a lenticular lens, or a similar structure.

[0010] In addition, another objective of the present invention is to provide a HUD device that performs the aforementioned distortion compensation method, and to provide a computer program stored on a medium to execute the aforementioned distortion compensation method in combination with hardware.

[0011] The technical problems that the present invention aims to solve are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0012] To achieve the above-mentioned objective, one aspect of the present invention (first embodiment) comprises a marker image preparation step of preparing a marker image including a plurality of markers displayed at a predetermined location on a display panel;

[0013] An image acquisition step of obtaining an image by reflecting the above marker image through a windshield and then obtaining an image through optical simulation or camera capture; and

[0014] A method for compensating for distortion of a HUD image caused by a windshield can be provided, comprising one or more steps of: generating a warping function that calculates warping from the above image; and generating a warping function.

[0015] According to an embodiment, in the marker image preparation step, the marker image may be characterized by including an image in which a plurality of markers are formed.

[0016] According to an embodiment, each of the plurality of markers may be characterized by being arranged in a grid shape with a constant spacing.

[0017] According to an embodiment, the optical simulation or camera shooting may be characterized by being performed at a plurality of viewpoints.

[0018] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eyebox.

[0019] According to an embodiment, each of the plurality of points may be characterized as being arranged in a grid shape within the eyebox at regular intervals.

[0020] According to an embodiment, the warping function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shot and a pixel position corresponding to a marker of the marker image as input parameters, and a compensated pixel position corresponding to a distortion-compensated image as output parameters.

[0021] According to an embodiment, the compensated pixel position may be characterized by being implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0022] According to an embodiment, the warping may be characterized by compensating for the shape distortion of a virtual object visible to the user's eye through the windshield.

[0023] According to an embodiment, it may be characterized by including a view image preparation step for preparing a view image.

[0024] According to an embodiment, the image acquisition step may be characterized by acquiring an image through optical simulation or camera shooting after reflecting the marker image and the view image through the windshield, respectively.

[0025] According to an embodiment, it may be characterized by including a mapping function generation step for generating a mapping function that calculates a view mapping from the image.

[0026] According to an embodiment, the mapping function may be characterized by taking an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera capture and a pixel position corresponding to a marker of the marker image as input parameters, and taking a view number visible to the user's eye as an output parameter.

[0027] According to an embodiment, the view number visible to the user may be characterized by being implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0028] According to an embodiment, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0029] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific period.

[0030] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths tilted in different directions by a parallax barrier or a lenticular lens.

[0031] According to an embodiment, the period and the degree of inclination may be characterized by being determined by the parallax barrier or the lenticular lens.

[0033] Another aspect of the present invention (a second embodiment) relates to a method for compensating for distortion of a head-up display image caused by a windshield, wherein

[0034] View image preparation step for preparing the view image;

[0035] An image acquisition step of obtaining an image by reflecting the above view image through a windshield and then obtaining an image through optical simulation or camera shooting; and

[0036] A method for compensating for distortion of a HUD image caused by a windshield may be provided, comprising one or more of the following steps: a mapping function generation step for generating a mapping function that calculates a view mapping from the above image.

[0037] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific period.

[0038] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths tilted in different directions by a parallax barrier or a lenticular lens.

[0039] According to an embodiment, the period and the degree of inclination may be characterized by being determined by the parallax barrier or the lenticular lens.

[0040] According to an embodiment, the optical simulation or camera shooting may be characterized by being performed at a plurality of viewpoints.

[0041] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eyebox.

[0042] According to an embodiment, each of the plurality of points may be characterized as being arranged in a grid shape within the eyebox at regular intervals.

[0043] According to an embodiment, the mapping function may be characterized by taking an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera capture and a pixel position corresponding to a marker of the marker image as input parameters, and taking a view number visible to the user's eye as an output parameter.

[0044] According to an embodiment, the view number visible to the user may be characterized by being implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0045] According to an embodiment, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0047] Another aspect of the present invention (third embodiment) relates to a method for compensating for distortion of a head-up display image caused by a windshield, wherein

[0048] Marker image preparation step for preparing a marker image;

[0049] View image preparation step for preparing the view image;

[0050] An image acquisition step of acquiring an image by reflecting the marker image and the view image through a windshield, respectively, and then acquiring an image through optical simulation or camera capture;

[0051] A method for compensating for distortion of a HUD image caused by a windshield can be provided, comprising one or more of the following steps: a function generation step of generating a function that calculates warping and view mapping from the above image.

[0052] According to an embodiment, in the marker image preparation step, the marker image may be characterized by including an image in which a plurality of markers are formed.

[0053] According to an embodiment, each of the plurality of markers may be characterized by being arranged in a grid shape with a constant spacing.

[0054] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific period on a black background.

[0055] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths tilted in different directions by a parallax barrier or a lenticular lens.

[0056] According to an embodiment, the period and the degree of inclination may be characterized by being determined by the parallax barrier or the lenticular lens.

[0057] According to an embodiment, the optical simulation or camera shooting may be characterized by being performed at a plurality of viewpoints.

[0058] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eyebox.

[0059] According to an embodiment, the function may be characterized by including a first function for calculating the warping and a second function for calculating the view mapping.

[0060] According to an embodiment, the first function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera capture and a pixel position corresponding to a marker of the marker image as input parameters, and a compensated pixel position corresponding to a distortion-compensated image as an output parameter.

[0061] According to an embodiment, the compensated pixel position may be characterized by being implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0062] According to an embodiment, the second function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera capture and a pixel position corresponding to a marker of the marker image as input parameters, and a view number visible to the user's eye as an output parameter.

[0063] According to an embodiment, the view number visible to the user may be characterized by being implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0064] According to an embodiment, the warping may be characterized by compensating for the shape distortion of a virtual object visible to the user's eye through the windshield.

[0065] According to an embodiment, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0066] Another aspect of the present invention may provide a HUD device that applies the method described above.

[0067] In addition, another aspect of the present invention may provide a computer program stored on a medium to execute the aforementioned method in combination with hardware. Effects of the invention

[0068] As described above, according to one embodiment of the present invention, a method for compensating for distortion of a HUD image caused by a windshield is provided, which compensates for shape distortion of an object in the image and also compensates for the path of light, so that a correct 3D image can be viewed without distortion caused by the windshield.

[0069] In addition, the method for compensating for distortion of HUD images caused by a windshield presented in the present invention can be applied to windshields of various vehicle types and has the effect of displaying correct 3D images without distortion in a HUD using a 3D display capable of lightfield rendering with a parallax barrier, a lenticular lens, or a similar structure.

[0070] The effect of the distortion compensation method according to the present invention is that, although the degree of distortion of shape or light path varies depending on the viewing angle, it is possible to ensure that an image is seen upright on a specific plane in space regardless of the viewing angle, and that the image is accurately directed to the left and right eyes. If the distorted shape differs between the two eyes, it becomes difficult to perceive it as a single object; furthermore, if the image is not accurately projected to each eye, the 3D image cannot be viewed properly and may even cause dizziness. The distortion compensation method according to the present invention can resolve these problems.

[0071] In addition, the distortion compensation method according to the present invention has the effect of having a small delay time because the execution time is short, since warping and view mapping can be performed in a parallel computing device and both can be performed simultaneously.

[0072] In addition, the present invention has various effects, such as excellent versatility depending on the embodiment, and such effects can be clearly confirmed in the description of the embodiment below. Brief explanation of the drawing

[0073] The following drawings attached to this specification illustrate an embodiment of the present invention and serve to further enhance understanding of the technical concept of the present invention together with the detailed description of the invention described above; therefore, the present invention should not be interpreted as being limited only to the matters described in such drawings. FIG. 1 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a first embodiment of the present invention. Figure 2 shows that in the method for compensating for distortion of the HUD image caused by the windshield of Figure 1, a view image preparation step and a mapping function generation step for generating a mapping function are additionally shown. FIG. 3 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a second embodiment of the present invention. Figure 4 shows that in the method for compensating for distortion of the HUD image caused by the windshield of Figure 3, a marker image preparation step and a warping function generation step for generating a warping function are additionally shown. FIG. 5 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a third embodiment of the present invention. Figure 6 shows a display capturing a result reflected through a windshield with a camera, according to an embodiment, of a marker image generated on a panel. FIG. 7 shows a marker image according to one embodiment of the present invention. Figure 8 shows a distorted image seen from a virtual driver's line of sight by generating the marker image of Figure 7 on a panel and reflecting it onto a windshield. Figure 9 shows the distorted image of Figure 8 and the target image corrected through warping superimposed. FIG. 10 shows a view image according to one embodiment of the present invention. Figure 11 shows a view image in which the subpixels can be viewed separately by actually displaying them on a screen. FIG. 12 shows an eye box according to one embodiment of the present invention. Figure 13 shows a view image by a specific pixel when a pixel with a specific number is turned on on the panel. Figure 14 shows an example of the view number of each pixel of a display panel. FIG. 15 shows an example of a view image, specifically view image 5. FIG. 16 shows an example of a view image, representing view image 1. FIG. 17 shows an example of a view image, specifically view image number 9. FIG. 18 shows the light path for each view image according to one embodiment of the present invention. FIG. 19 illustrates a view mapping method according to one embodiment of the present invention. Specific details for implementing the invention

[0074] The advantages and features of the present invention, and the methods for achieving them, will become clear by referring to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, detailed descriptions of related known technologies are omitted if it is determined that such detailed descriptions may obscure the essence of the present invention.

[0075] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0076] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Terms such as “first,” “second,” etc., may be used to describe various components, but the components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0077] Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical or corresponding components are given the same reference numerals, and redundant descriptions thereof are omitted.

[0078] FIG. 1 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a first embodiment of the present invention. FIG. 2 illustrates the method for compensating for distortion of a HUD image caused by a windshield according to FIG. 1, with additional steps for preparing a view image and generating a mapping function.

[0079] Referring to FIGS. 1 and 2, one aspect of the present invention (first embodiment) comprises a marker image preparation step (S100) of preparing a marker image including a plurality of markers (110) displayed at a predetermined location on a display panel;

[0080] An image acquisition step (S110) of obtaining an image by reflecting the above marker image through a windshield and then obtaining an image through optical simulation or camera shooting; and

[0081] A method for compensating for distortion of a HUD image caused by a windshield can be provided, comprising one or more steps of: generating a warping function (S120) that generates a warping function that calculates warping from the above image.

[0082] Depending on the embodiment, a view image preparation step (S130) for preparing a view image may be additionally included. In this case, depending on the embodiment, a mapping function generation step (S140) for generating a mapping function that calculates a view mapping from the image may be additionally included.

[0083] FIG. 3 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a second embodiment of the present invention. FIG. 4 illustrates the method for compensating for distortion of a HUD image caused by a windshield according to FIG. 3, with additional steps for preparing a marker image and generating a warping function.

[0084] Referring to FIGS. 3 and 4, a second embodiment of the present invention relates to a method for compensating for distortion of a head-up display image caused by a windshield, wherein

[0085] A view image preparation step (S200) for preparing a view image;

[0086] An image acquisition step (S210) of obtaining an image by reflecting the above view image through a windshield and then obtaining an image through optical simulation or camera shooting; and

[0087] A method for compensating for distortion of a HUD image caused by a windshield can be provided, comprising one or more steps including: a mapping function generation step (S220) for generating a mapping function that calculates a view mapping from the above image.

[0088] According to an embodiment, a marker image preparation step (S230) for preparing a marker image including a plurality of markers (110) displayed at a predetermined location on a display panel may be included. Additionally, according to an embodiment, a warping function generation step (S240) for generating a warping function that calculates warping from the image may be additionally included.

[0089] FIG. 5 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a third embodiment of the present invention.

[0090] Figure 6 shows a picture of a marker image generated on a display panel according to an embodiment being captured by a camera after being reflected through a windshield.

[0091] In the present invention, the display panel (210) may be a component installed in the HUD device (200). The HUD device (200) may include a device that emits a light containing an image to show the image to the driver, installed in a vehicle, etc. Here, the result is viewed from the viewpoint (A) of the camera. According to another embodiment, the result may be implemented by calculation through optical simulation instead of a camera.

[0092] FIG. 7 shows a marker image according to one embodiment of the present invention.

[0093] Referring to FIGS. 5, 6 and 7, the third embodiment of the present invention relates to a method for compensating for distortion of a head-up display image caused by a windshield (10), wherein

[0094] A marker image preparation step (S300) for preparing a marker image (100, marker image);

[0095] A view image preparation step (S310) for preparing a view image;

[0096] An image acquisition step (S320) in which the marker image (100) and the view image are each reflected through the windshield (10) and then an image is obtained through optical simulation or camera (300) shooting;

[0097] A method for compensating for distortion of a HUD image caused by a windshield (10) can be provided, comprising one or more of the following steps: a function generation step (S330) for generating a function that calculates warping and view mapping from the above image.

[0098] The third embodiment includes both a marker image (100) and a view image, the first embodiment describes a method for compensating for distortion of a HUD image including a warping process related to the marker image (100), and the second embodiment describes a method for compensating for distortion of a HUD image including a mapping process related to the view image.

[0099] Hereinafter, the third embodiment will be primarily described in this specification. Furthermore, since the first and second embodiments share substantially the same technical concept as the third embodiment, the description of the third embodiment will suffice to describe them, and a detailed description will be omitted.

[0100] Referring to FIG. 5, a method for compensating for distortion of a HUD image caused by a windshield (10) according to one embodiment of the present invention is a method for compensating for distortion of a HUD image (head-up display image) caused by a windshield (10, Windshield),

[0101] A marker image preparation step (S300) for preparing a marker image (100); a view image preparation step (S310) for preparing a view image; an image acquisition step (S320) for acquiring an image by reflecting the marker image (100) and the view image through a windshield (10) respectively, and then capturing the image through optical simulation or camera (300); and

[0102] It may be configured to include a function generation step (S330) for generating a function that calculates warping and view mapping from the above image.

[0103] In this specification, the warping may include compensating for shape distortion of a virtual object visible to the user's eye through the windshield (10). Additionally, the view mapping may include compensating for path distortion of light entering the user's eye through the windshield (10). Here, the virtual object may refer to a 2D or 3D image that the user can see through the windshield (10), which is an image generated on the display panel (210) of the HUD device (200) according to the embodiment.

[0104] Referring to FIG. 7, according to an embodiment, in the marker image preparation step (S300), the marker image (100) may include an image in which a plurality of markers (110) are formed. According to an embodiment, the marker image (100) may include an image in which a plurality of markers (110) are arranged as white points on a black background. According to an embodiment, the marker image (100) may be electronically implemented by assigning a setting value corresponding to the marker (110) to a pixel, or it may be mechanically or chemically implemented by a film, etc., on which the marker (110) is displayed.

[0105] Here, each marker (110) implemented in the marker image (100) has a position coordinate. Therefore, the user can know the position information of the corresponding marker (110) in the marker image (100) based on the aforementioned position coordinate. It is preferable to prepare multiple markers (110). If multiple markers (110) are arranged as densely as possible, the accuracy of distortion correction can be improved. According to an embodiment, each of the multiple markers (110) displayed in the marker image (100) can be arranged in a grid form with a constant spacing. By arranging the multiple markers (110) in a grid form with a constant spacing in this way, the accuracy of distortion correction caused by the windshield (10) can be improved. Here, being arranged in a grid form may include arranging the multiple markers (110) at right angles in the horizontal and vertical directions at constant intervals.

[0106] Generally, when the HUD has a flat display panel (210), it is desirable to place markers (110) so that they can fill the entire area of ​​the panel (210). It is even more desirable if multiple markers (110) cover most of the area of ​​the panel (210) and have a constant period.

[0107] The marker (110) can serve as a kind of reference point. When the marker image (100) with the marker (110) displayed is implemented on the panel (210) of the HUD, the implemented marker image (100) can be reflected through the windshield (10) and enter the user's eyes. Here, the user may include, for example, the driver of a vehicle in which the HUD is used.

[0108] FIG. 8 shows a distorted image (150) that is seen from a virtual driver's line of sight by generating the marker image of FIG. 7 on a panel (210) and reflecting it onto a windshield, and FIG. 9 shows the distorted image of FIG. 8 and the target image (160) corrected through warping superimposed.

[0109] In this specification, reference numeral 150 in FIG. 9 is indicated by enclosing a blue marker in parentheses in FIG. 9, but what is intended in this specification is to indicate a distorted image, which is a marker image in which the markers are displayed, and likewise reference numeral 160 in FIG. 9 is indicated by enclosing a red marker in parentheses in FIG. 9, but what is intended in this specification is to indicate a target image, which is a marker image in which the red markers are displayed.

[0110] Here, the distorted image (150) may refer to a marker image (100) in which the positions of the markers (110) appear distorted as the marker image (100) displayed on the panel (210) is reflected on the windshield according to the embodiment.

[0111] Here, the target image (160) may refer to an image that appears without distortion on the windshield (10) when viewed from the perspective of a driver sitting in the driver's seat. This target image (160) may refer to a warped image.

[0112] To further explain for better understanding, in order for the HUD device (200) to implement the target image (160), the marker image (100) generated on the panel (210) must exist in a distorted form. That is, if there is no distortion in the marker image generated on the panel from the beginning, the image that is reflected by the windshield and seen from the driver's perspective will be a distorted image caused by the windshield (10). Conversely, if the image seen from the driver's perspective is a correct image without distortion, the marker image on the panel (210) of the HUD device (200) that generates and projects the image must be a distorted image from the beginning in order to display the correct image that is the result reflected by the windshield (10). Here, the correct image is the target image (160).

[0113] The marker image (100) reflected through the windshield (10) is distorted due to the curved surface of the windshield (10) and enters the user's eyes, so a problem arises in which the user cannot see the correct 2D or 3D image. Therefore, the distortion of the marker image (100) must be corrected, and in this specification, correcting the distortion of the marker image (100) is defined as warping. According to the embodiment, in this specification, warping may mean correcting the shape distortion of a virtual object in 2D or 3D implemented through the windshield (10).

[0114] FIG. 10 shows a view image. Referring to FIG. 10, a view image according to one embodiment of the present invention shows an image with subpixels turned on with a specific period on a black background. The image of FIG. 10 is an image generated when the subpixel structure of the display is in RGB order.

[0115] Figure 11 shows a view image in which the subpixels can be viewed separately by actually displaying them on a screen.

[0116] According to an embodiment, the view image may include an image with subpixels that are turned on with a specific period on a black background. According to an embodiment, among the subpixels, subpixels belonging to the same period may have light paths tilted in different directions by a parallax barrier or a lenticular lens.

[0117] According to the embodiment, the degree to which the specific period and light path described above are tilted may be determined by a parallax barrier or lenticular lens formed on the panel (210).

[0118] Here, the number of views in the view image can be arbitrarily predetermined by a setter. The setter mentioned herein may refer to a designer who designs a method for compensating for distortion of a HUD image according to the present invention.

[0119] FIG. 12 shows an eye box according to one embodiment of the present invention.

[0120] According to the embodiment, optical simulation or camera (300) shooting may be performed at multiple viewpoints (A, view point). Additionally, the multiple viewpoints (A) may include a viewpoint (A) viewed from each of the multiple points (P) located within the virtual eyebox (170). For example, each of the multiple points (P) is represented as 3*3*3 in the embodiment of this specification, but this is exemplary, and in reality, once the transformation function (first function, second function) is determined, the value between the points (P) and the points (P) (image information, transformation function information, etc.) can also be calculated by a method such as interpolation.

[0121] For example, referring to FIGS. 8 and 9, when a marker image (100) is displayed on a panel (210) and reflected onto a windshield (10), a distorted marker image, i.e., a distorted image (150), is generated due to the curved surface of the windshield (10). This distorted image (150) can be captured by a camera (300) inside an eyebox (170) to obtain an image, or an image can be obtained by performing an optical simulation. Here, the position coordinates of a plurality of markers (110) on the distorted image (150) may be identical to the position coordinates of a pixel corresponding to the marker image (100) appearing on the panel (210) of the HUD. These pixel position coordinates may correspond to the position coordinates of the marker (110) in the marker image (100). In this specification, the position coordinates of the pixel position corresponding to the position coordinates of the marker image (100) are indicated as Px and Py.

[0122] In this case, an image reflected on the windshield (10) at multiple points (P) within the eyebox (170) can be captured by a camera (300) or an optical simulation can be performed to generate an image reflected on the windshield (10) at multiple points (P) within the eyebox (170).

[0123] Since the camera (300) has intrinsic parameters, if the position and direction of the camera (300) are known, it is possible to determine which pixel on the panel (210) a point (P) located in space appears as. Here, the intrinsic parameters refer to internal values ​​of the camera, such as focal length.

[0124] Here, the eye box (170) refers to a virtual space in which position coordinates are distributed in which a virtual driver's eye is thought to be located in three dimensions.

[0125] According to the embodiment, if the setter arbitrarily sets the eyebox (170) to 3*3*3, a total of 27 points (P) are formed, and each point (P) can be the line of sight (A) of the camera (300). That is, images viewed from each of the 27 points (P) can be captured. When viewing the distorted image (150) and the target image (160) from each point (P), the line of sight from each point (P) must be different, so a total of 27 images can be obtained.

[0126] According to the embodiment, the position coordinates of the point (P) of the eye position, i.e., the eye box (170), that forms the line of sight (A) can be expressed as Ex, Ey, Ez, which are coordinates in three-dimensional space in this specification.

[0127] Meanwhile, in this specification, the target image (160) may refer to a case where the marker image (100) reflected on the windshield (10) is a correct 3D image to the virtual driver's eye without distortion. In other words, the target image (160) may refer to a marker image (100) in which markers (110) having position coordinates that the user wants to see on the actual windshield (10) are distributed. By overlapping the target image (160) and the distorted image (150), a transformed position coordinate can be obtained in which each marker (110) on the distorted image (150) can be transformed into a marker (110) located at the correct position on the target image (160). The image captured by the camera (300) is intended to obtain the transformed position coordinates that can be transformed into the correct position. Specifically, the transformed position coordinates may refer to the actual position coordinates of the pixels that must be turned on on the panel (210) of the HUD to display the marker (110) located at the correct position on the target image (160).

[0128] That is, the transformed position coordinates are data that allows calculating which pixels to turn on on the panel (210) of the HUD in order to display the image desired by the user through the windshield (10). When a setting value is assigned to a pixel in this way on the panel (210), it is reflected on the windshield (10) and shown to the user, so that it appears as the coordinates of the target marker (110) on the target image (160). The transformed position coordinates can be expressed as Wx and Wy in this specification.

[0129] The parameters of the first and second regression functions for calculating warping and view mapping from the acquired images can be obtained. Here, the first function may refer to the warping function in the first embodiment or the second embodiment, and the second function may refer to the mapping function.

[0130] Here, the first and second regression functions may be regression equations that receive a total of five parameters, including eye positions Ex, Ey, Ez and pixel positions Px, Py, and calculate the view number and the degree of warping when viewing a pixel on the display from a certain eye position.

[0131] The aforementioned function may be configured to include a first function for calculating warping and a second function for calculating view mapping, depending on the embodiment. Here, the first function may be a type of regression equation, and may have an eye position corresponding to each of the multiple viewpoints (A) of the aforementioned optical simulation or the aforementioned camera (300) and a pixel position corresponding to the marker (110) of the marker image (100) as input parameters, and a compensated pixel position corresponding to the distortion-compensated image as an output parameter. Here, the compensated pixel position may refer to the aforementioned transformed position coordinates.

[0132] The first function can be expressed by the following mathematical formula 1.

[0133] [Mathematical Formula 1]

[0134] f(Ex, Ey, Ez, Px, Py) = (Wx, Wy)

[0135] When an image is obtained by capturing with a camera (300) or through optical simulation, output parameters Wx and Wy can be obtained. At this time, since the eye position and pixel position are already known values, the five parameters of eye position (Ex, Ey, Ez) and pixel position (Px, Py) form a data set with the two parameters of transformed position coordinates (Wx, Wy). Since this data set is generated for each marker (110) on the marker image (100), it can be obtained as many times as the number of markers (110) on the marker image (100). Therefore, in this embodiment, the marker image (100) is composed of 9*15 = 135 markers (110), so 135 data sets can be obtained.

[0136] Here, the eye positions (Ex, Ey, Ez) can include the left eye position and the right eye position, as the subpixels are each assigned to the left or right eye. Reflecting the results assigned to the subpixels, the left eye position (Ex L , Ey L , Ez L ) and right eye position (Ex R , Ey R , Ez R Includes ).

[0137] Mathematical Equation 1 based on left eye position is f(Ex L , Ey L , Ez L , Px, Py) = (Wx L , Wy L Since ), the left eye transformed position coordinates (Wx) by the left eye position L , Wy L ) is obtained, and mathematical formula 1 based on the right eye position is f(Ex R , Ey R , Ez R , Px, Py) = (Wx R , Wy R Since ), the right eye transformed position coordinates (Wx) by the right eye position R , Wy R) comes out.

[0138] Through the dataset obtained in this way, the first function, which is the regression equation, can be derived. For reference, since there are two output parameters, Wx and Wy, a total of two first functions can be generated: one for Wx and one for Wy. The compensated pixel position can be implemented for content corresponding to the user's left eye and content corresponding to the right eye, respectively.

[0139] Here, the second function, like the first function, takes as input parameters an eye position corresponding to each of the multiple viewpoints (A) of the optical simulation or camera (300) capture and a pixel position corresponding to the marker (110) of the marker image (100), and can take a view number visible to the user's eye as an output parameter. The difference between the process of creating the second function and the process of creating the first function is that the output parameter of the first function is a pixel position, whereas the output parameter of the second function is a view number. Here, the difference is that the pixel position consists of two X and Y coordinates in the planar coordinate system, whereas the view number consists of only one.

[0140] FIG. 13 shows a view image by a pixel when a specific number of pixels on a panel is turned on. FIG. 14 shows an example of a view number for each pixel of a display panel.

[0141] Each subpixel of the display panel can be assigned a view number. The view number assigned to each pixel does not necessarily have to be an integer. Also, the number of views can be arbitrarily determined by the setter. For example, the setter can assign views from 1 to 9. FIG. 14 shows the case where the number of views is 9.

[0142] Figure 14 shows two lenses among a lenticular lens array that covers a display panel diagonally. When lenticular lenses are stacked on a display panel, the number of views corresponding to each lens constituting the lenticular lens array is set to 9. One end and the other end of a single lens covering the display panel are shown, and the subpixels covered by the one end and the other end of the single lens are divided into 9 equal parts.

[0143] If one end of each lens in a lens array is set to '0' and the other end to '9', the view numbers of the subpixels covered by the lens area from one end to the other can be determined. In other words, if a reference point is established at the center of a subpixel, the view number can be determined by the relative position of this reference point with respect to the distance from one end to the other.

[0144] Here, the view number of each subpixel is displayed up to two decimal places, and if the view numbers differ by the number of views, they can be treated as the same number. Therefore, if one end of the lens is '0' and the other end is '9', the one end and the other end can be considered the same number.

[0145] For example, … -9=0=9=27… or … -4.7=4.3=13.3… that is, view numbers '-9', '0', '9', and '27' can be treated as the same number because they differ by 9, which is the number of views, and view numbers '-4.7', '4.3', and '13.3' can also be treated as the same number because they differ by 9, which is the number of views.

[0146] FIGS. 15 to 17 illustrate view images as examples.

[0147] According to an embodiment, an integer view number may be prepared to create a view image. According to an embodiment, the integer view number may be determined by, for example, 'rounding up' the aforementioned view number.

[0148] FIG. 15 shows, for example, view image 5, FIG. 16 shows view image 1, and FIG. 17 shows view image 9. In FIG. 15, when the view number is greater than 4.0 and less than or equal to 5.0, the integer view number is rounded up to 5, and FIG. 15 shows the appearance of view image 5, where only the subpixels are turned on, such that the view number is greater than 4.0 and less than or equal to 5.0, i.e., (4.0 < view number <= 5.0).

[0149] In FIG. 16, when the view number is greater than 0.0 and less than or equal to 1.0, the integer view number becomes 1 when rounded up, and FIG. 16 shows the appearance of the first view image where only the subpixels are turned on, such that the view number is greater than 0.0 and less than or equal to 1.0, i.e., (0.0 < view number <= 1.0).

[0150] In FIG. 17, when the view number is greater than 8.0 and less than or equal to 9.0, the integer view number is rounded up to 9, and FIG. 17 shows the appearance of the 9th view image, where only the subpixels are turned on, such that the view number is greater than 8.0 and less than or equal to 9.0 (8.0 < view number <= 9.0). In other words, a view image corresponding to the integer number can be prepared.

[0151] FIG. 18 shows the light path for each view image according to one embodiment of the present invention.

[0152] In the case where the number of views is 9, the light paths corresponding to the 5th view image, the 1st view image, and the 9th view image in the above-described embodiment are shown. Blue represents the 5th view, orange represents the 1st view, and green represents the 9th view.

[0153] FIG. 19 illustrates a view mapping method according to an embodiment of the present invention. Each subpixel may have three view numbers. That is, it may have its own view number, a left-eye view number, and a right-eye view number. Its own view number may be determined according to the stacking position of the lenses, and the left-eye view number and the right-eye view number may be determined according to the driver's eye position. In FIG. 19, the first row representing the subpixels indicates the view number of each subpixel, the second row indicates the left-eye view number, and the third row indicates the right-eye view number.

[0154] The view mapping method can compare a first value, which is the distance from the subpixel's own view number to the left eye view number, with a second value, which is the distance from the subpixel's own view number to the right eye view number, and assign it to the side closer to its own view number.

[0155] For example, the first value may represent the length from the subpixel's own view number to the left eye view number, and the second value may represent the length from the subpixel's own view number to the right eye view number. In this case, the first value and the second value may be compared and assigned to the one with the shorter length.

[0156] For example, in the case of the subpixel in the 4th column of Fig. 19, the subpixel's own view number is 8.13, the left eye view number is 2.28, and the right eye view number is 6.50. Since the second value, which is the distance from the subpixel's own view number to the left eye view number, is smaller than the first value, which is the distance from the subpixel's own view number to the left eye view number, it can be matched to the right eye. Fig. 19 shows that, looking at the assignment table, the subpixels in columns 1 through 3 are assigned to the left eye, and the subpixels in columns 4 through 8 are assigned to the right eye.

[0157] Each view number may have a view image seen from the left or right eye of the user or driver at an eye position on the eye box (170) at a specific angle, and the view image may be reflected by the windshield (10) and distorted before appearing to the user's eye. That is, the path of light is distorted, causing a distorted image to be displayed to the user's eye. Optical simulations can be performed on the distorted view image according to the eye position, or a view number having a correct view image can be obtained by capturing it with a camera (300). In the example above, a view number that enters the user's eye can be obtained, and this view number can be represented as V in this specification. In the embodiment above, there are a total of nine view images.

[0158] The second function can be expressed by the following mathematical formula 2.

[0159] [Mathematical Formula 2]

[0160] f(Ex, Ey, Ez, Px, Py) = (V)

[0161] Similar to the first function described above, an output parameter V can be obtained by capturing an image with a camera (300) or by optical simulation, and since the eye position and pixel position are already known values, the five parameters of the eye position (Ex, Ey, Ez) and pixel position (Px, Py) form a data set with one parameter of the view number (V).

[0162] Here, since the eye positions (Ex, Ey, Ez) are assigned to the left or right eye respectively, reflecting the results of the assignments to the subpixels gives the left eye position (Ex L , Ey L , Ez L ) and right eye position (Ex R , Ey R , Ez R It may include ),

[0163] Mathematical Equation 2 based on left eye position is f(ExL , Ey L , Ez L , Px, Py) = (V L Since ), the left eye view number (V) is determined by the left eye position L ) is obtained, and mathematical equation 2 based on the right eye position is f(Ex R , Ey R , Ez R , Px, Py) = (V R Since ), the right eye view number (V) is determined by the right eye position R ) comes out.

[0164] This data set is generated for each marker (110) on the marker image (100) just like the first function, so it can be secured in proportion to the number of markers (110) on the marker image (100). Therefore, in this embodiment, the marker image (100) is composed of 9 * 15 = 135 markers (110), so 135 data sets can be secured, and since 9 view numbers from view number 1 to view number 9 are applied to each data set, the second function can secure 9 times more data sets than the first function.

[0165] Through the dataset obtained in this way, the second function, which is the regression equation, can be derived. For reference, since there is only one output parameter V, a total of one second function can be generated for V.

[0166] And as described above, the view number visible to the user's eyes can be implemented for the content corresponding to the user's left eye and the content corresponding to the user's right eye, respectively, through view mapping.

[0167] Once a total of three functions, consisting of two first functions and one second function, have been created, the computing device can receive the user's eye position information in real time through the eye tracker and calculate the transformed position coordinates Wx, Wy and the view number V in real time. Here, if the view number is 5, the correct 3D image can be received by the user's eye when view number 5 is turned on for the pixel at the transformed position coordinates Wx, Wy.

[0168] When playing video on a 3D HUD using the obtained regression equations, warping and view mapping are performed so that the user, i.e., the driver, can see a proper 3D image. At this time, warping and view mapping receive the driver's eye position data from an eye tracker, and a parallel computing device such as a GPU calculates how much to warp and which view number to assign to each pixel of the display, and renders the left image to be visible to the driver's left eye and the right image to be visible to the driver's right eye.

[0169] According to the present invention, since both the shape of the virtual object and the distortion of the light path are corrected, the driver can see a proper 3D image without distortion.

[0170] The effect of the distortion compensation method according to the present invention is that, although the degree of distortion of shape or light path varies depending on the viewing angle, it is possible to ensure that an image is seen upright on a specific plane in space regardless of the viewing angle, and that the image is accurately directed to the left and right eyes. If the distorted shape differs between the two eyes, it becomes difficult to perceive it as a single object; furthermore, if the image is not accurately projected to each eye, the 3D image cannot be viewed properly and may even cause dizziness. The distortion compensation method according to the present invention can resolve these problems.

[0171] In addition, the distortion compensation method according to the present invention has the effect of having a small delay time because the execution time is short, since warping and view mapping can be performed in a parallel computing device and both can be performed simultaneously.

[0172] The embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, RAM, or flash memory.

[0173] Meanwhile, the above-mentioned computer program may be one specifically designed and configured for the present invention, or one known and available to those skilled in the art of computer software. Examples of computer programs may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0174] In the specification of the present invention (particularly in the claims), the use of the term "above" and similar descriptive terms may be in both singular and plural. Furthermore, where a range is described in the present invention, it is to include an invention to which individual values ​​belonging to said range are applied (unless otherwise stated), and this is equivalent to describing each individual value constituting said range in the detailed description of the invention.

[0175] Unless explicitly stated or contrary to the order of the steps constituting the method according to the present invention, said steps may be performed in a suitable order. The present invention is not necessarily limited by the order in which said steps are described. The use of all examples or exemplary terms (e.g., etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is not limited by said examples or exemplary terms unless limited by the claims. Furthermore, those skilled in the art will understand that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the claims or equivalents to which they are added.

[0176] Accordingly, the scope of the present invention should not be limited to the embodiments described above, and all scopes equivalent to or equivalently modified from the claims set forth below, as well as the claims set forth below, shall be considered to fall within the scope of the concept of the present invention. Explanation of the symbols

[0177] 10: Windshield 100: Marker image 110: Marker 150: Distorted image 160: Target image 170: iBox 200: HUD device 210: Display panel A: Point of view P: Points 300: Camera

Claims

Claim 1 A method for compensating for distortion of a HUD image caused by a windshield, comprising: a marker image preparation step of preparing a marker image including a plurality of markers displayed at a predetermined position on a display panel; an image acquisition step of obtaining an image through optical simulation or camera shooting after reflecting the marker image through a windshield; a warping function generation step of generating a warping function that calculates warping from the image; and a view image preparation step of preparing a view image; wherein the image acquisition step includes a mapping function generation step of obtaining an image through optical simulation or camera shooting after reflecting the marker image and the view image through the windshield, respectively, and generating a mapping function that calculates a view mapping from the image; wherein the warping function and the mapping function are functions that include eye position and pixel position as input parameters, and the transformation value is calculated in real time by the function according to the change in the user's eye position information. Claim 2 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in the marker image preparation step of claim 1, the marker image comprises an image in which a plurality of markers are formed. Claim 3 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in paragraph 2, each of the plurality of markers is arranged in a grid shape with a constant interval. Claim 4 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in claim 1, the optical simulation or camera shooting is performed at a plurality of viewpoints. Claim 5 A method for compensating for distortion of a HUD image caused by a windshield, characterized in that, in claim 4, the plurality of viewpoints include viewpoints viewed from a plurality of points located within a virtual eyebox. Claim 6 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in claim 5, each of the plurality of points is arranged in a grid form within the eye box at regular intervals. Claim 7 A method for compensating for distortion of a HUD image caused by a windshield according to claim 1, wherein the eye position, which is an input parameter of the warping function, corresponds to each of a plurality of viewpoints of the optical simulation or the camera shooting, and the pixel position corresponds to each of a plurality of viewpoints of the optical simulation or the camera shooting, and the warping function has a compensated pixel position corresponding to a distortion-compensated image as an output parameter. Claim 8 A method for compensating for distortion of a HUD image caused by a windshield, characterized in that, in claim 7, the compensated pixel position is implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye. Claim 9 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in claim 1, the warping includes compensating for shape distortion of a virtual object visible to the user's eye through the windshield. Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 A method for compensating for distortion of a HUD image caused by a windshield according to claim 1, wherein the eye position, which is an input parameter of the mapping function, corresponds to each of a plurality of viewpoints of the optical simulation or the camera capture, and the pixel position corresponds to each of a plurality of viewpoints of the optical simulation or the camera capture, and the mapping function has a view number visible to the user's eye as an output parameter. Claim 14 A method for compensating for distortion of a HUD image caused by a windshield, characterized in that, in claim 13, the view number visible to the user is implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye. Claim 15 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in claim 1, the view mapping includes compensating for the path distortion of light entering the user's eye through the windshield. Claim 16 A method for compensating for distortion of a HUD image caused by a windshield, wherein, in claim 1, the view image includes an image having subpixels that are turned on with a specific period. Claim 17 A method for compensating for distortion of a HUD image caused by a windshield according to claim 16, wherein subpixels belonging to the same period among the above subpixels each have light paths tilted in different directions by a parallax barrier or a lenticular lens. Claim 18 A method for compensating for distortion of a HUD image caused by a windshield, characterized in that, in claim 17, the period and the degree of inclination are determined by the parallax barrier or the lenticular lens. Claim 19 A HUD device applying the method of any one of paragraphs 1 through 9 and paragraphs 13 through 18. Claim 20 A computer program stored on a medium in combination with hardware to execute the method of any one of claims 1 through 9 and claims 13 through 18.

Images