Method for compensating for distortion of HUD images caused by windshield and device therefor

Abstract

An aspect of the present invention relates to a method and a device for compensating for distortion of head up display (HUD) images, wherein distortion caused by the windshield is compensated for such that the 3D HUD displays correct 3D images. The method may comprise at least one of: a step of preparing a marker image having a marker for detection in a predetermined position on a display panel; a step of preparing a view image that expresses subpixels to which a setting value is allocated so as to have a specific period; a step of securing an image through an optical simulation or an actual camera by reflecting the marker image and / or the view image on a windshield; and a step of securing a parameter of a regression equation for calculating warping and / or view mapping from the secured image. The present invention may provide a HUD device capable of producing correct 3D HUD images free from distortion caused by the windshield by performing a warping process of compensating for shape distortion of a virtual object entering the driver's field of view through the windshield in real time and / or a view mapping process of compensating for path distortion of light entering the driver's field of view through the windshield.

Description

Method and device for compensating for distortion of HUD image caused by windshield

[0001] One aspect of the present invention relates to a method and device for compensating for distortion of a head-up display image (HUD) to display a correct 3D image on a 3D HUD by compensating for distortion caused by a windshield.

[0002] The material described in this section merely provides background information on embodiments of the present invention and does not constitute prior art.

[0003] Head-Up Displays (HUDs) are commercially available in 2D and are installed in many vehicles. Using a large-screen 3D HUD, drivers can view instrument panels, road signs, and navigation more conveniently and safely.

[0004] 3D HUDs can be implemented in a variety of ways, but one approach is to reflect a 3D display onto a windshield. Lightfield rendering, which allows for different images to be viewed depending on the direction of travel, is a well-known method for rendering images on existing 3D displays.

[0005] However, when you look at a 3D display by reflecting it on a windshield, the shape of the object in the image played on the 3D display is distorted because the windshield is curved, and the path of the light is also bent from the original direction, so there is a problem in that you cannot see the correct 3D image.

[0006] Therefore, in order to view correct 3D images, it is urgent to develop a distortion compensation method that compensates for the shape distortion of objects in the image and also compensates for the path of light.

[0007] The background technology described above is technical information that the inventor possessed for the purpose of deriving embodiments of the present invention or acquired during the derivation process, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the filing of the embodiments of the present invention.

[0008] Accordingly, one aspect of the present invention is proposed to solve the above-mentioned problem, and the purpose of the present invention is to provide a distortion compensation method that compensates for the shape distortion of an object in an image and also compensates for the path of light so that a correct 3D image can be viewed without distortion caused by a windshield.

[0009] The distortion compensation method proposed in the present invention can be applied to windshields of various vehicle models, and is a method for displaying a correct 3D image without distortion in a HUD using a 3D display capable of light field rendering having a parallax barrier, a lenticular lens, or a similar structure.

[0010] In addition, another object of the present invention is to provide a HUD device that performs the distortion compensation method described above, and to provide a computer program stored in a medium to execute the distortion compensation method described above in combination with hardware.

[0011] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0012] In order to achieve the task raised above, one aspect of the present invention (first embodiment) comprises a marker image preparation step of preparing a marker image including a plurality of markers displayed at predetermined locations on a display panel;

[0013] An image acquisition step of reflecting the marker image through a windshield and then acquiring the image through optical simulation or camera shooting; and

[0014] A method for compensating for distortion of a HUD image caused by a windshield including at least one of a warping function generation step for generating a warping function for calculating warping from the above image;

[0015] According to an embodiment, in the marker image preparation step, the marker image may be characterized by including an image in which a plurality of markers are formed.

[0016] According to an embodiment, each of the plurality of markers may be characterized by being arranged in a grid shape with a constant interval.

[0017] According to an embodiment, the optical simulation or the camera shooting may be characterized by being performed from multiple view points.

[0018] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eye box.

[0019] According to an embodiment, each of the plurality of points may be arranged in the eye box in a grid shape with a constant interval.

[0020] According to an embodiment, the warping function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image as input parameters, and a compensated pixel position corresponding to an image in which distortion is compensated as an output parameter.

[0021] According to an embodiment, the compensated pixel positions may be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0022] In some embodiments, the warping may be characterized by compensating for shape distortion of a virtual object visible to the user through the windshield.

[0023] According to an embodiment, it may be characterized by including a view image preparation step for preparing a view image.

[0024] According to an embodiment, the image acquisition step may be characterized by reflecting the marker image and the view image through the windshield, respectively, and then acquiring the image through the optical simulation or the camera shooting.

[0025] According to an embodiment, the method may be characterized by including a mapping function generation step for generating a mapping function that calculates view mapping from the image.

[0026] According to an embodiment, the mapping function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image as input parameters, and a view number visible to the user's eyes as an output parameter.

[0027] According to an embodiment, the view number visible to the user's eyes may be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0028] In some embodiments, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0029] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific cycle.

[0030] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths that are tilted in different directions by a parallax barrier or a lenticular lens.

[0031] According to an embodiment, the period and the degree of inclination may be determined by the parallax barrier or the lenticular lens.

[0032]

[0033] Another aspect of the present invention (second embodiment) is a method for compensating for distortion of a HUD image (head up display image) caused by a windshield,

[0034] View image preparation stage that prepares a view image;

[0035] An image acquisition step of reflecting the above view image through a windshield and then acquiring the image through optical simulation or camera shooting; and

[0036] A method for compensating for distortion of a HUD image caused by a windshield including at least one of: a mapping function generation step for generating a mapping function for calculating view mapping from the above image;

[0037]

[0038] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific cycle.

[0039] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths that are tilted in different directions by a parallax barrier or a lenticular lens.

[0040] According to an embodiment, the period and the degree of inclination may be determined by the parallax barrier or the lenticular lens.

[0041] According to an embodiment, the optical simulation or the camera shooting may be characterized by being performed from multiple view points.

[0042] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eye box.

[0043] According to an embodiment, each of the plurality of points may be arranged in the eye box in a grid shape with a constant interval.

[0044] According to an embodiment, the mapping function may be characterized by having an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image as input parameters, and a view number visible to the user's eyes as an output parameter.

[0045] According to an embodiment, the view number visible to the user's eyes may be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0046] In some embodiments, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0047]

[0048] Another aspect of the present invention (the third embodiment) is a method for compensating for distortion of a HUD image (head up display image) caused by a windshield,

[0049] Marker image preparation step for preparing a marker image;

[0050] View image preparation stage that prepares a view image;

[0051] An image acquisition step of reflecting the marker image and the view image through a windshield and then acquiring the image through optical simulation or camera shooting;

[0052] A method for compensating for distortion of a HUD image caused by a windshield, comprising: a function generation step for generating a function for calculating warping and view mapping from the above image;

[0053] According to an embodiment, in the marker image preparation step, the marker image may be characterized by including an image in which a plurality of markers are formed.

[0054] According to an embodiment, each of the plurality of markers may be characterized by being arranged in a grid shape with a constant interval.

[0055] According to an embodiment, the view image may be characterized by including an image having subpixels that are turned on with a specific cycle on a black background.

[0056] According to an embodiment, subpixels belonging to the same period among the subpixels may be characterized by having light paths that are tilted in different directions by a parallax barrier or a lenticular lens.

[0057] According to an embodiment, the period and the degree of inclination may be determined by the parallax barrier or the lenticular lens.

[0058] According to an embodiment, the optical simulation or the camera shooting may be characterized by being performed from multiple view points.

[0059] According to an embodiment, the plurality of viewpoints may be characterized by including viewpoints viewed from a plurality of points located within a virtual eye box.

[0060] According to an embodiment, the function may be characterized by including a first function for calculating the warping and a second function for calculating the view mapping.

[0061] According to an embodiment, the first function may be characterized by having as input parameters an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image, and an output parameter a compensated pixel position corresponding to an image in which distortion is compensated.

[0062] According to an embodiment, the compensated pixel positions may be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0063] According to an embodiment, the second function may be characterized by having as input parameters an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image, and an output parameter a view number visible to the user's eyes.

[0064] According to an embodiment, the view number visible to the user's eyes may be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye.

[0065] In some embodiments, the warping may be characterized by compensating for shape distortion of a virtual object visible to the user through the windshield.

[0066] In some embodiments, the view mapping may be characterized by compensating for path distortion of light entering the user's eye through the windshield.

[0067] Another aspect of the present invention can provide a HUD device applying the above-described method.

[0068] Another aspect of the present invention may also provide a computer program stored on a medium to execute the above-described method in combination with hardware.

[0069] As described above, according to one embodiment of the present invention, a method for compensating for distortion of a HUD image caused by a windshield is provided, which compensates for shape distortion of an object in an image and also compensates for the path of light, so as to enable viewing of a correct 3D image without distortion caused by the windshield.

[0070] In addition, the method for compensating for distortion of a HUD image due to a windshield presented in the present invention can be applied to windshields of various vehicle models, and has the effect of displaying a correct 3D image without distortion in a HUD using a 3D display capable of lightfield rendering having a parallax barrier, a lenticular lens, or a similar structure.

[0071] The effect of the distortion compensation method according to the present invention is that, although the degree of distortion of the shape or light path varies depending on the viewing angle, it can make the image appear straight on a specific plane in space regardless of the viewing angle, and make the image enter the left and right eyes accurately. If the distorted shape is different for each eye, it becomes difficult to recognize and see it as a single object, and if the images are not accurately projected to each eye, the 3D image cannot be viewed properly and can even cause dizziness. The distortion compensation method according to the present invention can solve these problems.

[0072] In addition, the distortion compensation method according to the present invention has the effect of having a short execution time and a small delay time because warping and view mapping can be performed in a parallel computing device, and because both can be performed simultaneously.

[0073] In addition, the effects of the present invention have various effects, such as excellent versatility depending on the embodiment, and such effects can be clearly confirmed in the description of the embodiments described below.

[0074] The following drawings attached to this specification illustrate one embodiment of the present invention, and together with the detailed description of the invention described above, serve to further understand the technical idea of ​​the present invention, and therefore, the present invention should not be interpreted as being limited to matters described in such drawings.

[0075] FIG. 1 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a first embodiment of the present invention.

[0076] FIG. 2 illustrates a method for compensating for distortion of a HUD image caused by a windshield in FIG. 1, in which a view image preparation step and a mapping function generation step for generating a mapping function are additionally displayed.

[0077] FIG. 3 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a second embodiment of the present invention.

[0078] FIG. 4 shows a method for compensating for distortion of a HUD image caused by a windshield of FIG. 3, in which a marker image preparation step and a warping function generation step for generating a warping function are additionally displayed.

[0079] FIG. 5 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a third embodiment of the present invention.

[0080] FIG. 6 shows a display panel according to an embodiment, in which a marker image is captured by a camera as a result of reflection through a windshield.

[0081] Figure 7 shows a marker image according to one embodiment of the present invention.

[0082] Figure 8 shows a distorted image seen from a virtual driver's perspective by generating the marker image of Figure 7 on a panel and reflecting it on a windshield.

[0083] Figure 9 shows the distorted image of Figure 8 and the target image corrected through warping overlapping.

[0084] Figure 10 illustrates a view image according to one embodiment of the present invention.

[0085] Figure 11 shows a view image that is actually displayed on a display so that subpixels can be viewed separately.

[0086] Fig. 12 illustrates an eye box according to one embodiment of the present invention.

[0087] Figure 13 shows a view image by a pixel when a specific number of pixels on the panel is turned on.

[0088] Figure 14 shows an example of the view number of each pixel of the display panel.

[0089] Figure 15 shows an example of a view image, and shows view image number 5.

[0090] Figure 16 illustrates an example of a view image, and shows view image No. 1.

[0091] Figure 17 illustrates an example of a view image, and shows view image number 9.

[0092] Figure 18 shows the path of light for each view image according to one embodiment of the present invention.

[0093] Figure 19 illustrates a view mapping method according to one embodiment of the present invention.

[0094] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and it should be understood that it includes all transformations, equivalents, and substitutes included in the spirit and technical scope of the present invention. The embodiments presented below are provided to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. In describing the present invention, if a detailed description of a related known technology is judged to obscure the gist of the present invention, the detailed description thereof will be omitted.

[0095] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

[0096] In this application, terms such as "comprise" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood as not excluding in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by these terms. These terms are used solely for the purpose of distinguishing one component from another.

[0097] Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. In describing with reference to the attached drawings, identical or corresponding components are assigned the same drawing numbers and redundant descriptions thereof are omitted.

[0098] FIG. 1 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a first embodiment of the present invention. FIG. 2 illustrates the method for compensating for distortion of a HUD image caused by a windshield of FIG. 1, with a view image preparation step and a mapping function generation step for generating a mapping function additionally displayed.

[0099] Referring to FIGS. 1 and 2, one aspect of the present invention (first embodiment) comprises a marker image preparation step (S100) of preparing a marker image including a plurality of markers (110) displayed at predetermined locations on a display panel;

[0100] An image acquisition step (S110) of capturing an image by reflecting the marker image through a windshield and then capturing the image through optical simulation or camera shooting; and

[0101] A method for compensating for distortion of a HUD image caused by a windshield including at least one of a warping function generation step (S120) for generating a warping function for calculating warping from the above image can be provided.

[0102] According to an embodiment, a view image preparation step (S130) for preparing a view image may be additionally included. In this case, according to an embodiment, a mapping function generation step (S140) for generating a mapping function for calculating view mapping from the image may be additionally included.

[0103] FIG. 3 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a second embodiment of the present invention. FIG. 4 illustrates the method for compensating for distortion of a HUD image caused by a windshield of FIG. 3, with additional steps for preparing a marker image and generating a warping function.

[0104] Referring to FIGS. 3 and 4, the second embodiment of the present invention is a method for compensating for distortion of a HUD image (head up display image) caused by a windshield.

[0105] A view image preparation step (S200) for preparing a view image;

[0106] An image acquisition step (S210) of capturing an image by reflecting the above view image through a windshield and then capturing the image through optical simulation or camera shooting; and

[0107] A method for compensating for distortion of a HUD image caused by a windshield including at least one of: a mapping function generation step (S220) for generating a mapping function for calculating view mapping from the above image;

[0108] According to an embodiment, the method may include a marker image preparation step (S230) of preparing a marker image including a plurality of markers (110) displayed at predetermined locations on a display panel. In addition, according to an embodiment, the method may further include a warping function generation step (S240) of generating a warping function for calculating warping from the image.

[0109] FIG. 5 illustrates a method for compensating for distortion of a HUD image caused by a windshield according to a third embodiment of the present invention.

[0110] Fig. 6 shows a camera capturing the result of a marker image generated on a display panel reflected through a windshield according to an embodiment.

[0111] In the present invention, the display panel (210) may be a component installed in a HUD device (200). The HUD device (200) may include a device installed in a vehicle or the like that irradiates light containing an image to display the image to the driver. Here, the result is viewed from the camera's viewpoint (A). In another embodiment, the result may be implemented by calculation through optical simulation instead of the camera.

[0112] Figure 7 shows a marker image according to one embodiment of the present invention.

[0113] Referring to FIGS. 5, 6 and 7, a third embodiment of the present invention provides a method for compensating for distortion of a HUD image (head up display image) caused by a windshield (10).

[0114] Marker image preparation step (S300) for preparing a marker image (100, marker image);

[0115] A view image preparation step (S310) for preparing a view image;

[0116] An image acquisition step (S320) in which the marker image (100) and the view image are reflected through a windshield (10), respectively, and then images are acquired through optical simulation or camera (300) shooting;

[0117] A method for compensating for distortion of a HUD image caused by a windshield (10) including at least one of a function generation step (S330) for generating a function for calculating warping and view mapping from the above image can be provided.

[0118] The third embodiment includes both a marker image (100) and a view image, the first embodiment describes a method for compensating for distortion of a HUD image including a warping process related to the marker image (100), and the second embodiment describes a method for compensating for distortion of a HUD image including a mapping process related to the view image.

[0119] Hereinafter, the third embodiment will be primarily described in this specification. Since the first and second embodiments have substantially the same technical concepts as the third embodiment, a detailed description thereof will be omitted, replacing the description of the third embodiment.

[0120] Referring to FIG. 5, a method for compensating for distortion of a HUD image due to a windshield (10) according to one embodiment of the present invention is a method for compensating for distortion of a HUD image (head up display image) caused by a windshield (10).

[0121] A marker image preparation step (S300) for preparing a marker image (100); a view image preparation step (S310) for preparing a view image; an image acquisition step (S320) for reflecting the marker image (100) and the view image through a windshield (10), and then securing an image through optical simulation or camera (300) shooting; and

[0122] It can be configured to include a function generation step (S330) for generating a function that calculates warping and view mapping from the above image.

[0123] In this specification, the warping may include compensating for shape distortion of a virtual object visible to the user's eyes through the windshield (10). In addition, the view mapping may include compensating for path distortion of light entering the user's eyes through the windshield (10). Here, the virtual object may refer to a 2D or 3D image that the user can view through the windshield (10) as an image generated on the display panel (210) of the HUD device (200) according to an embodiment.

[0124] Referring to FIG. 7, in the marker image preparation step (S300), according to an embodiment, the marker image (100) may include an image in which a plurality of markers (110) are formed. The marker image (100) may include an image in which a plurality of markers (110) expressed as white points on a black background are arranged, according to an embodiment. The marker image (100) may be implemented electronically by assigning a setting value corresponding to the marker (110) to a pixel, according to an embodiment, or may be implemented mechanically or chemically by a film on which the marker (110) is displayed.

[0125] Here, the markers (110) implemented in the marker image (100) each have position coordinates. Therefore, the user can know the position information of the corresponding marker (110) in the marker image (100) by the aforementioned position coordinates. It is preferable that a plurality of markers (110) are prepared. If the plurality of markers (110) are arranged as densely as possible, the accuracy of distortion correction can be improved. According to an embodiment, each of the plurality of markers (110) displayed in the marker image (100) can be arranged in a grid shape with a regular interval. By arranging the plurality of markers (110) in a grid shape with a regular interval in this way, the accuracy of distortion correction due to the windshield (10) can be improved. Here, arranging in a grid shape may include arranging the plurality of markers (110) at a regular interval in the horizontal and vertical directions at right angles.

[0126] In general, when a HUD has a flat display panel (210), it is desirable to place markers (110) so that they can fill the entire area of ​​the panel (210). It is more desirable if the plurality of markers (110) cover most of the area of ​​the panel (210) and have a regular cycle.

[0127] The marker (110) can serve as a kind of reference point. When a marker image (100) displaying the marker (110) is implemented on the panel (210) of the HUD, the implemented marker image (100) can be reflected through the windshield (10) and enter the user's eyes. Here, the user may include, for example, the driver of the vehicle in which the HUD is used.

[0128] FIG. 8 shows a distorted image (150) seen from a virtual driver's perspective by generating the marker image of FIG. 7 on a panel (210) and reflecting it on a windshield, and FIG. 9 shows an appearance in which the distorted image of FIG. 8 and a target image (160) corrected through warping are overlapped.

[0129] In this specification, drawing number 150 on FIG. 9 indicates a distorted image, which is a marker image in which the blue markers are displayed, by enclosing them in parentheses on FIG. 9, and drawing number 160 on FIG. 9 also indicates a target image, which is a marker image in which the red markers are displayed, by enclosing them in parentheses on FIG. 9.

[0130] Here, the distorted image (150) may mean a marker image (100) displayed on the panel (210) according to an embodiment, in which the positions of the markers (110) appear distorted due to the marker image (100) being reflected on the windshield.

[0131] Here, the target image (160) may refer to an image that appears without distortion on the windshield (10) when the driver sitting in the driver's seat looks at the windshield. This target image (160) may refer to a warped image.

[0132] To help understanding, to explain further, in order for the HUD device (200) to implement the target image (160), the marker image (100) generated on the panel (210) must exist in a distorted form. That is, if there is no distortion in the marker image generated on the panel from the beginning, the image of the undistorted marker image reflected on the windshield and seen from the driver's point of view will be an image distorted by the windshield (10). Conversely, if the image seen from the driver's point of view is a correct image without distortion, in order to show the correct image resulting from reflection on the windshield (10), the marker image on the panel (210) of the HUD device (200) that generates and examines the image must be a distorted image from the beginning. Here, the correct image is the target image (160).

[0133] The marker image (100) reflected through the windshield (10) is distorted due to the curve of the windshield (10) and enters the user's eyes, which causes a problem in that the user cannot see the correct 2D or 3D image. Therefore, the distortion of the marker image (100) must be corrected, and in this specification, correcting the distortion of the marker image (100) is defined as warping. According to an embodiment, in this specification, warping may mean correcting the shape distortion of a 2D or 3D virtual object implemented through the windshield (10).

[0134] Fig. 10 illustrates a view image. Referring to Fig. 10, a view image according to one embodiment of the present invention represents an image in which subpixels are turned on at a specific cycle on a black background. The image of Fig. 10 is an image generated when the subpixel structure of the display is in RGB order.

[0135] Figure 11 shows a view image that is actually displayed on a display so that subpixels can be viewed separately.

[0136] In some embodiments, the view image may include an image in which subpixels are turned on with a specific cycle on a black background. In some embodiments, subpixels belonging to the same cycle among the subpixels may have light paths that are tilted in different directions by a parallax barrier or a lenticular lens.

[0137] Depending on the embodiment, the specific period and degree of inclination of the light path described above may be determined by a parallax barrier or lenticular lens formed on the panel (210).

[0138] Here, the number of views in the view image can be arbitrarily predetermined by a setter. The setter mentioned here may refer to a designer who designs a method for compensating for distortion of a HUD image according to the present invention.

[0139] Fig. 12 illustrates an eye box according to one embodiment of the present invention.

[0140] According to an embodiment, optical simulation or camera (300) shooting may be performed from multiple viewpoints (A, view piont). In addition, the multiple viewpoints (A) may include viewpoints (A) viewed from each of multiple points (P) located within a virtual eye box (170). For example, although each of the multiple points (P) is expressed as 3*3*3 in the embodiment of the present specification, this is exemplary, and in reality, when the conversion function (first function, second function) is determined, values ​​(image information, conversion function information, etc.) between points (P) can also be calculated by a method such as interpolation.

[0141] For example, referring to FIGS. 8 and 9, when a marker image (100) is displayed on a panel (210) and reflected on a windshield (10), a distorted marker image, that is, a distorted image (150), is generated due to the curved surface of the windshield (10). This distorted image (150) can be captured by using a camera (300) in an eye box (170) to obtain an image, or an image can be obtained by performing an optical simulation. Here, the position coordinates of the plurality of markers (110) on the distorted image (150) may be the same as the position coordinates of the pixels corresponding to the marker images (100) appearing on the panel (210) of the HUD. The position coordinates of these pixels may correspond to the position coordinates of the markers (110) in the marker image (100). In this specification, the position coordinates of the pixel positions corresponding to the position coordinates of the marker images (100) are denoted as Px and Py.

[0142] In this case, an image reflected on the windshield (10) at multiple points (P) within the eyebox (170) can be captured using a camera (300) or an optical simulation can be performed to generate an image reflected on the windshield (10) at multiple points (P) within the eyebox (170).

[0143] Since the camera (300) has intrinsic parameters, if the position and direction of the camera (300) are known, it is possible to know at which location on the panel (210) a point (P) located in space appears as a pixel. Here, the intrinsic parameters refer to internal values ​​of the camera, such as the focal length.

[0144] Here, the eye box (170) refers to a virtual space in which the position coordinates where the virtual driver's eyes are thought to be located in three dimensions are distributed.

[0145] According to an embodiment, if the setter arbitrarily sets the eye box (170) to 3*3*3, a total of 27 points (P) are formed, and each point (P) can be the line of sight (A) of the camera (300). That is, an image viewed from each of the 27 points (P) can be captured. When viewing the distorted image (150) and the target image (160) from each point (P), the line of sight viewed from each point (P) cannot but be different, so a total of 27 images can be acquired.

[0146] According to an embodiment, the position coordinates of the point (P) of the eye position (170) forming the line of sight (A) can be expressed as Ex, Ey, Ez, which are coordinates in three-dimensional space in this specification.

[0147] Meanwhile, in the present specification, the target image (160) may mean a case where the marker image (100) reflected on the windshield (10) is a correct 3D image to the eyes of a virtual driver who is not distorted. In other words, the target image (160) may mean a marker image (100) in which markers (110) having position coordinates that the user wants to see on the actual windshield (10) are distributed. By overlapping the target image (160) and the distorted image (150), the transformed position coordinates that can transform each marker (110) on the distorted image (150) into a marker (110) located at the correct position on the target image (160) can be obtained. The image captured by the camera (300) is for obtaining the transformed position coordinates that can be transformed into the correct position. Specifically, the transformation position coordinates may refer to the actual position coordinates of the pixels that must be turned on in the panel (210) of the HUD to indicate a marker (110) located at the correct position on the target image (160).

[0148] That is, the transformation position coordinates are data that enable calculation of which pixels should be turned on on the panel (210) of the HUD to display the image desired by the user through the windshield (10), and when a setting value is assigned to a pixel in this way on the panel (210), when it is reflected on the windshield (10) and shown to the user, it appears as the coordinates of the target marker (110) on the target image (160). The transformation position coordinates can be expressed as Wx and Wy in this specification.

[0149] From the acquired images, parameters of the first and second regression functions for calculating warping and view mapping can be obtained. Here, the first function may refer to the warping function in the first or second embodiment, and the second function may refer to the mapping function.

[0150] Here, the first and second regression functions may be regression equations that receive five parameters, namely eye positions Ex, Ey, Ez and pixel positions Px, Py, and calculate the view number and the degree of warping when viewing a pixel of the display from a certain eye position.

[0151] The above-described function may be configured to include a first function for calculating warping and a second function for calculating view mapping, depending on the embodiment. Here, the first function may be a type of regression equation, and may have as input parameters an eye position corresponding to each of a plurality of viewpoints (A) of the above-described optical simulation or the above-described camera (300) shooting and a pixel position corresponding to a marker (110) of a marker image (100), and may have as output parameters a compensated pixel position corresponding to an image in which distortion is compensated. Here, the compensated pixel position may mean the above-described transformed position coordinates.

[0152] The first function can be expressed as the following mathematical expression 1.

[0153] [Mathematical Formula 1]

[0154] f(Ex, Ey, Ez, Px, Py) = (Wx, Wy)

[0155] When an image is acquired by camera (300) shooting or optical simulation, output parameters Wx and Wy can be obtained. At this time, since the eye position and pixel position are already known values, five parameters of the eye position (Ex, Ey, Ez) and the pixel position (Px, Py) form a data set with two parameters of the transformed position coordinates (Wx, Wy). Since this data set is generated for each marker (110) on the marker image (100), the number of data sets can be secured equal to the number of markers (110) on the marker image (100). Therefore, in the present embodiment, the marker image (100) is composed of 9*15 = 135 markers (110), so 135 data sets can be secured.

[0156] Here, the eye position (Ex, Ey, Ez) can include the left eye position and the right eye position, since the subpixels are each assigned to the left or right eye. If the result of assignment to the subpixels is reflected, the left eye position (Ex L , Ey L , Ez L ) and right eye position (Ex R , Ey R , Ez R ) is included.

[0157] Mathematical expression 1 by left eye position is f(Ex L , Ey L , Ez L , Px, Py) = (Wx L , Wy L ) so the left eye transformation position coordinate (Wx) is determined by the left eye position. L , Wy L ) is shown, and the mathematical expression 1 by the right eye position is f(Ex R , Ey R , Ez R , Px, Py) = (Wx R , Wy R ) so the right eye transformation position coordinate (Wx) is determined by the right eye position. R , Wy R) appears.

[0158] Using the data set secured in this way, the first function, which is a regression equation, can be derived. Note that since there are two output parameters, Wx and Wy, two first functions can be created: one for Wx and one for Wy. The compensated pixel positions can be implemented for each content corresponding to the user's left eye and the other corresponding to the user's right eye.

[0159] Here, the second function, like the first function, may have as input parameters an eye position corresponding to each of multiple viewpoints (A) of optical simulation or camera (300) shooting and a pixel position corresponding to a marker (110) of a marker image (100), and may have as output parameters a view number visible to the user's eyes. The difference between the process of creating the second function and the process of creating the first function is that the output parameter of the first function is the pixel position, whereas the output parameter of the second function is the view number. Here, the pixel position is different in that there are two coordinates, X and Y, on the plane, whereas the view number is only one.

[0160] Figure 13 shows a view image by a specific number of pixels on the panel when those pixels are turned on. Figure 14 shows an example of the view number of each pixel on the display panel.

[0161] Each subpixel on the display panel can be assigned a view number. The view number assigned to each pixel does not necessarily have to be an integer. Furthermore, the number of views can be arbitrarily determined by the user. For example, the user can assign views from 1 to 9. Figure 14 illustrates a case where the number of views is 9.

[0162] Figure 14 shows two lenses among a lenticular lens array that diagonally covers a display panel. When lenticular lenses are stacked on the display panel, the number of views corresponding to each lens constituting the lenticular lens array is set to 9. One end and the other end of one lens covering the display panel are shown, and the subpixels covered by one end and the other end of one lens are divided into 9 equal parts.

[0163] If one end of each lens in the lens array is set to '0' and the other end is set to '9', the view numbers of the subpixels covered by the lens area from one end to the other can be determined. In other words, if a reference point is set at the center of a subpixel, the relative position of this reference point with respect to the distance from one end to the other of the lens can be determined as the view number.

[0164] Here, the view number for each subpixel is displayed up to two decimal places. If the view number differs by the number of views, they can be treated as the same number. Therefore, if one end of the lens is '0' and the other end is '9', the two ends can be considered the same number.

[0165] For example, ... -9=0=9=27... or ... -4.7=4.3=13.3... That is, view numbers '-9', '0', '9', and '27' can be treated as the same number because they differ by the number of views, 9, and view numbers '-4.7', '4.3', and '13.3' can also be treated as the same number because they differ by the number of views, 9.

[0166] Figures 15 to 17 illustrate view images as examples.

[0167] In some embodiments, an integer view number may be prepared to create a view image. In some embodiments, the integer view number may be determined by, for example, 'rounding up' the aforementioned view number.

[0168] Fig. 15 shows, for example, a view image number 5, Fig. 16 shows a view image number 1, and Fig. 17 shows a view image number 9. In Fig. 15, when the view number is greater than 4.0 and less than or equal to 5.0, the integer view number becomes 5 when rounded up, and Fig. 15 shows a view image number 5 in which only subpixels with a view number greater than 4.0 and less than or equal to 5.0, that is, (4.0<view number<=5.0), are turned on.

[0169] In Fig. 16, if the view number is greater than 0.0 and less than or equal to 1.0, the integer view number becomes 1 when rounded up, and Fig. 16 shows the appearance of the view image 1 in which only the subpixels with the view number greater than 0.0 and less than or equal to 1.0, that is, (0.0<view number<=1.0), are turned on.

[0170] In Fig. 17, if the view number is greater than 8.0 and less than or equal to 9.0, the integer view number is rounded up to 9, and Fig. 17 shows the 9th view image in which only the subpixels with the view number greater than 8.0 and less than or equal to 9.0, that is, (8.0<view number<=9.0) are turned on. In other words, a view image corresponding to the integer number can be prepared.

[0171] Figure 18 shows the path of light for each view image according to one embodiment of the present invention.

[0172] In the case where the number of views is 9, the light paths corresponding to the 5th view image, the 1st view image, and the 9th view image in the above-described embodiment are shown. Blue represents view 5, red represents view 1, and green represents view 9.

[0173] Fig. 19 illustrates a view mapping method according to one embodiment of the present invention. Each subpixel can have three view numbers. That is, the subpixel can have its own view number, a left-eye view number, and a right-eye view number. The view number can be determined according to the stacking position of the lens, and the left-eye view number and the right-eye view number can be determined according to the driver's eye position. In Fig. 19, the first row representing subpixels represents the view number of each subpixel, the second row represents the left-eye view number, and the third row represents the right-eye view number.

[0174] The view mapping method can compare the first value, which is the distance from the subpixel's own view number to the left-eye view number, and the second value, which is the distance from the subpixel's own view number to the right-eye view number, and assign the one closer to the subpixel's own view number.

[0175] For example, the first value may mean the length from the subpixel's own view number to the left-eye view number, and the second value may mean the length from the subpixel's own view number to the right-eye view number. In this case, the first value and the second value may be compared and assigned to the one with the shorter length.

[0176] Taking the subpixel in the 4th column of Fig. 19 as an example, the subpixel's own view number is 8.13, the left-eye view number is 2.28, and the right-eye view number is 6.50. Therefore, the second value, which is the distance from the subpixel's own view number to the left-eye view number, is smaller than the first value, which is the distance from the subpixel's own view number to the left-eye view number, so it can be matched to the right eye. Looking at the allocation table in Fig. 19, the subpixels from the 1st to the 3rd columns are allocated to the left eye, and the subpixels from the 4th to the 8th columns are allocated to the right eye.

[0177] Each view number may have a view image seen from the left or right eye of the user or driver at the eye position on the eye box (170) at a specific angle, and the view image here may be reflected on the windshield (10) and distorted and appear to the user's eyes. That is, the path of light is distorted, resulting in a distorted image being displayed to the user's eyes. The distorted view image may be subjected to optical simulation according to the eye position or captured with a camera (300) to obtain a view number having a correct view image. In the example described above, the view number when it enters the user's eye can be obtained, and this view number can be expressed as V in this specification. In the embodiment described above, there are a total of 9 view images.

[0178] The second function can be expressed by the following mathematical expression 2.

[0179] [Equation 2]

[0180] f(Ex, Ey, Ez, Px, Py) = (V)

[0181] As with the first function described above, when an image is acquired by taking a picture with a camera (300) or through optical simulation, the output parameter V can be obtained. At this time, since the eye position and pixel position are already known values, the five parameters of the eye position (Ex, Ey, Ez) and the pixel position (Px, Py) form a data set with one parameter of the view number (V).

[0182] Here, the eye position (Ex, Ey, Ez) reflects the result of assignment to the subpixels, as the subpixels are assigned to the left or right eye, respectively. L , Ey L , Ez L ) and right eye position (Ex R , Ey R , Ez R ) may be included,

[0183] Mathematical expression 2 by left eye position is f(ExL , Ey L , Ez L , Px, Py) = (V L ) so the left eye view number (V) is determined by the left eye position. L ) is shown, and the mathematical expression 2 by the right eye position is f(Ex R , Ey R , Ez R , Px, Py) = (V R ) so the right eye view number (V) is determined by the right eye position. R ) appears.

[0184] Since this data set is generated for each marker (110) on the marker image (100), as in the first function, the number of markers (110) on the marker image (100) can be secured. Accordingly, in the present embodiment, the marker image (100) is composed of 9*15 = 135 markers (110), so 135 data sets can be secured, and since 9 view numbers from the 1st view number to the 9th view number are applied to each data set, the second function can secure 9 times more data sets than the first function.

[0185] Using the data set secured in this way, we can derive the second function, which is a regression equation. Note that since there is one output parameter of V, a total of one second function can be created for V.

[0186] And as described above, the view number visible to the user's eyes can be implemented for each of the content corresponding to the user's left eye and the content corresponding to the user's right eye through view mapping.

[0187] If a total of three functions, two first functions and one second function, are created, the calculation device can receive the user's eye position information in real time through the eye tracker through these three functions and calculate the transformation position coordinates Wx, Wy and the view number V in real time. Here, if the view number is 5, the correct 3D image can enter the user's eyes when the view number 5 is turned on for the pixels of the transformation position coordinates Wx, Wy.

[0188] When playing back images on a 3D HUD using the derived regression equations, warping and view mapping are performed so that the user, i.e. the driver, can see a proper 3D image. At this time, warping and view mapping receive the driver's eye position data from the eye tracker, and a parallel processing device such as a GPU calculates how much to warp and which view number to assign to each pixel of the display, so that the left image can be rendered to the driver's left eye, and the right image can be rendered to the driver's right eye.

[0189] According to the present invention, since both the shape of the virtual object and the distortion of the light path are corrected, the driver can view a proper 3D image without distortion.

[0190] The effect of the distortion compensation method according to the present invention is that, although the degree of distortion of the shape or light path varies depending on the viewing angle, it can make the image appear straight on a specific plane in space regardless of the viewing angle, and make the image enter the left and right eyes accurately. If the distorted shape is different for each eye, it becomes difficult to recognize and see it as a single object, and if the images are not accurately projected to each eye, the 3D image cannot be viewed properly and can even cause dizziness. The distortion compensation method according to the present invention can solve these problems.

[0191] In addition, the distortion compensation method according to the present invention has the effect of having a short execution time and a small delay time because warping and view mapping can be performed in a parallel computing device, and because both can be performed simultaneously.

[0192] The embodiments of the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the medium may include a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a hardware device specifically configured to store and execute program instructions, such as a ROM, a RAM, a flash memory, etc.

[0193] Meanwhile, the computer program may be specifically designed and constructed for the present invention, or may be one known and available to those skilled in the computer software field. Examples of computer programs may include not only machine language code, such as that generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

[0194] The use of the term "above" and similar referential terms in the specification of the present invention (especially in the claims) may refer to both singular and plural. Furthermore, when a range is described in the present invention, it is intended that the invention encompasses inventions that apply individual values ​​falling within the range (unless otherwise stated), and is equivalent to describing each individual value constituting the range in the detailed description of the invention.

[0195] Unless the steps constituting the method according to the present invention are explicitly described in a specific order or are not described to the contrary, the steps may be performed in any appropriate order. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (e.g., "for example," etc.) in the present invention is merely intended to illustrate the present invention in detail, and the scope of the present invention is not limited by the examples or exemplary terms unless otherwise defined by the claims. Furthermore, those skilled in the art will appreciate that various modifications, combinations, and variations can be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

[0196] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and not only the scope of the patent claims described below but also all scopes equivalent to or equivalently modified from the scope of the patent claims are considered to fall within the scope of the idea of ​​the present invention.

[0197]

[0198] ** Explanation of symbols **

[0199] 10: Windshield 100: Marker Image 110: Marker

[0200] 150: Distorted image 160: Target image 170: Eyebox

[0201] 200: HUD device 210: Display panel A: Viewpoint

[0202] P: Point 300: Camera

Claims

1. A marker image preparation step of preparing a marker image including a plurality of markers displayed at a set location on a display panel; An image acquisition step of reflecting the marker image through a windshield and then acquiring the image through optical simulation or camera shooting; and A warping function generation step for generating a warping function that calculates warping from the above image; A method for compensating for distortion of a HUD image caused by a windshield including a .

2. In paragraph 1, A method for compensating for distortion of a HUD image caused by a windshield, characterized in that in the marker image preparation step, the marker image includes an image in which a plurality of markers are formed.

3. In paragraph 2, A method for compensating for distortion of a HUD image caused by a windshield, wherein each of the plurality of markers is arranged in a grid shape with a constant interval.

4. In paragraph 1, A method for compensating for distortion of a HUD image caused by a windshield, wherein the optical simulation or the camera shooting is performed from multiple view points.

5. In paragraph 4, A method for compensating for distortion of a HUD image caused by a windshield, wherein the plurality of viewpoints include viewpoints viewed from a plurality of points located within a virtual eye box.

6. In paragraph 5, A method for compensating for distortion of a HUD image caused by a windshield, wherein each of the plurality of points is arranged in a grid shape within the eye box at a constant interval.

7. In paragraph 1, A method for compensating for distortion of a HUD image caused by a windshield, wherein the warping function has as input parameters an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image, and as an output parameter a compensated pixel position corresponding to an image in which distortion has been compensated.

8. In paragraph 7, A method for compensating for distortion of a HUD image caused by a windshield, wherein the above compensated pixel positions are implemented for each of content corresponding to the user's left eye and content corresponding to the user's right eye.

9. In paragraph 1, A method for compensating for distortion of a HUD image caused by a windshield, wherein the warping comprises compensating for distortion of the shape of a virtual object visible to the user's eyes through the windshield.

10. In paragraph 1, A method for compensating for distortion of a HUD image caused by a windshield, characterized in that it includes a view image preparation step of preparing a view image.

11. In paragraph 10, A method for compensating for distortion of a HUD image caused by a windshield, characterized in that the image acquisition step comprises reflecting the marker image and the view image through the windshield, respectively, and then acquiring the image through the optical simulation or the camera shooting.

12. In paragraph 11, A method for compensating for distortion of a HUD image caused by a windshield, characterized in that it includes a mapping function generation step for generating a mapping function for calculating view mapping from the above image.

13. In paragraph 12, A method for compensating for distortion of a HUD image caused by a windshield, wherein the mapping function has as input parameters an eye position corresponding to each of a plurality of viewpoints of the optical simulation or the camera shooting and a pixel position corresponding to a marker of the marker image, and as an output parameter a view number visible to the user's eyes.

14. In paragraph 13, A method for compensating for distortion of a HUD image caused by a windshield, wherein the view numbers visible to the user's eyes are implemented for each of the contents corresponding to the user's left eye and the contents corresponding to the user's right eye.

15. In paragraph 12, A method for compensating for distortion of a HUD image caused by a windshield, wherein the view mapping comprises compensating for distortion of the path of light entering the user's eye through the windshield.

16. In paragraph 10, A method for compensating for distortion of a HUD image caused by a windshield, wherein the view image includes an image having subpixels that are turned on with a specific cycle.

17. In paragraph 16, A method for compensating for distortion of a HUD image caused by a windshield, wherein subpixels belonging to the same period among the above subpixels have light paths that are inclined in different directions due to a parallax barrier or a lenticular lens.

18. In paragraph 17, A method for compensating for distortion of a HUD image caused by a windshield, wherein the period and the degree of inclination are determined by the parallax barrier or the lenticular lens.

19. A HUD device applying the method of any one of clauses 1 to 18.

20. A computer program stored on a medium for executing the method of any one of claims 1 to 18 in combination with hardware.

Images