Method and apparatus for controlling head-up display and head-up display device

Abstract

A method and apparatus for controlling a head-up display and a head-up display device are provided. The method includes confirming an eye tracking state based on a result of eye tracking, and confirming a rendering mode of a HUD image as one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking state.

Description

[0001] This application claims priority to Korean Patent Application No. 10-2020-0106554, filed on August 24, 2020, and Korean Patent Application No. 10-2020-0130402, filed on October 8, 2020, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference. TECHNICAL FIELD

[0002] Methods and apparatuses consistent with example embodiments relate to a method and apparatus for controlling a head-up display (HUD) based on an eye tracking state. BACKGROUND

[0003] A head-up display (HUD) system generates a virtual image in front of a driver of a vehicle and provides various information to the driver by displaying information in the virtual image. The information provided to the driver can include, for example, navigation information and dashboard information such as vehicle speed, fuel level, and revolutions per minute (RPM) of an engine. The driver can more easily recognize the information displayed in front during driving without turning his or her gaze, and thus, driving safety can be improved. In addition to the navigation information and the dashboard information, the HUD system can provide, for example, a lane indicator, a construction indicator, an accident indicator, or a pedestrian detection indicator to the driver using augmented reality (AR) to assist driving when a field of view is poor and / or insufficient. SUMMARY

[0004] One or more example embodiments can address at least the above problems and / or disadvantages and other disadvantages not described above. Also, the example embodiments need not overcome the above disadvantages, and the example embodiments can not overcome any of the problems described above.

[0005] According to an aspect of the disclosure, there is provided a method of controlling a head-up display (HUD), the method including: performing eye tracking of eyes of a user in a photographed image; confirming an eye tracking state based on a result of the eye tracking; confirming a rendering mode of a HUD image as one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking state; and rendering the HUD image in the confirmed rendering mode.

[0006] The step of confirming the eye tracking state can include classifying the eye tracking state as one of a stable state and an unstable state based on whether eye coordinates exist in the result of the eye tracking or based on a change rate of the eye coordinates.

[0007] The step of confirming the rendering mode can include confirming the rendering mode as the 3D rendering mode based on the eye tracking state being classified as the stable state, and confirming the rendering mode as the 2D rendering mode based on the eye tracking state being classified as the unstable state.

[0008] The eye tracking state can be classified as a stable state based on the fact that the eye coordinates are included in the result of the eye tracking and a change speed of the eye coordinates is less than a reference value.

[0009] The reference value can correspond to a system processing rate.

[0010] The eye tracking state can be classified as an unstable state based on the fact that the eye coordinates are included in the result of the eye tracking and a change speed of the eye coordinates is greater than a reference value, or based on the fact that the eye coordinates are not included in the result of the eye tracking.

[0011] The HUD image can be rendered based on a first source image of a first viewpoint and a second source image of a second viewpoint.

[0012] The step of rendering the HUD image can include rendering the HUD image by setting the first viewpoint and the second viewpoint to a single viewpoint identically based on the fact that the confirmed rendering mode is 2D rendering.

[0013] The step of rendering the HUD image can include setting the first viewpoint and the second viewpoint to a center viewpoint of current eye coordinates identically based on the fact that the result of the eye tracking includes the current eye coordinates of both eyes and a change speed of the current eye coordinates is greater than a reference value, and setting the first viewpoint and the second viewpoint to a center viewpoint of previous eye coordinates identically based on the fact that the result of the eye tracking does not include the current eye coordinates.

[0014] The step of rendering the HUD image can include rendering the HUD image by setting the first viewpoint and the second viewpoint to different viewpoints based on the fact that the confirmed rendering mode is a 3D rendering mode.

[0015] Based on the fact that the rendering mode is confirmed, the rendering mode can be switched from a 3D rendering mode to a 2D rendering mode or from a 2D rendering mode to a 3D rendering mode during a buffering time corresponding to a plurality of frames.

[0016] The HUD image can be rendered based on a first source image of a first viewpoint and a second source image of a second viewpoint, wherein the step of rendering the HUD image includes rendering the HUD image while gradually changing the first viewpoint and the second viewpoint to a single viewpoint used in a 2D rendering mode within a buffering time based on the fact that the rendering mode is switched from a 2D rendering mode to a 3D rendering mode.

[0017] According to another aspect of the disclosure, a non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method is provided.

[0018] According to another aspect of the disclosure, there is provided an apparatus for controlling a head-up display (HUD), the apparatus including: a memory configured to store one or more instructions, and a processor configured to execute the one or more instructions to: perform eye tracking of a user's eyes in a photographed image, confirm an eye tracking state based on a result of the eye tracking, confirm a rendering mode of a HUD image as one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking state, and render the HUD image in the confirmed rendering mode.

[0019] The processor can be further configured to classify the eye tracking state as one of a stable state and an unstable state based on whether the eye coordinates exist in the result of the eye tracking or based on a change rate of the eye coordinates.

[0020] The processor can be further configured to confirm the rendering mode as the 3D rendering mode based on the eye tracking state being classified as the stable state, and confirm the rendering mode as the 2D rendering mode based on the eye tracking state being classified as the unstable state.

[0021] Based on the rendering mode being confirmed, the rendering mode can be switched from the 3D rendering mode to the 2D rendering mode or from the 2D rendering mode to the 3D rendering mode during a buffering time corresponding to a plurality of frames.

[0022] According to another aspect of the disclosure, there is provided an apparatus for controlling a head-up display (HUD), the apparatus including: a memory configured to store one or more instructions, and a processor configured to execute the one or more instructions to: perform eye tracking of a user's eyes in a photographed image, confirm an eye tracking state based on a result of the eye tracking, confirm a rendering mode of a HUD image as one of a two-dimensional (2D) rendering mode and a three-dimensional (3D) rendering mode based on the eye tracking state, and render the HUD image in the confirmed rendering mode.

[0023] The processor can be further configured to classify the eye tracking state as one of a stable state and an unstable state based on whether the eye coordinates exist in the result of the eye tracking or based on a change rate of the eye coordinates.

[0024] The processor can be further configured to switch the rendering mode from the 3D rendering mode to the 2D rendering mode or from the 2D rendering mode to the 3D rendering mode during a buffering time corresponding to a plurality of frames based on the rendering mode being confirmed.

[0025] According to another aspect of the disclosure, a display device is provided, including: a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions to: receive eye tracking information about eyes of a user in a photographed image; confirm whether eye tracking is stable or unstable based on the eye tracking information; output a virtual object in a two-dimensional (2D) image based on the eye tracking being unstable; and output the virtual object in a three-dimensional (3D) image based on the eye tracking being stable.

[0026] According to another aspect of the disclosure, a display method is provided, including: receiving eye tracking information about eyes of a user in an image; confirming whether eye tracking is stable or unstable based on the eye tracking information; outputting a virtual object in a two-dimensional (2D) image based on the eye tracking being unstable; and outputting the virtual object in a three-dimensional (3D) image based on the eye tracking being stable. BRIEF DESCRIPTION OF DRAWINGS

[0027] The above and / or other aspects will be more apparent by describing certain example embodiments, by reference to the accompanying drawings, in which:

[0028] FIG. 1A A head-up display (HUD) device according to an example embodiment is illustrated;

[0029] FIG. 1B A path of light related to the HUD device according to an example embodiment is illustrated;

[0030] FIG. 2 A structure of a display device according to an example embodiment is illustrated;

[0031] FIG. 3 A three-dimensional (3D) augmented reality (AR) according to an example embodiment is illustrated;

[0032] FIG. 4 A process of generating a HUD image according to an example embodiment is illustrated;

[0033] FIG. 5 An eye tracking state according to an example embodiment is illustrated;

[0034] FIG. 6 Eye movement in a viewing space according to an example embodiment is illustrated;

[0035] FIG. 7 A process of switching a rendering mode according to an example embodiment is illustrated;

[0036] FIG. 8 A buffer viewpoint and a buffer source image for switching a rendering mode according to an example embodiment are illustrated;

[0037] FIG. 9An image in a 3D rendering mode according to an example embodiment is shown;

[0038] FIG. 10 An image in a two-dimensional (2D) rendering mode according to an example embodiment is shown;

[0039] FIG. 11 And FIG. 12 A process of tracking an eye using a tracking area according to an example embodiment is shown;

[0040] FIG. 13 A process of generating a HUD image based on eye tracking according to an example embodiment is shown;

[0041] FIG. 14 A method of controlling a HUD in consideration of an eye tracking state according to an example embodiment is shown;

[0042] FIG. 15 A configuration of a HUD control device according to an example embodiment is shown; and

[0043] FIG. 16 A configuration of an electronic device according to an example embodiment is shown. DETAILED DESCRIPTION

[0044] Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. However, various changes and modifications can be made to the example embodiments. Here, the example embodiments are not to be interpreted as being limited to the disclosure. The example embodiments are to be understood as including all changes, equivalents, and substitutes within the idea and technological scope of the disclosure.

[0045] The terms used herein are merely used to describe specific example embodiments, and are not intended to limit the example embodiments. Singular forms are intended to include plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "has," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0046] Unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the example pertains. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0047] When the example embodiments are described with reference to the drawings, the same reference numbers are used throughout the drawings and the same description is omitted. In the description of the example embodiments, when it is considered that a detailed description of related structures or functions known to those skilled in the art will cause obscuring the present disclosure, such description will be omitted.

[0048] Also, in the description of components, when the disclosed components are described, terms such as first, second, A, B, (a), (b), etc. can be used herein. These terms are used only to distinguish one constituent element from another constituent element, and the nature, sequence, or order of the constituent elements are not limited by these terms. When one constituent element is described as being "connected", "coupled", or "attached" to another constituent element, it should be understood that the one constituent element can be directly connected or attached to the other constituent element, and a middle constituent element can also be "connected", "coupled", or "attached" to the constituent elements.

[0049] The same names can be used to describe the elements included in the above-described example embodiments and elements having common functions. Unless otherwise specified, the description of the example embodiments can be applied to the following example embodiments, and thus, repetitive description will be omitted for the sake of brevity.

[0050] FIG. 1A A head-up display (HUD) apparatus according to an example embodiment is illustrated. Referring to FIG. 1A The HUD apparatus 100 includes a HUD control device 110, a display apparatus 120, an eye tracking camera 130, and a semi-transparent optical apparatus 140. The HUD apparatus 100 can be installed on a vehicle (e.g., a car or an airplane) to provide a HUD image to a user (e.g., a driver, a pilot, etc.). The HUD apparatus 100 can provide the HUD image using augmented reality (AR). For example, the content provided by the AR HUD can include dashboard information, navigation information, a lane indicator, a construction indicator, an accident indicator, a pedestrian detection indicator, etc. The AR can be applied to a HUD, a see-through head-mounted display (HMD), etc. Hereinafter, the HUD will be described. However, the following description can also be applied to the HMD or other display apparatuses.

[0051] The display apparatus 120 can include a light source, a display panel, a three-dimensional (3D) optical layer, and an optical assembly. The optical assembly can include a cata-dioptric system. Light corresponding to the HUD image can be provided by the display panel and the light source of the display apparatus 120, and the cata-dioptric system can reflect the light corresponding to the HUD image toward the semi-transparent optical apparatus 140. In this case, the cata-dioptric system can refract the light corresponding to the HUD image to magnify the HUD image. A light-emitting diode (LED) or a laser can be used as the light source.

[0052] The virtual screen 150 can be formed by light corresponding to the HUD image output by the display device 120. A portion of the light output by the display device 120 can be reflected by the semi-transparent optical device 140 located in front of the user and be seen by the user. The semi-transparent optical device 140 can be a windshield of a car or an airplane, or a combiner provided separately from the windshield to reflect the HUD image. The user simultaneously views light in front passing through the semi-transparent optical device 140, and a portion of the light emitted by the display device 120 that is reflected by the semi-transparent optical device 140. Accordingly, real objects and virtual objects can overlap each other and be provided to the user as AR content. For example, the real objects can be objects in the surrounding environment visible through the semi-transparent optical device 140.

[0053] The HUD control device 110 can display a virtual object at a position corresponding to a real object. For example, driving direction information, lane information, danger information, etc. of a vehicle can be displayed as a virtual object at a position corresponding to a real object through the HUD. The position on the background or surrounding environment where the virtual object is to be displayed can be referred to as a target position. The HUD control device 110 can display the virtual object at the target position using a transformation relationship between a coordinate system of the eye tracking camera 130 and a coordinate system of the virtual screen 150, 3D information about the background, and eye position information.

[0054] For example, the 3D information about the background can be obtained through a camera or a 3D sensor disposed to face the front of the vehicle. The eye position of the user can be obtained through the eye tracking camera 130 disposed to face the user. The eye tracking camera 130 can photograph the user and generate a user image including the user (e.g., the face of the user), and the HUD control device 110 can obtain the eye position by performing eye tracking on the user image. The HUD control device 110 can generate a HUD image for displaying a virtual object at an intersection where a line connecting the eye position of the user and the target position intersects the virtual screen 150, and the display device 120 can represent the virtual object at the target position by displaying the HUD image.

[0055] The display apparatus 120 can provide a 3D image through a 3D optical layer. The HUD control device 110 can generate a first source image (e.g., a left image) of a first viewpoint (e.g., a left eye) and a second source image (e.g., a right image) of a second viewpoint (e.g., a right eye), and render a HUD image based on an eye position tracked by the eye tracking camera 130, the first source image, and the second source image. Here, the viewpoint can correspond to a viewing position (e.g., a position of an eye of a viewer). The operation of rendering the HUD image can include determining pixel values of the HUD image such that the first source image can be viewed at the first viewpoint and the second source image can be viewed at the second viewpoint. Hereinafter, an example in which two viewpoints are used for a 3D image will be described. However, the disclosed embodiments are not limited to the following description, and thus, according to another example embodiment, two or more viewpoints can be used for a light field.

[0056] The display apparatus 120 can display the HUD image generated as described above. The display apparatus 120 can display the HUD image on a display panel. The HUD image can pass through a 3D optical layer and be provided to a user. In this case, different images corresponding to the first source image and the second source image can be provided to both eyes of the user. The HUD control device 110 can render the HUD image for each of the first source image and the second source image such that the virtual object can be displayed at an intersection point at which a line connecting each eye position of the user and a target position intersects the virtual screen 150.

[0057] FIG. 1B A path of light related to a HUD apparatus according to an example embodiment is illustrated. Referring to FIG. 1B , the HUD apparatus 101 includes a display apparatus 161 and mirrors 162 and 163. The display apparatus 161 can correspond to the display apparatus 120 of FIG. 1A . The display apparatus 161 can include a display panel and a light source, and provide light corresponding to a HUD image through the display panel and the light source. For example, the light source can include a backlight unit (BLU).

[0058] The light corresponding to the HUD image output by the display apparatus 161 can be reflected by the mirrors 162 and 163 and projected onto the windshield 170. At least one of the mirrors 162 and 163 can correspond to an aspheric mirror, and adjust a path of the light corresponding to the HUD image to magnify the HUD image. The user can view a virtual image corresponding to the HUD image on a virtual screen 180 through light reflected by the windshield 170 toward the viewing frame 190.

[0059] In this way, the HUD device 101 can display information by projecting on a virtual screen 180 disposed in front of the user. In order to provide AR information through the HUD, the virtual screen 180 on which the HUD image is viewed can be implemented with a wide field of view (FOV). If the size of the image to be represented is not large enough or the FOV is not wide enough, it can be difficult to represent information about objects or backgrounds in front of the vehicle using AR.

[0060] The display device 161 and the mirrors 162 and 163 can be installed in the dashboard of the vehicle. The display device 161 and the mirrors 162 and 163 can be designed to provide a wide enough FOV to implement AR. For example, the BLU of the display device 161 can use a secondary lens array to optimize the output angle of light output from the LED and use a side reflector to compensate for the lack of output angle. In this case, the diffusion angle of the diffuser plate and the polarized diffuser plate can be maintained at a small value, so that the efficiency of the BLU can be prevented from decreasing. Accordingly, a compact BLU volume, a wide FOV, uniformity, improved side brightness, etc. can be implemented.

[0061] FIG. 2 A structure of a display device according to an example embodiment is illustrated. Referring to FIG. 2 , the display device 200 includes a light source 210, a diffuser 220, a display panel 230, and a 3D optical layer 240. The light source 210 can correspond to the BLU. According to an example embodiment, the light source can include a white LED, a red / green / blue (RGB) LED, or an RGB laser. If a non-spherical mirror is used as a magnifying and reflecting mirror, any one of the white LED, the RGB LED, and the RGB laser can be used. However, if a holographic mirror is used, the RGB LED or the RGB laser can be used according to a recording characteristic. The diffuser 220 can be implemented in the form of a film, and light uniformity between the light source 210 and the display panel 230 can be provided by the diffuser 220. According to an example embodiment, the diffuser 220 can be formed directly on the display panel 230. According to another example embodiment, the diffuser 220 can be spaced apart from the display panel 230.

[0062] The display panel 230 can include a liquid crystal display (LCD) panel or a spatial light modulator (SLM) such as a digital light processor (DLP) and a liquid crystal on silicon (LCoS). The 3D optical layer 240 can be any one of a parallax barrier, a lenticular lens, and a directional backlight unit. The display panel 230 can display a HUD image, and the 3D optical layer 240 can control the path of light corresponding to the HUD image. For example, the 3D optical layer 240 can impart directionality to light corresponding to the HUD image so that images of different viewpoints can be provided to the user's both eyes.

[0063] FIG. 3 This illustrates 3D AR according to an example embodiment. (Refer to...) FIG. 3 A virtual object 335 is displayed at the intersection of the line connecting the user's eye position 310 and the target position 350 with the virtual screen 330. The eye position 310 can be tracked by an eye-tracking camera 320. In this case, the scene 331 can be viewed from the eye position 310. The scene 331 includes the virtual object 335 and the real object 340. By using the relationship between the coordinate system of the eye-tracking camera 320 and the coordinate system of the virtual screen 330, 3D information about the background, and information about the eye position 310, the virtual object 335 can be accurately displayed at the target position 350.

[0064] This processing can be performed for each of the user's eyes. For example, a first source image (e.g., the left image) can be generated such that virtual object 335 can be displayed at the intersection of the line connecting the first viewpoint (e.g., the left eye) and the target position 350 with the virtual screen 330, and a second source image (e.g., the right image) can be generated such that virtual object 335 can be displayed at the intersection of the line connecting the second viewpoint (e.g., the right eye) and the target position 350 with the virtual screen 330. Subsequently, by rendering HUD images based on the first and second source images, scene 331 can be implemented as a 3D AR HUD.

[0065] According to an example embodiment, the 3D HUD can represent a virtual object 335 at various depths in response to changes in the user's position. Therefore, the 3D HUD can display the virtual object 335 more accurately at the target position 350 than a 2D HUD. However, in order to stably provide such a 3D HUD, continuous tracking of the eye position 310 may be necessary, and the virtual object 335 can be displayed on the virtual screen 330 based on the tracked eye position 310.

[0066] In one example scenario, eye position 310 may not be tracked due to environmental factors (such as low light) or because the eye is obscured. Furthermore, the appropriate HUD image corresponding to the current eye position 310 may not be generated due to system factors (such as system latency). In this example, 3D HUD quality degradation may occur (e.g., crosstalk observed in the image when the image for the left eye is provided to the right eye). In this case, driving information can be stably provided by providing a 2D HUD instead of a low-quality 3D HUD. According to the example embodiment, 2D rendering or 3D rendering can be selectively performed based on the current environment associated with eye tracking, thereby improving HUD stability.

[0067] FIG. 4A process of generating a HUD image according to an example embodiment is illustrated. The operations 410 to 440 described below can be performed on a current frame of a user image. Referring to FIG. 4, the process of generating a HUD image according to an example embodiment is illustrated. FIG. 4 In operation 410, the HUD control device performs eye tracking. For example, the HUD control device can generate a user image using an eye tracking camera and perform eye tracking on the user image. The HUD control device can generate an eye tracking result while performing eye tracking. If the eye tracking is successful, the eye tracking result can include eye coordinates. If the eye tracking fails, the eye tracking result can not include the eye coordinates. Instead, the eye tracking result can include information indicating that the eye tracking has failed. The eye coordinates can include 3D coordinate values for each eye.

[0068] In operation 420, the HUD control device determines an eye tracking state. For example, the HUD control device can classify the eye tracking state as one of a stable state and an unstable state based on whether the eye tracking result meets 3D rendering conditions. Here, the 3D rendering conditions can be defined based on the presence of eye coordinates and the speed of change of the eye coordinates. As described above, in order to maintain the quality of the 3D HUD image, the eye coordinates must be confirmed, and system performance for tracking changes in the eye coordinates is required.

[0069] For example, if there are no eye coordinates, or if there are eye coordinates but the eye coordinates change so severely that the rendering performance of the system cannot cope with the change, crosstalk can be observed in the 3D HUD image. Accordingly, in a first state in which the eye tracking result includes the eye coordinates and the speed of change of the position with respect to the eye coordinates is less than or equal to a threshold value, the eye tracking state can be classified as a stable state. In this case, the threshold value can correspond to a system processing rate. Further, in a second state in which the eye tracking result includes the eye coordinates and the speed of change of the position with respect to the eye coordinates is greater than the threshold value, or in a third state in which the eye tracking result does not include the eye coordinates, the eye tracking state can be classified as an unstable state.

[0070] In operation 430, the HUD control device determines a rendering mode. Here, the determined rendering mode can be used to render a HUD image corresponding to the current frame of the user image. The rendering mode can include a 2D rendering mode and a 3D rendering mode. The HUD control device can determine the rendering mode of the HUD image as one of the 2D rendering mode and the 3D rendering mode based on the eye tracking state. For example, if the eye tracking state is classified as a stable state, the HUD control device can determine the rendering mode as the 3D rendering mode. Conversely, if the eye tracking state is classified as an unstable state, the HUD control device can determine the rendering mode as the 2D rendering mode.

[0071] The HUD control device can render the HUD image such that the same HUD image is provided to both eyes of the user in the 2D rendering mode, or the HUD control device can render the HUD image such that different images are provided to both eyes of the user in the 3D rendering mode. For example, the HUD control device can generate a first source image (e.g., a left image) for a first viewpoint (e.g., a left eye) and a second source image (e.g., a right image) for a second viewpoint (e.g., a right eye), and the HUD control device can render the HUD image such that the first source image can be provided to the first viewpoint and the second source image can be provided to the second viewpoint. If the rendering mode is determined to be the 2D rendering mode, the HUD control device can render the HUD image by setting the first viewpoint and the second viewpoint identically to a single viewpoint. On the contrary, if the rendering mode is determined to be the 3D rendering mode, the HUD control device can render the HUD image by setting the first viewpoint and the second viewpoint to different viewpoints.

[0072] The 2D rendering mode can include a tracking 2D rendering mode and a fixed 2D rendering mode. As described above, in a second state in which the eye tracking result includes eye coordinates and a speed of change in position with respect to the eye coordinates is greater than a threshold value, or in a third state in which the eye tracking result does not include the eye coordinates, the eye tracking state can be classified as an unstable state. In the case of the second state, since the eye coordinates exist, the tracking 2D rendering mode can be performed using the eye coordinates. For example, if the eye tracking result includes current eye coordinates of both eyes and a speed of change in position with respect to the current eye coordinates is greater than a threshold value, the first viewpoint of the first source image and the second viewpoint of the second source image can be identically set to a center viewpoint of the current eye coordinates. On the other hand, in the case of the third state, since the eye coordinates do not exist, the fixed 2D rendering mode can be performed using previously obtained eye coordinates. For example, if the eye tracking result does not include the current eye coordinates, the first viewpoint and the second viewpoint can be identically set to a center viewpoint of the last used previous eye coordinates.

[0073] In operation 440, the HUD control device renders the HUD image in the determined rendering mode. The HUD image can be displayed by the display device and provided to the user through the 3D optical layer. If the 3D rendering is performed, the HUD image can pass through the 3D optical layer such that images of different viewpoints can be provided to both eyes of the user. Even if the 2D rendering is performed, the HUD image can be provided to the user through the 3D optical layer. However, in this case, unlike the 3D rendering, images of the same viewpoint can be provided to both eyes of the user. After operation 440 is performed on the current frame as described above, operations 410 to 440 can be performed on a subsequent frame. This process can be performed on each frame of the user image.

[0074] FIG. 5 An eye tracking state according to an example embodiment is illustrated. Referring toFIG. 5 Eye coordinates at the locations marked with "X" can be obtained for each frame in user image 510 by performing eye tracking on user image 510. Further, since the eye coordinates of each frame do not change much, the eye tracking state of user image 510 can be classified as a stable state.

[0075] For user image 520, eye coordinates at the locations marked with "X" can also be obtained for each frame by eye tracking. However, since the eye coordinates for each frame of user image 520 change a lot, the eye tracking state of user image 520 can be classified as an unstable state. For example, as shown in user image 520, the eye position can change quickly when the vehicle is driving on a speed bump, on an uneven road, or making a sharp turn.

[0076] User image 530, which shows a case of eye tracking failure, does not have "X" marks corresponding to the eye position of each frame. For example, due to environmental factors such as low illumination or occlusion, eye tracking can fail as in user image 530.

[0077] If the eye tracking state is classified as a stable state, as in user image 510, the HUD image can be rendered by a 3D rendering mode. If the eye tracking state is classified as an unstable state, as in user images 520 and 530, the HUD image can be rendered by a 2D rendering mode. If there are eye coordinates in user image 520, a tracking 2D rendering mode can be performed. If there are no eye coordinates in user image 530, a fixed 2D rendering mode can be performed.

[0078] FIG. 6 Eye movement in a viewing space according to an example embodiment is shown. Referring to FIG. 6 , a viewing space 600 includes a first viewing space SI in which a first source image is viewed and a second viewing space S2 in which a second source image is viewed. An eye position 611 is a position of a first viewpoint (e.g., left eye) at time t1, and an eye position 621 is a position of a second viewpoint (e.g., right eye) at time t1. A difference between time t1 and time t2 can correspond to a time difference between two consecutive frames. A user can view the first source image through the first viewpoint of eye position 611 and view the second source image through the second viewpoint of eye position 621. The first viewing space SI and the second viewing space S2 can be divided by a boundary line 630. The HUD control device can adjust the boundary line 630 in response to changes in eye positions 611 and 621 so that eye position 611 can stay in the first viewing space SI and eye position 621 can stay in the second viewing space S2.

[0079] The eye position 611 is a position of the first viewpoint at a time t1, and the eye position 612 is a position of the first viewpoint at a time t2. Also, the eye position 621 is a position of the second viewpoint at the time t1, and the eye position 622 is a position of the second viewpoint at the time t2. Accordingly, a change speed (or a moving speed) of the eye positions 611 and 621 (or eye coordinates) can be defined as V e Also, an adjustment speed (or a moving speed) of the boundary line 630 can be defined as V t . V e and V t may correspond to changes in the eye positions 611 and 621 and changes in the boundary line 630 during a time difference between two consecutive frames. Since a system process such as updating a HUD image is required to adjust the boundary line 630, a maximum value of V t may be limited by a system processing rate. If V e is greater than the maximum value of V t , the eye position 612 of the first viewpoint can be in the second viewing space S2 and the eye position 622 of the second viewpoint can be in the first viewing space S1, for example, as shown in FIG. 6 . Accordingly, crosstalk can be observed.

[0080] The threshold value can be set based on the system processing rate. The threshold value can be a speed of the boundary line 630 that can be adjusted to the maximum value based on the system processing rate. For example, the threshold value can be set to 240 millimeters per second (mm / s). In this case, if the change speed of the eye positions 611 and 621 (or eye coordinates) is greater than the threshold value in a frame of a user image, an eye tracking state for the frame can be determined as an unstable state. Accordingly, the HUD image corresponding to the frame can be rendered by the 2D rendering mode. In detail, since there are eye coordinates, the tracking 2D rendering mode can be used.

[0081] FIG. 7 A process of switching a rendering mode according to an example embodiment is shown. For example, the switching of the rendering mode can include switching from the 2D rendering mode to the 3D rendering mode and switching from the 3D rendering mode to the 2D rendering mode. In another example, the switching of the rendering mode can include switching from one of the tracking 2D rendering mode, the fixed 2D rendering mode, and the 3D rendering mode to another of the tracking 2D rendering mode, the fixed 2D rendering mode, and the 3D rendering mode. While the rendering mode is switched, a change in a viewpoint can occur in the HUD image, and cause a user to feel uncomfortable when viewing the HUD image. According to an example embodiment, in order to reduce such discomfort, the switching of the rendering mode can be performed for a predetermined time.

[0082] Referring to FIG. 7 , in operation 710, the HUD control device determines whether to switch a rendering mode. For example, in the case where the eye positions 611 and 621 (or eye coordinates) are changed at a speed greater than the threshold value, the HUD control device can determine to switch the rendering mode.FIG. 4 After determining the rendering mode in operation 430, in operation 710, it is determined whether to switch the rendering mode based on the determination in operation 430. For example, if the rendering mode iterated for the previous frame is determined to be the 3D rendering mode in operation 430, and the rendering mode iterated for the current frame is determined to be the 2D rendering mode in operation 430, the rendering mode will be switched. In this case, operation 720 can be performed.

[0083] In operation 720, the HUD control device performs a switching operation during a buffer time. The buffer time can correspond to a plurality of frames. For example, if the frame rate of the HUD image is 60 frames per second (fps), the buffer time can correspond to 1 second = 60 frames. If the rendering mode is to be switched from the 3D rendering mode to the 2D rendering mode, the HUD control device can render the HUD image while gradually changing the first viewpoint and the second viewpoint to a single viewpoint used in the 2D rendering mode for the buffer time. The switching operation will be further described with reference to FIG. 8

[0084] If the rendering mode iterated for the previous frame is determined to be the 3D rendering mode in operation 430, and the rendering mode iterated for the current frame is maintained to be the 3D rendering mode in operation 430, the rendering mode will not be switched. In this case, the HUD image corresponding to the current frame can be rendered in the 3D rendering mode by operation 440 of the HUD control device. FIG. 4

[0085] FIG. 8 A buffer viewpoint and a buffer source image for switching the rendering mode according to an example embodiment are illustrated. FIG. 8 An example of switching the rendering mode from the 3D rendering mode to the 2D rendering mode is illustrated. However, the disclosure is not limited thereto, and FIG. 8 The example and the following description of the example can also be applied to another type of switching process according to another example embodiment. Reference is made to FIG. 8 In the viewing space 810, a first viewpoint 820 (e.g., left eye), a second viewpoint 830 (e.g., right eye), and a center viewpoint 840 are illustrated. The center viewpoint 840 can be located in the middle of the first viewpoint 820 and the second viewpoint 830. The buffer viewpoints can be between the first viewpoint 820 and the center viewpoint 840 and between the second viewpoint 830 and the center viewpoint 840. The number of buffer viewpoints can correspond to the buffer time. For example, if the buffer time corresponds to 60 frames, 60 buffer viewpoints can be between the first viewpoint 820 and the center viewpoint 840, and 60 buffer viewpoints can be between the second viewpoint 830 and the center viewpoint 840.

[0086] ​​The first source image 825 can correspond to the first viewpoint 820, the second source image 835 can correspond to the second viewpoint 830, and the center source image 845 can correspond to the center viewpoint 840. Further, there can be buffer source images corresponding to the buffer viewpoints. In the 3D rendering mode, a 3D HUD can be provided by rendering a HUD image based on the first source image 825 and the second source image 835. Further, in the 2D rendering mode, a 2D HUD can be provided by rendering a HUD image based on the center source image 845. If the rendering mode is switched from the 3D rendering mode to the 2D rendering mode, a 3D HUD image can be rendered based on the first source image 825 and the second source image 835 at time t1, a 2D HUD image can be rendered based on the center source image 845 at time t2, and a 2D HUD image can be rendered based on the buffer source images at time t3. The time t2 can be after the time t1, and the time t3 can be after the time t2. B-1 A buffer HUD image can be rendered based on the buffer source images, and can be rendered at time t B A 2D HUD image can be rendered based on the center source image 845. B can correspond to a buffer time.

[0087] The buffer source images can be generated based on an interpolation operation from the first source image 825, the second source image 835, and the center source image 845. For example, a buffer source image corresponding to a buffer viewpoint between the first viewpoint 820 and the center viewpoint 840 can be generated using the first source image 825 and the center source image 845 through an interpolation operation, and a buffer source image corresponding to a buffer viewpoint between the second viewpoint 830 and the center viewpoint 840 can be generated using the second source image 835 and the center source image 845 through an interpolation operation.

[0088] FIG. 9 An image in the 3D rendering mode according to an example embodiment is illustrated. Referring to FIG. 9 , the first source image 910 includes a virtual object 915 at a position corresponding to a first viewpoint, and the second source image 920 includes a virtual object 925 at a position corresponding to a second viewpoint. In FIG. 9 , the first viewpoint and the second viewpoint are different viewpoints. The first source image 910 is generated for displaying the virtual object 915 at an intersection point at which a line connecting the first viewpoint and a target position intersects a virtual screen, and the second source image 920 is generated for displaying the virtual object 925 at an intersection point at which a line connecting the second viewpoint and the target position intersects the virtual screen. A HUD image 930 can be generated based on the first source image 910 and the second source image 920 through 3D rendering, so that a user viewing the HUD image 930 can experience a 3D virtual object 915.

[0089] FIG. 10 An image in the 2D rendering mode according to an example embodiment is illustrated. Unlike FIG. 9 the example, FIG. 10The first viewpoint and the second viewpoint in the first source image 1010 and the second source image 1020 correspond to the same viewpoint. Thus, the virtual object 1015 of the first source image 1010 and the virtual object 1025 of the second source image 1020 are located at the same position. The HUD image 1030 can be generated based on the first source image 1010 and the second source image 1020 through 2D rendering.

[0090] For example, FIG. 9 The first source image 910 and the second source image 920 can correspond to FIG. 8 The first source image 825 and the second source image 835 of FIG. 8A can correspond to FIG. 10 The first source image 1010 and the second source image 1020 can correspond to FIG. 8 The center source image 845 of FIG. 8B. Also, the HUD image 1030 can be generated based on the first source image 910, the second source image 920, and the first source image 1010 (or the second source image 1020) through an interpolation operation. FIG. 8 The buffer source image of FIG. 8C.

[0091] FIG. 11 and FIG. 12 FIG. 9 illustrates a process of tracking an eye using a tracking region according to an example embodiment. The process of tracking an eye can include using the tracking region described below. Referring to FIG. 9, FIG. 11 Operation 1110 and operation 1111 are operations performed on a first frame F1 of a user image. In operation 1110, the HUD control device performs eye detection on the entire image region of the first frame F1. For example, the HUD control device can determine a detection region corresponding to an eye by scanning the entire image. In operation 1111, the HUD control device determines a tracking region. The HUD control device can determine the tracking region based on the detection region. For example, the size of the tracking region can be determined based on the size of the detection region, and the position of the tracking region can be determined as including the detection region at its center.

[0092] Next, operation 1121 to operation 1123 are operations performed on a second frame F2 of a user image. In operation 1121, the HUD control device performs eye tracking based on the tracking region. For example, the HUD control device can detect an eye within the tracking region by scanning the tracking region instead of scanning the entire image region of the second frame F2. Such eye detection using the tracking region can be referred to as eye tracking. In operation 1122, the HUD control device determines whether an eye is present in the tracking region. If an eye is in the tracking region, in operation 1123, the HUD control device updates the tracking region. The tracking region can be updated based on the detection region of the second frame F2 in the same manner as the tracking region is determined based on the detection region of the first frame F1.

[0093] Next, operations 1130 to 1133 are operations performed on the third frame F3 of the user image. In response to the determination in operation 1122 that there is no eye in the tracking region, in operation 1130, the HUD control device terminates the tracking mode and returns to the detection mode to perform eye detection in the entire image region of the third frame F3. If the tracking region is updated in operation 1123 in response to the determination in operation 1122 that there is an eye in the tracking region, in operation 1131, the HUD control device performs eye tracking based on the updated tracking region. Details of operations 1131 to 1133 are the same as those of operations 1121 to 1123. As described above, if there is an eye in the tracking region, the tracking region can be updated and the tracking mode can be maintained. However, if there is no eye in the tracking region, the detection mode can be activated again to scan the entire image.

[0094] Referring to FIG. 12 , frames F1, F2, and F3 of a user image are shown. The HUD control device determines a detection region in frame F1, and determines a first tracking region 1210 based on the detection region. The eye in frame F2 can be in a position more to the right and up than the eye in frame F1. The HUD control device detects the eye from the first tracking region 1210 in frame F2. Since the eye is detected in the first tracking region 1210, the HUD control device updates the tracking region based on the detection region within the first tracking region 1210 in frame F2. Accordingly, a second tracking region 1220 is determined. In the same manner as in frame F2, the eye can be detected from the second tracking region 1220 in frame F3, and a third tracking region 1230 can be determined by updating the tracking region. In this way, in response to determining that the eye is included in each tracking region, the HUD control device can track the eye without scanning the entire image.

[0095] FIG. 13 A process of generating a HUD image based on eye tracking according to an example embodiment is shown. Referring to FIG. 13 , in operation 1310, the HUD control device determines an eye tracking state. In operation 1320, the HUD control device determines whether the eye tracking state is a stable state. According to the example embodiment described with reference to FIG. 11 and FIG. 12 , eye tracking can be performed using a tracking region.

[0096] If the eye tracking state corresponds to the stable state, the HUD control device performs 3D rendering in operation 1330. If the eye tracking state does not correspond to the stable state, the HUD control device performs 2D rendering in operation 1340. For example, if the eye position is determined in the tracking area through eye tracking, but the change speed of the eye position is greater than a threshold value, tracking 2D rendering can be performed in operation 1341. When the eye position is not determined in the tracking area through eye tracking, fixed 2D rendering can be performed in operation 1342. In this case, the tracking mode of eye tracking can be canceled, and the detection mode can be activated again.

[0097] While 2D rendering is performed through operation 1340, the HUD control device checks whether the state is changed in operation 1350. For example, cases of the state change can include a case in which the change speed of the eye position is reduced to be lower than or equal to a threshold value while tracking 2D rendering is performed, and a case in which the change speed of the eye position is less than or equal to a threshold value while fixed 2D rendering is performed although the eye is detected outside the tracking area. If the state is changed, the HUD control device switches the rendering mode from the 2D rendering mode to the 3D rendering mode during a buffer time in operation 1360. Then, the HUD control device performs 3D rendering in operation 1330.

[0098] Similarly, while 3D rendering is performed through operation 1330, the HUD control device checks whether the state is changed in operation 1370. For example, cases of the state change can include a case in which the change speed of the eye position exceeds a threshold value, and a case in which the eye is not detected in the tracking area. If the state is changed, the HUD control device switches the rendering mode from the 3D rendering mode to the 2D rendering mode during a buffer time in operation 1380. Then, the HUD control device performs 2D rendering in operation 1340.

[0099] FIG. 14 A method of controlling a HUD in consideration of an eye tracking state according to an example embodiment is illustrated, and an example of FIG. 4 is further described with reference to FIG. 14 In operation 1410, the HUD control device generates an eye tracking result by performing eye tracking on a user image. In operation 1420, the HUD control device determines an eye tracking state related to a change in an eye position based on the eye tracking result. In operation 1430, the HUD control device determines a rendering mode of a HUD image to be one of a 2D rendering mode and a 3D rendering mode based on the eye tracking state. In operation 1440, the HUD control device renders the HUD image in the determined rendering mode. Further, with reference to FIGS. 1A-13 , FIG. 15 and FIG. 16The description provided can be applied to the method of controlling the HUD, and thus, for the sake of brevity, a detailed description will be omitted.

[0100] FIG. 15 A configuration of a HUD control device according to an example embodiment is illustrated. Referring to FIG. 15 , the HUD control device 1500 includes a processor 1510 and a memory 1520. The memory 1520 is connected to the processor 1510 and can store instructions executable by the processor 1510, data to be calculated by the processor 1510, or data processed by the processor 1510. The memory 1520 can include a non-transitory computer readable medium (e.g., a high-speed random access memory) and / or a non-volatile computer readable medium (e.g., a disk storage, a flash memory device, or another non-volatile solid state memory device). However, the disclosure is not limited thereto, and according to another example embodiment, the memory 1520 can be a storage device configured to store data, information, and / or instructions.

[0101] The processor 1510 can execute instructions to perform operations described with reference to FIGS. 1A-14 and FIG. 16 . For example, the processor 1510 can generate an eye tracking result by performing eye tracking on a user image, determine an eye tracking state related to a change in an eye position based on the eye tracking result, determine a rendering mode of a HUD image as one of a 2D rendering mode and a 3D rendering mode based on the eye tracking state, and render the HUD image in the determined rendering mode. Further, the description provided with reference to FIGS. 1A-14 and FIG. 16 can be applied to the HUD control device 1500, and thus, for the sake of brevity, a detailed description will be omitted.

[0102] FIG. 16 A configuration of an electronic device according to an example embodiment is illustrated. Referring to FIG. 16 , the electronic device 1600 can acquire a user image, track an eye from the acquired user image, and provide an AR HUD image based on an eye tracking state. The electronic device 1600 can structurally and / or functionally include the HUD device 100 of FIG. 1A , the HUD control device 110 of FIG. 1A , and / or the HUD control device 1500 of FIG. 15 . For example, the HUD device 100 of FIG. 1A , the control device 110 of FIG. 1A , and / or the HUD control device 1500 of FIG. 15 may be implemented as the electronic device 1600.

[0103] The electronic device 1600 can include a processor 1610, a memory 1620, a camera 1630, a storage 1640, an input device 1650, an output device 1660, and a network interface 1670. However, the disclosure is not limited thereto, and thus, according to another example embodiment, the electronic device 1600 can include other components, or FIG. 16 One or more of the components shown in FIG. 16 can be omitted from the electronic device 1600. The processor 1610, the memory 1620, the camera 1630, the storage 1640, the input device 1650, the output device 1660, and the network interface 1670 can communicate with each other through the communication bus 1680. For example, the electronic device 1600 can be implemented as a part of a vehicle such as a car or an airplane.

[0104] The processor 1610 executes instructions or functions to be executed in the electronic device 1600. For example, the processor 1610 can process instructions stored in the memory 1620 or the storage 1640. The processor 1610 can execute operations described below. FIGS. 1A-15

[0105] The memory 1620 stores various data for providing a HUD image. The memory 1620 can include a computer readable storage medium or a computer readable storage device. The memory 1620 can store instructions to be executed by the processor 1610, and can store related information when the electronic device 1600 executes software and / or an application.

[0106] The camera 1630 can photograph a photo and / or a video. For example, the camera 1630 can photograph a user image including a user (e.g., a face of the user). In detail, the camera 1630 can include an eye tracking camera 130. FIG. 1A The camera 1630 can provide a 3D image including depth information about an object.

[0107] The storage 1640 includes a computer readable storage medium or a computer readable storage device. The storage 1640 can store various data for providing a HUD image. Compared with the memory 1620, the storage 1640 can store a larger amount of information for a long time. For example, the storage 1640 can include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other non-volatile memory known in the art.

[0108] The input device 1650 can receive an input from a user through a conventional input method such as a keyboard and a mouse, and a new input method such as a touch input, a voice input, and an image input. For example, the input device 1650 can include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects an input from a user and transmits the detected input to the electronic device 1600.

[0109] ​The output device 1660 can provide output of the electronic device 1600 to the user through a visual, auditory, or tactile channel. The output device 1660 can include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides output to the user. In detail, the output device 1660 can include a display device 120 of FIG. 1. FIG. 1A The network interface 1670 can communicate with an external device through a wired network or a wireless network.

[0110] The method according to the above-described examples can be recorded in non-transitory computer-readable media including program instructions to implement various operations of the above-described example embodiments. The media can also include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the media can be specially designed and configured for the purposes of the example embodiments, or they can be of the type well known and available to those having skill in the computer software field. Examples of non-transitory computer-readable media include magnetic media, such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROM disks, DVDs, and / or Blu-ray disks; magneto-optical media, such as optical disks; and hardware devices that are specially configured to store and perform program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., a USB flash drive, a memory card, a memory stick, and the like). Examples of program instructions include both machine code, such as produced by a compiler, and files containing a higher level code that can be executed by the computer using an interpreter. The device can be configured to function as a software module to perform the operations of the above-described examples, or vice versa.

[0111] The software can include a computer program, a code segment, an instruction, or some combination thereof to independently or collectively instruct or configure a processing device to operate as desired. The software and data can be permanently or temporarily stored in any type of machine, component, physical or virtual device, computer storage medium or device, or propagated signal waves capable of instructing a processing device or being interpreted by a processing device. The software can also be distributed over networked computer systems so that the software is stored and executed in a distributed fashion. The software and data can be stored by a non-transitory computer-readable recording medium.

[0112] A number of example embodiments have been described above. However, it should be understood that various modifications can be made to the example embodiments. For example, suitable results can be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in a different manner and / or replaced or supplemented by other components or equivalents thereof. Therefore, other implementations are within the scope of the claims.

Claims

1. A method for controlling a head-up display, the method comprising: performing eye tracking of eyes of a user in a photographed image; confirming an eye tracking state based on a result of the eye tracking; confirming a rendering mode of a head-up display image as one of a two-dimensional rendering mode and a three-dimensional rendering mode based on the eye tracking state; and rendering the head-up display image in the confirmed rendering mode, wherein the confirming of the eye tracking state comprises classifying the eye tracking state as one of a stable state and an unstable state based on whether eye coordinates exist in the result of the eye tracking and / or based on a change speed of the eye coordinates, wherein the confirming of the rendering mode comprises: confirming the rendering mode as the three-dimensional rendering mode based on the eye tracking state being classified as the stable state; and confirming the rendering mode as the two-dimensional rendering mode based on the eye tracking state being classified as the unstable state, wherein the eye tracking state is classified as the stable state based on the eye coordinates being included in the result of the eye tracking and the change speed of the eye coordinates being less than or equal to a reference value. The reference value corresponds to a system processing rate.

2. The method of claim 1, wherein, The eye tracking state is classified as the unstable state based on the eye coordinates being included in the result of the eye tracking and the change speed of the eye coordinates being greater than the reference value, or based on the eye coordinates not being included in the result of the eye tracking.

3. The method of claim 1, wherein, The head-up display image is rendered based on a first source image of a first viewpoint and a second source image of a second viewpoint.

4. The method of any one of claims 1 to 3, wherein, The rendering of the head-up display image comprises rendering the head-up display image by setting the first viewpoint and the second viewpoint as a same viewpoint based on the confirmed rendering mode being the two-dimensional rendering mode.

5. The method of claim 4, wherein, The rendering of the head-up display image comprises:

6. The method according to claim 5, wherein, setting the first viewpoint and the second viewpoint as a same viewpoint as a center viewpoint of current eye coordinates based on the result of the eye tracking including the current eye coordinates and the change speed of the current eye coordinates being greater than a reference value; and setting the first viewpoint and the second viewpoint as a same viewpoint as a center viewpoint of previous eye coordinates based on the result of the eye tracking not including the current eye coordinates. The rendering of the head-up display image comprises rendering the head-up display image by setting the first viewpoint and the second viewpoint as different viewpoints based on the confirmed rendering mode being the three-dimensional rendering mode.

7. The method of claim 4, wherein, The rendering mode is switched from the three-dimensional rendering mode to the two-dimensional rendering mode or from the two-dimensional rendering mode to the three-dimensional rendering mode during a buffering time corresponding to a plurality of frames based on the rendering mode being confirmed.

8. The method of any one of claims 1 to 3, wherein, The head-up display image is rendered based on a first source image of a first viewpoint and a second source image of a second viewpoint, and 9. The method of claim 8, wherein, wherein the rendering of the head-up display image comprises rendering the head-up display image while gradually changing the first viewpoint and the second viewpoint to a single viewpoint used in the two-dimensional rendering mode during the buffering time based on the rendering mode being switched from the three-dimensional rendering mode to the two-dimensional rendering mode. 10.An apparatus for controlling a head-up display, the apparatus comprising: a memory configured to store one or more instructions, and a processor configured to execute the one or more instructions to: perform eye tracking of eyes of a user in a photographed image, confirm an eye tracking state based on a result of the eye tracking, ​ confirming a rendering mode of a heads-up display image as one of a two-dimensional rendering mode and a three-dimensional rendering mode based on an eye tracking state, and rendering the heads-up display image in the confirmed rendering mode, wherein the processor is further configured to: classify the eye tracking state as one of a stable state and an unstable state based on whether the eye coordinates exist in a result of the eye tracking and / or based on a change speed of the eye coordinates, confirm the rendering mode as the three-dimensional rendering mode based on the eye tracking state being classified as the stable state, and confirm the rendering mode as the two-dimensional rendering mode based on the eye tracking state being classified as the unstable state, wherein the processor is configured to classify the eye tracking state as the stable state based on the eye coordinates being included in the result of the eye tracking and the change speed of the eye coordinates being less than or equal to a reference value.

11. The apparatus of claim 10, wherein, switch the rendering mode from the three-dimensional rendering mode to the two-dimensional rendering mode or from the two-dimensional rendering mode to the three-dimensional rendering mode during a buffering time corresponding to a plurality of frames based on the rendering mode being confirmed. 12.A heads-up display apparatus comprising: an eye tracking camera configured to photograph an image including a user; a processor configured to perform eye tracking on the photographed image, confirm an eye tracking state based on a result of the eye tracking, confirm a rendering mode of a heads-up display image as one of a two-dimensional rendering mode and a three-dimensional rendering mode based on the eye tracking state, and render the heads-up display image in the confirmed rendering mode; and a display apparatus configured to provide the heads-up display image to the user using augmented reality, wherein the processor is further configured to: classify the eye tracking state as one of a stable state and an unstable state based on whether the eye coordinates exist in a result of the eye tracking and / or based on a change speed of the eye coordinates, confirm the rendering mode as the three-dimensional rendering mode based on the eye tracking state being classified as the stable state, and confirm the rendering mode as the two-dimensional rendering mode based on the eye tracking state being classified as the unstable state, wherein the processor is configured to classify the eye tracking state as the stable state based on the eye coordinates being included in the result of the eye tracking and the change speed of the eye coordinates being less than or equal to a reference value.

13. The head-up display device of claim 12, wherein, The processor is further configured to switch the rendering mode from the three-dimensional rendering mode to the two-dimensional rendering mode or from the two-dimensional rendering mode to the three-dimensional rendering mode during a buffering time corresponding to a plurality of frames based on the rendering mode being confirmed. 14.A display device comprising: a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions to: receive eye tracking information about eyes of a user in a photographed image; confirm whether eye tracking is stable or unstable based on the eye tracking information; output a virtual object in a two-dimensional image based on the eye tracking being unstable; and output the virtual object in a three-dimensional image based on the eye tracking being stable, wherein the processor is configured to confirm the eye tracking to be stable based on eye coordinates being included in a result of the eye tracking and a change speed of the eye coordinates being less than or equal to a reference value. 15.A display method comprising: ​ receiving eye tracking information about eyes of a user in an image; determining whether the eye tracking is stable or unstable based on the eye tracking information; outputting the virtual object in a two-dimensional image based on the eye tracking being unstable; and outputting the virtual object in a three-dimensional image based on the eye tracking being stable, wherein it is determined that the eye tracking is stable based on eye coordinates being included in a result of the eye tracking and a change speed of the eye coordinates being less than or equal to a reference value. 16.A non-transitory computer-readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 9 and 15.

Images