Vehicle head-up display system with augmented reality and visual tracking device

Abstract

A head-up display system for a vehicle has a controller having a processor and tangible non-transitory memory with instructions recorded thereon. The controller is adapted to project a virtual image on a screen in a vehicle. The virtual image is superimposed on a real-world view of the vehicle visible to the operator. The at least one tracking device is configured to obtain visual data of an operator. The controller is adapted to determine a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing the screen. The controller is adapted to identify a visual category of the operator. The controller is adapted to modify the virtual image projected on the screen based in part on the gaze category and the visual classification.

Description

[0001] INTRODUCTION The present disclosure relates to heads-up display systems in vehicles. More specifically, the present disclosure relates to vehicle heads-up display systems with augmented reality and visual tracking devices, and methods of operating the display systems. Heads-up display systems in vehicles allow drivers to view virtual images superimposed on real-world views through their windshields, providing them with information without having to remove their eyes from the road. This virtual information is presented at a fixed focal distance from the driver's eyes. However, there are challenges when an operator's eyes attempt to focus on both the projected virtual information and real-world scenery outside the windshield at different distances simultaneously. Conflicting depth information can result in visual challenges. SUMMARY

[0002] Disclosed herein is a display system for an operator in a vehicle. The system includes a controller having a processor and a tangible non-transitory memory having instructions recorded thereon. The controller is adapted to project a virtual image on a screen in the vehicle. The virtual image is superimposed on a real-world view visible to the operator. At least one tracking device is operably connected to the vehicle. The tracking device is configured to obtain visual data of the operator, the controller being adapted to receive the visual data. The controller is adapted to determine a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen. The controller is adapted to identify a visual classification of the operator based on the visual data, the visual classification including whether the operator has binocular vision or monocular vision. The controller is adapted to modify the virtual image projected on the screen based in part on the gaze category and the visual classification.

[0003] The tracking device can include a source and a camera. The source is adapted to emit infrared light, and the camera is adapted to detect infrared light reflected by at least one eye of the operator. Determining the visual classification includes projecting a dynamic dot on the screen. The tracking device is adapted to capture a corresponding change in pupil diameter of the operator as the dynamic dot moves on the screen.

[0004] The virtual image includes at least one bounding box. Modifying the virtual image can include increasing a size of the at least one bounding box when the operator is not gazing at the screen. Modifying the virtual image can include shifting a position of the at least one bounding box on the screen. Modifying the virtual image can include removing one or more boundary lines of the at least one bounding box.

[0005] The system can include an operator feedback module to solicit feedback from the operator regarding display preferences. The operator feedback module is selectively executable by the controller. The controller is adapted to correct the virtual image projected on the screen based in part on the feedback. Soliciting the feedback includes projecting a graphic at a corresponding predetermined location on the screen and asking the operator to rank or rate the visibility of the predetermined location.

[0006] The system can include a facial recognition module accessible by the controller and adapted to store facial profile data associated with the operator. The visual data of the operator is linked with the facial profile data such that when the operator is recognized by the facial recognition module, the previously acquired set of visual data is accessible to the controller.

[0007] Disclosed herein is a method of operating a heads-up display system in a vehicle having a screen and a controller having a processor and a tangible non-transitory memory. The method includes projecting, via the controller, a virtual image on the screen, the virtual image being superimposed on a real-world view visible to an operator. The method includes obtaining, via at least one tracking device in communication with the controller, visual data of the operator. The method includes determining, via the controller, a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen. The method includes identifying, via the controller, a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision. The method includes modifying, via the controller, the virtual image projected on the screen based in part on the gaze category and the visual classification.

[0008] The present disclosure provides the following embodiments.

[0009] 1. A heads-up display system for an operator in a vehicle having a screen, the system comprising: at least one tracking device configured to obtain visual data of the operator; a controller adapted to receive the visual data, the controller having a processor and a tangible non-transitory memory having instructions recorded thereon; wherein the controller is adapted to project a virtual image on the screen, the virtual image being superimposed on a real-world view visible to the operator; wherein the controller is adapted to: determine a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen; identify a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision; and modify the virtual image projected on the screen based in part on the gaze category and the visual classification.

[0010] 2. The system of embodiment 1, wherein the at least one tracking device comprises a source and a camera, the source adapted to emit infrared light and the camera adapted to detect infrared light reflected by at least one eye of the operator.

[0011] 3. The system of embodiment 1, wherein determining the visual classification comprises projecting a dynamic dot on the screen, and wherein the tracking device is adapted to capture respective changes in pupil diameter of the operator as the dynamic dot moves on the screen.

[0012] 4. The system of embodiment 1, wherein: the virtual image comprises at least one bounding box; and modifying the virtual image comprises increasing a size of the at least one bounding box when the operator is not gazing at the screen.

[0013] 5. The system of embodiment 1, wherein: the virtual image comprises at least one bounding box; and modifying the virtual image comprises shifting a position of the at least one bounding box on the screen.

[0014] 6. The system of embodiment 1, wherein: the virtual image comprises at least one bounding box; and modifying the virtual image comprises removing one or more boundary lines of the at least one bounding box.

[0015] 7. The system of embodiment 1, further comprising: an operator feedback module for soliciting feedback from the operator regarding display preferences, the operator feedback module selectively executable by the controller, the controller adapted to correct the virtual image projected on the screen based in part on the feedback.

[0016] 8. The system of embodiment 7, wherein soliciting the feedback comprises projecting a graphic at a respective predetermined location on the screen and asking the operator to rank or rate a visibility of the predetermined location.

[0017] 9. The system of embodiment 1, further comprising: a facial recognition module accessible by the controller and adapted to store facial profile data associated with the operator, the visual data of the operator linked with the facial profile data such that the previously acquired set of visual data is accessible to the controller when the operator is recognized by the facial recognition module.

[0018] 10. A method of operating a heads-up display system in a vehicle, the heads-up display system having a screen and a controller, the controller having a processor and a tangible non-transitory memory, the method comprising: projecting, via the controller, a virtual image on the screen, the virtual image being superimposed on a real-world view visible to the operator; obtaining, via at least one tracking device in communication with the controller, visual data of the operator; determining, via the controller, a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen; identifying, via the controller, a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision; and modifying, via the controller, the virtual image projected on the screen based in part on the gaze category and the visual classification.

[0019] 11. The method of embodiment 10, further comprising: determining the visual classification by projecting a dynamic dot on the screen and capturing, via the at least one tracking device, respective changes in pupil diameter of the operator as the dynamic dot moves on the screen.

[0020] 12. The method of embodiment 10, further comprising: including at least one bounding box in the virtual image and modifying the virtual image by increasing a size of the at least one bounding box when the operator is not gazing at the screen.

[0021] 13. The method of embodiment 10, further comprising: including at least one bounding box in the virtual image and modifying the virtual image by shifting a position of the at least one bounding box on the screen.

[0022] 14. The method of embodiment 10, further comprising: including at least one bounding box in the virtual image and modifying the virtual image by removing one or more boundary lines of the at least one bounding box.

[0023] 15. The method of embodiment 10, further comprising: soliciting feedback from the operator regarding display preferences through an operator feedback module selectively executable by the controller; and correcting, via the controller, the virtual image projected on the screen based in part on the feedback.

[0024] 16. The method of embodiment 15, further comprising: soliciting the feedback by projecting graphics at respective predetermined locations on the screen and requiring the operator to rank or assess a visibility of the predetermined locations.

[0025] 17. The method of embodiment 10, further comprising: facial profile data associated with the operator is stored in a facial recognition module accessible by the controller, and the operator's visual data is linked with the facial profile data such that when the operator is recognized by the facial recognition module, the previously acquired set of visual data is accessible to the controller.

[0026] 18. A vehicle comprising: a screen proximate to the operator and sufficiently transparent such that a real world view is visible to the operator through the screen; a heads-up display system having a controller adapted to project a virtual image on the screen, the virtual image being superimposed on the real world view, the controller having a processor and a tangible non-transitory memory having instructions recorded thereon; at least one tracking device operably connected to the vehicle and configured to obtain visual data of the operator; wherein the controller is adapted to receive the visual data, and: determine a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen; identify a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision; and modify the virtual image projected on the screen based in part on the gaze category and the visual classification.

[0027] 19. The vehicle of embodiment 18, wherein determining the visual classification includes projecting a dynamic dot on the screen, and wherein the tracking device is adapted to capture respective changes in the operator's pupil diameter as the dynamic dot moves on the screen.

[0028] 20. The vehicle of embodiment 19, wherein the virtual image includes at least one bounding box, and modifying the virtual image includes at least one of: increasing a size of the at least one bounding box when the operator is not gazing at the screen; shifting a position of the at least one bounding box on the screen; and removing one or more border lines of the at least one bounding box.

[0029] The above features and advantages of the present disclosure, as well as other features and advantages of the present disclosure, are readily apparent from the following detailed description of the best modes for carrying out the present disclosure, when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is a schematic perspective view of a heads-up display system in a vehicle; Figure 2 is a schematic flowchart of an example method executable by a controller of Figure 1 is a schematic flowchart of an example method executable by a controller ofFigure 3 is a diagram of a sample screen; Figure 4 This diagram shows the operator Figure 1 A schematic diagram of a screen viewing object in a vehicle; Figure 5A 、 5B is another example of an operator passing Figure 1 A schematic diagram of an object viewed from the front screen of a vehicle in which the operator Figure 5A is looking at the screen and Figure 5B is looking at the object; and Figure 6 This diagram shows the Figure 1 Schematic diagram of the various types of virtual elements used in the system. DETAILED DESCRIPTION

[0031] Referring to the drawings, wherein like reference numerals refer to like components, Figure 1 A heads-up display system 10 (hereinafter referred to as the "system") for a vehicle 12 is schematically illustrated. The vehicle 12 may be a mobile platform such as, but not limited to, a passenger car, a sport utility vehicle, a light truck, a heavy truck, an ATV, a minivan, a bus, a transportation vehicle, a bicycle, a robot, agricultural implements (e.g., a tractor), sports-related equipment (e.g., a golf cart), a boat, an airplane, and a train. The vehicle 12 may take many different forms and include multiple and / or alternative components and features. It is understood that the vehicle 12 may take many different forms and have additional components.

[0032] refer to Figure 1 The system 10 includes a processing unit 14 that processes data from multiple sources, and a projection unit 16 that generates and projects images onto a surface (referred to herein as a screen 20). The projection surface of the display or screen 20 is typically the windshield or front screen of the vehicle. The screen 20 is located near the operator 22 and is sufficiently transparent so that the real-world view is visible to the operator through the screen. Alternatively, the screen 20 can be an additional panel constructed of glass or plastic that is positioned between the operator 22 and the windshield on the dashboard.

[0033] In some embodiments, the projection unit 16 is placed on the dashboard of the vehicle 12 and includes a series of mirrors and lenses that reflect, refract, focus, and magnify the image for display. The projected image can travel up the light path between the two planes of the treated laminated glass, where it is displayed on the windshield at the appropriate eye level. The projection unit 16 can be adapted to use an image generating device and a light source to generate a virtual image. The virtual image is projected onto the screen at a specific angle and position. The screen 20 reflects the projected image toward the eyes of the operator 22, creating a virtual image that appears in front of or at the screen 20. In some embodiments, the projection unit 16 includes a dedicated graphics image generator for modifying the graphics for augmentation.

[0034] The human brain can have difficulty fusing virtual information from an augmented reality display with real world scenery, resulting in perceptual conflicts. Additionally, virtual information on an augmented reality display can lack full range depth cues that the brain relies on for accurate depth perception. The system 10 is adapted to project a virtual image on a screen, where the virtual image is superimposed on a real world view visible to the operator 22, helping the driver through challenging driving conditions.

[0035] Reference Figure 1 The system 10 includes a controller C having at least one processor P and at least one memory M (or non-transitory, tangible computer-readable storage medium) having instructions recorded thereon for performing a method 100 for operating the system 10, which is described below with reference to Figure 2 The controller C is adapted to track the pupil(s) of the operator 22 and modify the blended graphics based on the location of the blended graphics, the gaze of the eyes, and the vision (binocular or monocular).

[0036] Reference Figure 1 The system 10 includes at least one tracking device 24 in communication with the controller C and configured to acquire vision data of the operator 22. The tracking device 24 can be employed to measure changes in pupil size and / or reactivity, such as light reflex of pupil size, symmetry. The tracking device 24 can be mounted on a surface of the vehicle 12, such as the dashboard. The tracking device 24 can employ a visible spectrum pupillometer. As described below, the controller C is adapted to determine a gaze category of the operator 22 based on the vision data from the tracking device 24. The gaze category includes a determination of whether the operator 22 is gazing at the screen 20.

[0037] The controller C is adapted to identify a visual classification of the operator 22 based on the visual data, including a binary determination of whether binocular vision or monocular vision is applicable. The controller C is adapted to modify the virtual image projected on the screen based in part on the gaze category and the visual classification. The system 10 can be adapted to further correct the augmentation based on operator feedback using a trial-and-error approach. The system 10 increases the driver’s awareness of the vehicle’s performance and current road events. In some embodiments, the position and / or focal point of the virtual image can be matched to the operator’s 22 line of sight to ensure optimal visibility.

[0038] In one embodiment, the controller C is embedded in the vehicle 12. In another embodiment, the controller C is stored in a “non-vehicle-mounted” or remotely located cloud computing service 26, as shown in Figure 1 The cloud computing service 26 can include one or more remote servers 28 hosted on the Internet to store, manage, and process data. The cloud computing service 26 can be managed at least in part by personnel at different locations, such as at a “back office.” Communication between the various components of the vehicle 12 and the cloud computing service 26 can be conducted over a wireless network 30. The wireless network 30 can be a short-range network or a long-range network.

[0039] Referring to Figure 1 , the controller C can be configured to receive and transmit data through a mobile application 32, which can be installed on a smartphone, laptop, tablet, or other electronic device, and can include a touchscreen interface or I / O device. Circuitry and components of a mobile application (“app”) available to those skilled in the art can be employed.

[0040] Referring now to Figure 2 , a flowchart of a method 100 stored on and executable by the controller C of Figure 1 is shown. The method 100 can be embodied as computer-readable code or instructions stored on and partially executable by the controller C of Figure 1 The memory M can store a set of controller-executable instructions, and the processor P can execute the set of controller-executable instructions stored in the memory M. The method 100 need not be applied in the particular order described herein. Furthermore, it is to be understood that some steps can be omitted.

[0041] According to block 102, the method 100 includes projecting a dynamic dot 130 (see Figure 3 ) on the screen 20 and capturing visual data of an operator (e.g., the operator 150 in Figure 4 and the operator 250 in Figure 5A , 5B via the tracking device 24. Figure 3is a schematic diagram of an example screen 20 having a left side 132, a right side 134, and a central portion 136. The tracking device 24 is adapted to capture respective changes in the pupil diameter of the operator 22 as the dynamic point 130 moves across the screen 20.

[0042] The method 100 proceeds to block 104, where the controller C is programmed to receive and analyze visual data captured by the tracking device 24. The controller C is adapted to obtain gaze data, i.e., a video recording of the eye position of the operator 22 as the operator 22 is looking on the screen 20 or in the world. From the gaze data, a gaze category of the operator is established, including a binary determination of whether the operator 22 is gazing at the screen 20. The operator 22 is classified as gazing at the screen 20 when the gaze position is within a negligible distance of the screen 20.

[0043] In some embodiments, the tracking device 24 includes a source adapted to emit infrared light, and a camera adapted to detect the infrared light reflected by at least one eye of the operator 22. The reflected information can be analyzed to extract eye rotation data from changes in the reflection. The gaze, or the position at which the eye is looking, can be ascertained through filtering and calculation.

[0044] In some embodiments, the tracking device 24 includes a video-based detector that tracks corneal reflections, such as the first Purkinje image and the pupil center, as features for tracking over time. For example, the tracking device 24 can include a dual-Purkinje eye tracker that uses reflections from the front of the cornea (referred to as the first Purkinje image) and the back of the lens (referred to as the fourth Purkinje image) as features for tracking over time. It is to be understood that the tracking device 24 can employ other sensor technologies available to those skilled in the art, with the pupil diameter remaining unchanged or the same.

[0045] Proceeding to block 106, the controller C is programmed to identify a visual classification of the operator 22 based on the visual data, including whether the operator 22 has binocular vision or monocular vision. Monocular vision is characterized by one functioning eye, and lacks the depth perception and three-dimensionality provided by binocular vision. It is to be understood that the cutoff or threshold for determining whether an eye is functioning sufficiently can vary based on the application at hand. Figure 4 is a schematic diagram illustrating an operator 150 with monocular vision looking at a distant object 152 through a screen 120, such as a windshield in a vehicle 12, with a line of sight 154. Figure 4 The operator 150 in is provided with a prosthetic eye 156 and a functioning eye 158.

[0046] If the operator has monocular vision (block 106 =Yes), the method 100 proceeds to block 108 where the controller C is programmed to enhance the display based on the functional eye 158 of the operator 150, including modifying the virtual image. Figure 4 Modifying the virtual image includes shifting the position of the bounding box 160 on the screen 120 based on the functional eye 158. For example, if the functional eye 158 is the right eye, the virtual image may be positioned toward the right side 134 of the screen 120 (see Figure 3 ).

[0047] Figure 5A and 5B An operator 250 with binocular vision is shown viewing a distant object 252 through a screen 220 (e.g., the windshield of a vehicle 12) with a line of sight 254. Binocular vision is the ability to perceive depth and three-dimensionality by combining slightly different images from both eyes. The human eyes are horizontally separated, meaning that each eye receives a slightly different view of the horizontal field of view.

[0048] If the operator has binocular vision (block 106 = No), the method 100 proceeds to block 110 where the controller C is programmed to enhance the display based on the gaze category of the operator 22, including modifying the virtual image. Modifying the virtual image includes increasing the size of the bounding box when the operator 22 is not looking at the screen 20. Figure 5A In FIG, the operator 250's gaze is at the screen 220, which has a bounding box 260A projected thereon. Figure 5B , the operator 250's gaze is not at the screen 220, but instead at the object 252. The controller C is adapted to increase the size of the bounding box when the operator's 250 gaze moves away from the screen 220. Thus, Figure 5B The bounding box 260B projected on the screen 220 is larger than Figure 5A Bounding box 260A in.

[0049] Figure 6 3 is a schematic diagram of various types of virtual images 300 that may be employed in system 10, such as first element 310, second element 312, and third element 314. Virtual image 300 includes at least one bounding box, which may be rectangular or take on various shapes. The virtual image may include text and / or numbers depicting navigation instructions and speed data. Modifying virtual image 300 may include removing one or more boundary lines B of the bounding box (as shown in second element 312). The enhancement may be further modified in shape and position based on operator 22's adjustment and visual acuity. Controller C may be adapted to automatically select the need for accessible enhancement using pupillometric data from tracking device 24. Modifications may further include incorporating depth cues into the virtual information.

[0050] Referring to Figure 1 , the system 10 can include an operator feedback module 34 selectively executable by the controller C for actively soliciting feedback from the operator 22 regarding display preferences. The controller C is adapted to correct the virtual image projected on the screen 20 based in part on feedback from the operator 22. In some embodiments, soliciting feedback from the operator 22 includes projecting a graphic at a corresponding predetermined location on the screen 20 and asking the operator 22 to rank or rate the visibility of that predetermined location. For example, if the operator 22 indicates a bias or preference toward the right side 134 of the screen 120 in Figure 1 , the virtual image can be shifted toward the right side 134. It is to be understood that the screen 120 can be divided into quadrants or other divisions. Figure 3

[0051] Referring to Figure 1 , the system can include a facial recognition module 36 accessible by the controller C and adapted to store facial profile data associated with the operator 22. The visual data of the operator 22 is linked with the facial profile data such that when the operator 22 is recognized by the facial recognition module 36, the previously acquired set of visual data is accessible to the controller C. In other words, a profile of the operator 22 can be created and linked with the facial recognition module 36 to save calibration data. This provides the advantage that calibration does not have to be run every time a particular user is driving.

[0052] In general, the system 10 (via execution of the method 100) utilizes internal vehicle sensors, such as the tracking device 24, to obtain visual data of the operator 22. The tracking device 24 can include a camera-based system, such as an infrared camera, that detects pupil dilation and constriction. The controller C is adapted to recognize the vision (monocular or binocular) and gaze of the operator 22 in the vehicle 12 to modify the full screen graphic or augmentation.

[0053] The wireless network 30 can be a communication bus, which can be in the form of a serial controller area network (CAN bus). The wireless network 30 can be a serial communication bus in the form of a local area network, which can include, but is not limited to, a controller area network (CAN), a controller area network with flexible data rate (CAN-FD), Ethernet, Bluetooth, WIFI, and other forms of data. The wireless network 30 can be a wireless local area network (LAN) that links multiple devices using a wireless distribution method, a wireless metropolitan area network (MAN) that connects several wireless LANs, or a wireless wide area network (WAN) that covers a large area such as a neighborhood town and city. Other types of network technology or communication protocols available to those skilled in the art can be employed.

[0054] ​The controller C includes computer-readable media (also referred to as processor-readable media) including non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of the computer). Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media can include, for example, optical or magnetic disks and other persistent memory. Volatile media can include, for example, dynamic random access memory (DRAM), which can form a main memory in one example. This media can be transported on one or more transmission media, including coxial cables, copper wire, and optical fibers, including wires that comprise a system bus coupled to a computer processor. Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, other magnetic media, a CD-ROM, DVDs, other optical media, other physical media with patterns of holes, a RAM, a PROM, and EPROM, a FLASH- EPROM, other memory chips or cartridges, or any other medium from which a computer can read. The controller C can include a processor, such as one or more processors 102 of the computer 100. The processor(s) 102 can be implemented as various hardware components, such as one or more microprocessors, microcontrollers, digital signal processors, central processing units, graphics processing units, field programmable gate arrays, programmable logic devices, or any other devices suitable for retrieval and execution of instructions. The processor(s) 102 can be implemented as a combination of devices, such as a combination of a microprocessor and a microcontroller, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core or core(s), or any other such configuration.

[0055] The lookup tables, databases, data repositories, or other data storage repositories described herein can include various mechanisms for storing, accessing, and retrieving various data including hierarchical databases, spread sheets, applications with associated databases, relational databases, and the like. Each such data storage repository can be included within a computing device employing a computer operating system such as one of those mentioned above, and can be accessed via one or more of a variety of means well-known in the art including without limitation via the Internet. A file system can be employed to access the data stored in a computer operating system, and can include files stored in various formats including without limitation files stored in a structured language convention, such as HyperText Markup Language (HTML), Extensible Markup Language (XML), and the like. A database can be employed to store, manipulate, and retrieve data, and can include without limitation a relational database, a structured query language (SQL) database, a NoSQL database, and the like. A database can be segmented into one or more databases each of which can employ all or a subset of the database functionalities as provided by an underlying database platform.

[0056] The flow diagrams illustrate the architecture, functionality, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). It should also be noted that each block in the flow diagrams and combinations of blocks in the flow diagrams can be implemented by special-purpose hardware-based systems that perform the specified functions or actions, or combinations of special-purpose hardware and computer instructions. These computer program instructions can also be stored in a computer readable medium that can direct a controller or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the functions / acts specified in the flow diagrams and / or block diagrams.

[0057] The numerical values of parameters (e.g., quantities or conditions) in this specification (including the appended claims) should be understood to be approximations, unless otherwise indicated in the art, whether or not precisely expressed or recited therein. The numerical values of parameters (e.g., quantities or conditions) in this specification (including the appended claims) should be understood to be approximations, unless otherwise indicated in the art, whether or not precisely expressed or recited therein. The "about" indicates that slight deviations from the stated value are acceptable, up to the value of the exactness of the value; about or reasonably close to the value; nearly. If the imprecision provided by "about" is not otherwise understood in the art, then "about" as used herein indicates at least the variations that can be caused by the ordinary methods of measuring and using such parameters. In addition, the disclosure of ranges includes every value and further divided range within the range. Every value within the range and the endpoints of the range are disclosed herein as separate embodiments.

[0058] The specific embodiments and the drawings or figures described herein are illustrative of the principles of the present disclosure and are not meant to limit the scope of the disclosure. While the best mode has been described for practicing the claimed disclosure, various alternative designs and embodiments exist for practicing the disclosure defined in the appended claims. Furthermore, the features of the embodiments shown in the drawings or mentioned in the specification can not necessarily be understood as independent from each other. Rather, every feature described in one of the examples of embodiments can be combined with one or more other desired features from other embodiments, resulting in other embodiments not described in words or through reference to the drawings. Therefore, such other embodiments fall within the scope of the appended claims.

Claims

1. A heads-up display system for an operator in a vehicle having a screen, the system comprising: at least one tracking device configured to obtain visual data of the operator; a controller adapted to receive the visual data, the controller having a processor and a tangible non-transitory memory having instructions recorded thereon; wherein the controller is adapted to project a virtual image on the screen, the virtual image being superimposed on a real-world view visible to the operator; wherein the controller is adapted to: determine a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen; identify a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision; and modify the virtual image projected on the screen based in part on the gaze category and the visual classification.

2. The system of claim 1, wherein the at least one tracking device includes a source and a camera, the source being adapted to emit infrared light and the camera being adapted to detect infrared light reflected by at least one eye of the operator.

3. The system of claim 1, wherein determining the visual classification includes projecting a dynamic dot on the screen, and wherein the tracking device is adapted to capture respective changes in pupil diameter of the operator as the dynamic dot moves on the screen.

4. The system of claim 1, wherein: the virtual image includes at least one bounding box; and modifying the virtual image includes increasing a size of the at least one bounding box when the operator is not gazing at the screen.

5. The system of claim 1, wherein: the virtual image includes at least one bounding box; and modifying the virtual image includes shifting a position of the at least one bounding box on the screen.

6. The system of claim 1, wherein: the virtual image includes at least one bounding box; and modifying the virtual image includes removing one or more border lines of the at least one bounding box.

7. The system of claim 1, further comprising: an operator feedback module for soliciting feedback from the operator regarding display preferences, the operator feedback module being selectively executable by the controller, the controller being adapted to correct the virtual image projected on the screen based in part on the feedback.

8. The system of claim 7, wherein soliciting the feedback includes projecting a graphic at a respective predetermined location on the screen and asking the operator to rank or rate a visibility of the predetermined location.

9. The system of claim 1, further comprising: a facial recognition module accessible by the controller and adapted to store facial profile data associated with the operator, the visual data of the operator being linked with the facial profile data such that a set of previously obtained visual data is accessible to the controller when the operator is recognized by the facial recognition module.

10. A method of operating a heads-up display system in a vehicle, the heads-up display system having a screen and a controller, the controller having a processor and a tangible non-transitory memory, the method comprising: projecting a virtual image on the screen via the controller, the virtual image being superimposed on a real-world view visible to the operator; ​ obtaining visual data of the operator via at least one tracking device in communication with the controller; determining, via the controller, a gaze category of the operator based on the visual data, including a binary determination of whether the operator is gazing at the screen; identifying, via the controller, a visual classification of the operator based on the visual data, the visual classification including binocular vision and monocular vision; and modifying, via the controller, a virtual image projected on the screen based in part on the gaze category and the visual classification.

Images