AUGMENTED REALITY VEHICLE ASSISTANCE FOR COLOR BLINDNESS

Abstract

Method for a vehicle with an augmented reality (“AR”) headset, wherein the method comprises: Determining the location of the vehicle using a GPS unit compliant with Dedicated Short Range Communication (DSRC), with the location having an accuracy of plus or minus 1.5 meters; Identifying an illuminated light in a vehicle's driving environment based on the vehicle's location; Determine that the illuminated light is a point of interest, based on the color of the illuminated light and a driver of the vehicle who is affected by some form of color blindness; Determining a vehicle action to be taken in response to the identification of the illuminated light in the vehicle's driving environment, where the illuminated light is the point of interest; Receiving head position data from one or more internal cameras of the vehicle, wherein the head position data describes an orientation of the driver's head; Determine, based on the head position data, that the orientation of the driver's head indicates that the point of interest lies within the driver's field of vision; and In response to a determination that the point of interest is within the driver's field of vision, an AR overlay is displayed using the AR headset, visually representing a word that describes the vehicle action to be taken in response to the identification of the illuminated light.

Description

BACKGROUND

[0001] The specification concerns an augmented reality system for providing vehicle assistance for color-blind drivers.

[0002] Color blindness, also known as color vision deficiency, is the reduced ability to see colors or differences in colors. Red-green color blindness is the most common form, followed by blue-yellow color blindness and total color blindness. Red-green color blindness affects up to 8% of men and 0.5% of women of Northern European descent. There is no medical cure for color blindness.

[0003] The ability to see colors also declines with age. Being colorblind can make people unfit for certain professions or jobs in certain countries. For example, in some countries, people who are colorblind are unfit to work as pilots, train conductors, or taxi drivers.

[0004] The publication DE 10 2008 020 728 B4 discloses a traffic sign assistance system that has traffic signs or traffic infrastructure elements equipped with transmitting devices for wireless communication. A display device is configured to display information about the direction of travel related to the traffic signs and information. The information is derived from the driving status of a vehicle and the current status of the traffic signs.

[0005] US 2016 0 070 965 A1 discloses a detection system for vehicle operation, including, among other things, at least one camera capable of generating an image and a display unit capable of displaying information on a screen. A control unit is connected to the at least one camera. The control unit is configured to command the display unit to transmit the information in response to detecting data within the image corresponding to visible light emitted in at least one predetermined frequency range.

[0006] The document DE 10 2013 019 550 B3 discloses a method for assisting a driver of a motor vehicle with regard to traffic light switching. SUMMARY

[0007] Described below is an augmented reality system (hereinafter referred to as an "AR system") operable to provide a vehicle with new or different vehicle functionality. Augmented reality may be referred to as "AR" below.

[0008] The AR system may be incorporated into a vehicle. The AR system includes an AR headset and one or more AR gloves worn by a driver of a vehicle.

[0009] Many drivers suffering from color blindness report difficulties while driving. Specifically, these drivers tend to have difficulty interpreting (1) traffic signals and (2) other vehicles' brake lights. For example, a driver with red-green color blindness may sometimes confuse red and green traffic signals, which unfortunately results in traffic accidents. These drivers are also unable to tell whether the brake lights of the vehicle in front of them are on or off. This situation also results in traffic accidents.

[0010] The AR system described herein controls the operation of AR hardware (e.g., a head-up display, "HUD," or AR glasses) and improves the performance of a vehicle by assisting a color-blind driver to understand and correctly interpret their driving environment when operating the vehicle.

[0011] One of the difficulties in designing vehicle systems to assist color-blind drivers is understanding that there are varying degrees of color blindness, and any vehicle system attempting to address the problem of color blindness for vehicle applications must be able to accommodate all degrees of color blindness. In general, existing solutions attempting to address the problem of color blindness for vehicle applications focus on making displays more accessible to people with color blindness by detecting lights using sensors and displays. These existing solutions are inadequate because they are unable to accommodate all degrees of color blindness.Some people with higher levels of color blindness may still have difficulty distinguishing traffic lights, even when supported by existing solutions. Thus, existing solutions are unable to help such drivers understand or correctly interpret their driving environment while driving a vehicle. For comparison, the AR system described here is capable of assisting all color-blind drivers, regardless of their degree of color blindness.

[0012] In some embodiments, the AR system is implemented using an on-board vehicle computer, electronic control unit ("ECU"), or other on-board unit ("OBU") of a vehicle. The AR system is operable to control the operation of one or more of the following elements of the vehicle: (1) an AR headset; and (2) an AR glove. In particular, the AR system operates one or more of the AR headset and the AR glove to assist color-blind drivers of the vehicle in understanding and interpreting traffic signals and brake lights located in their driving environment.

[0013] The AR headset includes a HUD or AR glasses. In some embodiments, the HUD is a three-dimensional HUD ("3D HUD"), such as the one shown in Fig. 5. The color-blind driver views their driving environment through the HUD or AR glasses. If this driving environment includes traffic signals or brake lights, the AR system generates a graphic overlay that graphically depicts one or more words that assist the driver in understanding and correctly interpreting illuminated traffic signals or brake lights within their driving environment. In some embodiments, the words describe the actions the driver should take based on an illuminated traffic signal or brake light within the driving environment. In some embodiments, it is preferred that the number of words used be minimized so that the instruction provided by the graphic overlay can be quickly understood by the driver.

[0014] In an example for traffic lights, in some embodiments, the AR system detects the location of the illuminated traffic light on the HUD or AR headset using a combination of GPS information and image capture algorithms, and then causes the AR headset to display a graphic overlay depicting the words "GO" for a green light, "SLOW DOWN" for a yellow light, and "STOP" for a red light. See, for example, Fig. 4A to 4E.

[0015] In an example for brake lights, in some embodiments, the AR system detects the location of one or more illuminated brake lights of a second vehicle traveling in front of the driver's vehicle using depth sensors and image capture algorithms and then causes the AR headset to display a graphic overlay depicting the word "BRAKE" when the brake lights of the second vehicle are illuminated. If the brake lights of the second vehicle are not illuminated, the graphic overlay is not displayed by the AR headset. See, for example, Fig. 4E.

[0016] In some embodiments, the AR system identifies one or more points of interest present for that geographic location that are difficult for the driver to understand due to color blindness. For example, the AR system identifies one or more illuminated traffic signals or brake lights.

[0017] In some embodiments, the AR system identifies one or more points of interest using one or more of the following types of digital data: one or more on-board vehicle sensors that detect the presence of the point of interest and its range from the vehicle including the AR system and generate environmental data describing the presence of the point of interest and its range from the vehicle; geographic positioning system data ("GPS data") describing a geographic location of the vehicle including the AR system; GPS data describing a geographic location of a point of interest (for example, which may be transmitted to the AR system via a wireless notification, a dedicated short range communication ("DSRC wireless notification", DRSC), or some other wireless notification protocol);and a light data structure comprising digital data describing the geographical location of static points of interest, such as traffic signals and road signs, that can be queried or cross-referenced using the GPS data for the vehicle to identify whether the vehicle is near a point of interest described by the digital data stored in the light data structure;

[0018] In some embodiments, the AR system generates graphical data to cause the AR headset to display one or more geographic overlays for the one or more points of interest. The one or more graphic overlays are configured by the AR system to specifically alleviate the driver's color blindness.

[0019] The AR system provides the graphical data to the AR glasses (or, in some embodiments, a HUD or a 3D HUD) to cause them to display these one or more graphic overlays. See, for example, Fig. 4A to 4E. Our research indicates that the AR system dramatically improves the driving experience and safety for drivers with color blindness, and that the AR system works for all degrees of color blindness due to the use of words that precisely describe what vehicle actions the driver should take to respond to an otherwise unseen illuminated color in their driving environment.

[0020] A system of one or more computers may be configured to perform certain operations or actions by means of software, firmware, hardware, or a combination of these installed on the system, which, in operation, causes the system to perform the actions. One or more computer programs may be configured to perform certain operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

[0021] A general aspect includes a method for a vehicle having an AR headset according to claim 1, the method comprising: identifying an illuminated light in a driving environment of the vehicle; Determining a location of the vehicle using a GPS unit compliant with dedicated short range communication, DSRC, wherein the location is accurate to within plus or minus 1.5 meters; Identifying an illuminated light in a driving environment of the vehicle based on the location of the vehicle; determining that the illuminated light is a point of interest based on a color of the illuminated light and that a driver of the vehicle is impaired by a type of color blindness; Determining a vehicle action to be taken in response to the illuminated light being identified in the driving environment of the vehicle and the illuminated light being the point of interest; Receiving head position data from one or more internal cameras of the vehicle, wherein the head position data describes an orientation of the driver's head; Determining, based on the head position data, that the orientation of the driver’s head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, displaying an AR overlay using the AR headset that visually depicts a word describing the vehicle action to be taken in response to identifying the illuminated light.

[0022] Further aspects and implementations are set forth in the dependent claims. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0023] A general aspect includes a system of a vehicle according to claim 9, comprising: an augmented reality (“AR”) headset; and an on-board vehicle computer system that is communicatively coupled with the AR headset, wherein the on-board vehicle computer system comprises a non-volatile memory storing computer code that, when executed by the on-board vehicle computer system, causes the on-board vehicle computer system to: to determine a location of the vehicle using a GPS unit compliant with dedicated short-range communication (DSRC), the location being accurate to within plus or minus 1.5 meters; identify an illuminated light in a driving environment of the vehicle based on the location of the vehicle; determine that the illuminated light is a point of interest based on a color of the illuminated light and that a driver of the vehicle is impaired by a type of color blindness; determine a vehicle action to be taken in response to the illuminated light being identified in the vehicle's driving environment and the illuminated light being the point of interest; receive head position data from one or more internal cameras of the vehicle, the head position data describing an orientation of the driver's head; determine, based on the head position data, that the orientation of the driver's head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, cause the AR headset to display an AR overlay that visually represents a word describing the vehicle action to be taken in response to identifying the illuminated light.

[0024] Further embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0025] Further aspects and implementations are set forth in the dependent claims. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

[0026] A general aspect includes a computer program product according to claim 16, comprising a non-volatile memory of an on-board vehicle computer system of a vehicle storing computer-executable code that, when executed by the on-board vehicle computer system, causes the on-board vehicle computer system to: to determine a location of the vehicle using a GPS unit compliant with dedicated short-range communication (DSRC), the location being accurate to within plus or minus 1.5 meters; identify an illuminated light in a driving environment of the vehicle based on the location of the vehicle; to determine that the illuminated light is a point of interest based on a color of the illuminated light and that a driver of the vehicle is impaired by a type of color blindness determine a vehicle action to be taken in response to the illuminated light being identified in the vehicle's driving environment and the illuminated light being the point of interest; receive head position data from one or more internal cameras of the vehicle, the head position data describing an orientation of the driver's head; determine, based on the head position data, that the orientation of the driver's head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, display an augmented reality ("AR") overlay on an AR headset of the vehicle that visually depicts a word describing the vehicle action to be taken in response to identifying the illuminated light.

[0027] Further embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0028] Implementations may include one or more of the following features: The computer program product wherein the illuminated light is identified based on measurements of the driving environment recorded by one or more on-board sensors of the vehicle. The computer program product wherein the illuminated light is identified based on digital data contained in a dedicated short-range communication message received by the vehicle, the digital data describing the illuminated light. The computer program product wherein the AR headset is a three-dimensional heads-up display unit operable to display the AR overlay so that it visually appears as if it had three dimensions in the real world. The computer program product wherein the AR overlay consists of text for a word describing the vehicle action or driving action.Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The disclosure is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings, in which like reference numerals are used to designate similar elements. Fig. 1 is a block diagram illustrating an operating environment for a vehicle AR system according to some embodiments. Fig. 2 is a block diagram illustrating an example of a computer system including the AR system of a vehicle according to some embodiments. Fig. 3 includes a flowchart of an example of a method for providing AR-based assistance to a color-blind driver according to some embodiments. Fig. 4A to 4E are examples of a graphic overlay provided by the AR system according to some embodiments. Fig. 5 is a block diagram illustrating a 3D HUD according to some embodiments. DETAILED DESCRIPTION

[0030] AR and virtual reality ("VR") are not the same thing. In VR, a user wears a VR headset that prevents the user from seeing the outside world and a pair of headphones that provide audio information corresponding to the images displayed through the VR headset, while also canceling out the sound of the real world. In other words, the purpose of VR is to immerse the user in a VR world, completely forgetting the real world.

[0031] Accordingly, VR is unsuitable for use in vehicles because it distracts the driver from the road surface present in the real world, and therefore poses a safety risk. The AR system and the AR manager do not provide a VR experience to a driver, as this would be a life-threatening safety risk.

[0032] In AR, a user wears an AR headset that includes a transparent screen (or plastic or other suitable transparent material) configured to allow the user to see the real world when looking through the AR headset. The AR headset displays virtual objects that include graphic images overlaid on the real world. The virtual objects can appear visually transparent, translucent, opaque, or solid or permanent. The virtual objects enhance or modify the way the real world looks when viewed through the AR headset. The user can also wear an AR glove that enhances or modifies the way the real world feels. In other words, the purpose of AR is to add experiences to the real world without causing the user to completely forget the real world, since the real world is part of the AR experience.

[0033] Embodiments of a system of a vehicle are described herein that assist a color-blind driver in understanding and correctly interpreting his driving environment through the use of AR. Example overview

[0034] Referring to Fig. 1 illustrates an operating environment 100 for an AR system 199 of a vehicle 123 according to some embodiments. The operating environment 100 may include one or more of the vehicle 123 and a server 107. These elements may be communicatively coupled to one another via a network 105. Although a vehicle 123, a server 107, and a network 105 are Fig. 1, the operating environment 100 may in practice include one or more vehicles 123, one or more servers 107, and one or more networks 105.

[0035] In some embodiments, the vehicle 123 is referred to as an "ego vehicle" or "reference vehicle" because the vehicle 123 includes the AR system 199. In some embodiments, for example, the AR system 199 assists a driver of the vehicle 123 in seeing and correctly interpreting the brake lights of a second vehicle traveling in front of the vehicle 123 (see, for example, Fig. 4E), and in these embodiments, the vehicle 123 may be referred to as an ego vehicle or reference vehicle to distinguish it from the second vehicle traveling in front of the ego vehicle or reference vehicle.

[0036] The AR System 199 is in Fig. 1 with a dashed line as an element of the server 107 and the vehicle 123. This is because, in some embodiments, the functionality of the AR system 199 is distributed across a plurality of endpoints, such as the server 107 and the vehicle 123. In some embodiments, for example, the light data structure 184 is an element of the server 107, and the functionality of the AR system 199 with respect to the light data structure 184 is provided by the server 107, since the server 107 may, for example, have more computing power than the vehicle 123.

[0037] In some embodiments, the server 107 is an optional feature of the operating environment 100. For example, in some embodiments, the AR system 199 of the vehicle 123 is capable of providing all of the functionality of the AR system 199 without using the server 107. For example, in some embodiments, the light data structure 184 is an element of the vehicle 123, and the AR system 199 of the vehicle 123 provides the AR system functionality related to the light data structure 184.

[0038] Network 105 may be of a conventional type, wired or wireless, and may have numerous different configurations, including a star configuration, a token ring configuration, or other configurations. Furthermore, network 105 may include a local area network (LAN), a wide area network (WAN) (e.g., the Internet), or other interconnected data paths over which multiple devices and / or entities may communicate. In some embodiments, network 105 may include a peer-to-peer network. Network 105 may also be coupled to or include portions of a telecommunications network for sending data in a variety of different communication protocols.In some embodiments, network 105 comprises a Bluetooth® communication network or a cellular communication network for sending and receiving data, including a short message service (SMS), a multimedia message service (MMS), hypertext transfer protocol (HTTP), a direct data connection, a wireless application protocol (WAP), email, DSRC, full-duplex wireless communication, etc. Network 105 may also comprise a mobile data network, which may include 3G, 4G, LTE, LTE-V2X, VoLTE, or any other mobile data network or communication of mobile data networks. Furthermore, network 105 may comprise one or more IEEE 802.11 wireless networks.

[0039] In some embodiments, vehicle 123 is a DSRC-equipped vehicle. In some embodiments, a DSRC-equipped vehicle is a vehicle that includes a DSRC radio 144 and a DSRC-compliant GPS unit 150. The DSRC radio 144 is electronic hardware devices that include a DSRC transmitter and a DSRC receiver licensed to lawfully transmit and receive DSRC messages on the 5.9 GHz band in the jurisdiction where the DSRC-equipped vehicle is located.

[0040] A DSRC message is a wireless message specifically configured to be sent and received by highly mobile devices such as vehicles, and conforms to one or more of the following DSRC standards, including any derivatives and forks thereof: EN 12253:2004 Dedicated Short Range Communications - Physical Layer using Microwaves at 5.8 GHz (Review); EN 12795:2002 Dedicated Short Range Communications (DSRC) - DSRC Data Link Layer: Media Access and Logical Link Control (Review); EN 12834:2002 Dedicated Short Range Communications - Application Layer (Review); and EN 13372:2004 Dedicated Short Range Communications (DSRC) - DSRC Profiles for RTTT Applications (Review); EN ISO 14906:2004 Electronic Billing - Application Interface.

[0041] In the United States, Europe, and Asia, DSRC messages are transmitted at 5.9 GHz. In the United States, 75 MHz of spectrum in the 5.9 GHz band is allocated to DSRC messages. In Europe and Asia, 30 MHz of spectrum in the 5.9 GHz band is allocated to DSRC messages. Therefore, a wireless message is not a DSRC message unless it operates in the 5.9 GHz band. Likewise, a wireless message is not a DSRC message unless it is transmitted by a DSRC transmitter of a DSRC radio.

[0042] Accordingly, a DSRC message is not any of the following: a Wi-Fi message; a 3G message; a 4G message; an LTE message; a millimeter wave communication message; a Bluetooth message; a satellite communication; and a short-range radio message transmitted or broadcast by a key fob at 315 MHz or 433.92 MHz. For example, in the United States, key fobs for remote keyless systems include a short-range radio transmitter operating at 315 MHz, and transmissions or broadcasts from these short-range radio transmitters are not DSRC messages because, for example, such transmissions or broadcasts do not conform to any DSRC standard, are not transmitted by a DSRC transmitter of a DSRC radio, and are not transmitted at 5.9 GHz.As another example, in Europe and Asia, key fobs for remote keyless systems include a short-range radio transmitter operating at 433.92 MHz, and transmissions or broadcasts from this short-range radio transmitter are not DSRC messages for the same reasons described above for the remote keyless system in the United States.

[0043] In some embodiments, a DSRC-equipped vehicle does not include a conventional global positioning system (“GPS”) unit, and instead includes a DSRC-compliant GPS unit 150. A conventional GPS unit provides position information describing a position of the conventional GPS unit with an accuracy of plus or minus 10 meters from the actual position of the conventional GPS unit. For comparison, a DSRC-compliant GPS unit 150 provides GPS data 185 describing a position of the DSRC-compliant GPS unit 150 with an accuracy of plus or minus 1.5 meters from the actual position of the DSRC-compliant GPS unit 150. This level of accuracy is referred to as “lane-level accuracy” because a lane of a road orRoadway is generally approximately 3 metres wide and an accuracy of plus or minus 1.5 metres is sufficient to identify which lane vehicle 123 is traveling in, even if vehicle 123 is traveling on a roadway with multiple lanes where traffic flows in the same direction.

[0044] In some embodiments, the DSRC-compliant GPS unit 150 is operable to identify, monitor, and track its two-dimensional position within 1.5 meters of its actual position 68% of the time in the open air. Since lanes of a roadway are typically no less than 3 meters wide, whenever the two-dimensional error of the GPS data 185 is less than 1.5 meters, the AR system 199 described herein can analyze the GPS data 185 provided by the DSRC-compliant GPS unit 150 and determine which lane of the roadway the vehicle 123 is traveling in based on the relative positions of a plurality of vehicles on the roadway in which the vehicle 123 is in the majority.

[0045] An example of a DSRC message is a basic safety message (“BSM message”, BSM). The DSRC radio 144 of vehicle 123 transmits BSM messages at regular intervals (for example, once every 0.1 seconds or at some other interval configurable by the user). Each BSM message includes BSM data describing the vehicle that broadcast the BSM message. The BSM data is digital data that describes, among other things, a unique identifier of the vehicle that broadcast the BSM message and the GPS data 185 for that vehicle. In this way, individual vehicles of a plurality of DSRC-equipped vehicles traveling on the roadway can identify themselves to other DSRC-equipped vehicles and inform them of their geographical location with lane-level accuracy.The AR system 199 described herein may use this information, for example, to identify the presence of brake lights and whether their range relative to the vehicle 123 makes them a point of interest to the driver of the vehicle 123.

[0046] In some embodiments, a roadside device, such as traffic signals, may include a DSRC radio 144, and these traffic signals may transmit DSRC messages that include digital data describing their geographical location and, optionally, information indicating which light of the traffic signal is currently illuminated (whether it is the red light, yellow light, or green light). The AR system 199 described herein receives this DSR message, extracts the digital data, and causes the AR headset 198 to display a graphic overlay corresponding to the illuminated light of the traffic signal described by the digital data extracted from the DSCR message.

[0047] Network 105 may include one or more communication channels shared by vehicle 123 and other vehicles on the roadway. The communication channel may include DSRC, LTE-V2X, full-duplex wireless communication, or any other wireless communication protocol. For example, network 105 may be used to transmit a DSRC message, a DSRC probe, a BSM, or a full-duplex message, including any of the data described herein.

[0048] Vehicle 123 is any type of vehicle. For example, vehicle 123 is one of the following types of vehicles: a car; a truck; a sports utility vehicle or SUV; a bus; a semi-trailer truck; a drone; or any other road-based conveyance.

[0049] In some embodiments, the vehicle 123 is an autonomous vehicle or a semi-autonomous vehicle.

[0050] In some embodiments, the vehicle 123 includes one or more of the following elements: a processor 125A; a memory 127A; a communication unit 145A including the DSRC radio 144; an AR headset 198; one or more AR gloves 196; and an AR system 199. These elements of the vehicle 123 are communicatively coupled to one another via a bus 120A.

[0051] Server 107 is a processor-based computing device. For example, server 107 may comprise one or more of the following types of processor-based computing devices: a personal computer; a laptop; a mainframe; or any other processor-based computing device capable of operating as a server. Server 107 may comprise a hardware server.

[0052] In some embodiments, server 107 includes one or more of the following elements: a processor 125B; a memory 127B; a communications unit 145B; and an AR system 199. These elements of server 107 are communicatively coupled to one another via a bus 120B.

[0053] The processor 125A of the vehicle 123 and the processor 125B of the server 107 may be referred to collectively or individually herein as the "processor 125," since, for example, the processor 125A of the vehicle 123 provides similar functionality to the components of the vehicle 123 as the processor 125B of the server 107. For similar reasons, the description provided herein uses the following terms when referring to elements common to the vehicle 123 and the server 107: the "memory 127" when referring to the memory 127A and the memory 127B collectively or individually; and the "communications unit 145" when referring to the communication unit 145A and the communication unit 145B collectively or individually.

[0054] The vehicle 123 and the server 107 will now be described. Vehicle 123

[0055] In some embodiments, the processor 125 and the memory 127 are elements of an on-board vehicle computer system (such as a computer system 200 described below with reference to Fig. 2). The on-board vehicle computer system is operable to cause or control the operation of the AR system 199. The on-board vehicle computer system is operable to access and execute the data stored on the memory 127 to provide the functionality described herein for the AR system 199 or its elements (see, for example, Fig. 2). The on-board vehicle computer system is operable to execute the AR system 199, which causes the on-board vehicle computer system to perform one or more of the steps of the method 300 described below with reference to Fig. 3 described.

[0056] Processor 125 includes an arithmetic logic unit, a microprocessor, a general-purpose controller, or other processing array to perform calculations and provide electronic display signals to a display device. Processor 125 processes data signals and may include various computing architectures, including a Complex Instruction Set Computer (CISC) architecture, a Reduced Instruction Set Computer (RISC) architecture, or an architecture implementing a combination of instruction sets. Vehicle 123 may include one or more processors 125. Other processors, operating systems, sensors, displays, and physical configurations may be possible.

[0057] Memory 127 stores instructions or data that can be accessed or executed by processor 125. The instructions or data may include code for performing the techniques described herein. Memory 127 may be a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory, or some other storage device. In some embodiments, memory 127 also includes non-volatile memory or a similar permanent storage device and media, including a hard disk drive, a floppy disk drive, a CD-ROM device, a DVD-ROM device, a DVD-RAM device, a DVD-RW device, a flash memory device, or some other mass storage device for storing information on a permanent basis.A portion of memory 127 may be reserved for use as a buffer or as virtual random access memory (virtual RAM). Vehicle 123 may include one or more memories 127.

[0058] The memory 127 of the vehicle 123 may store one or more of the following types of digital data: GPS data 185; environmental data 186; graphic data 187; and head position data 188. In some embodiments, the memory 127 stores the light data structure 184.

[0059] GPS data 185 is digital data describing the geographical location of vehicle 123. In some embodiments, GPS data 185 describes the geographical location of vehicle 123 with lane-level accuracy. GPS data 185 is used by AR system 199, for example, to identify whether vehicle 123 is located at a geographical location that includes a traffic signal or a brake light.

[0060] In some embodiments, GPS data 185 includes digital data describing the geographic location of other DSRC-equipped objects within the driving environment of vehicle 123, such as other DSRC-equipped vehicles, DSRC-equipped traffic signals, or DSRC-equipped road signs. In these embodiments, AR system 199 includes code and routines operable when executed by processor 125 to cause processor 125 to determine whether vehicle 123 is located at a geographic location that includes a traffic signal or a brake light by comparing the geographic location of vehicle 123 with the geographic location of the other DSRC-equipped objects.

[0061] The light data structure 184 is a table or other data structure that includes digital data describing the latitude and longitude for different traffic lights or traffic signs within a geographic area. The AR system 199 includes code and routines operable, when executed by the processor 125, to cause the processor 125 to compare the GPS data 185 with the digital data stored in the light data structure 184 to identify whether the current GPS location for the vehicle 123 corresponds to the location for a traffic light or traffic sign.

[0062] The environmental data 186 is digital data describing the driving environment outside the vehicle 123. In some embodiments, the environmental data 186 describes the measurements from one or more external sensors of the sensor set 170. The AR system 199 includes code and routines operable, when executed by the processor 125, to cause the processor 125 to analyze the environmental data 186 to identify whether the driving environment for the vehicle 123 includes a traffic light or a brake light for another vehicle traveling in front of the vehicle 123.

[0063] In some embodiments, the environmental data 186 describes one or more points of interest. A point of interest is an object within the driving environment of the vehicle 123 that is illuminated and has a color affected by the particular type and degree of color blindness of the driver of the vehicle 123. In some embodiments, the AR system 199 includes code and routines operable, when executed by the processor 125, to cause the processor 125 to analyze the environmental data 186 (as well as other inputs, such as the GPS data 185 and the light data structure 184) to identify one or more points of interest in the current driving environment for the vehicle 123. The AR system 199 may repeat this analysis at regular intervals over time so that the identified points of interest are regularly updated.

[0064] The graphical data 187 is digital data operable to cause the AR headset 198 to display one or more AR overlays or AR overlays based on the one or more points of interest described by the environmental data 186. For example, the graphical data 187 is operable to cause the AR headset 198 to display one or more of the AR overlays described in Fig. 4A to 4E are shown.

[0065] Head position data 188 is digital data describing a head orientation of a driver of vehicle 123. In some embodiments, head position data 188 describes measurements from one or more internal sensors included in sensor set 170. For example, sensor set 170 includes one or more internal cameras that track the head orientation of the driver of vehicle 123, and head position data 188 describes the measurements from these sensors. In some embodiments, AR headset 198 includes one or more accelerometers that track the driver's head orientation.

[0066] The communication unit 145 transmits and receives data to and from a network 105 or another communication channel. In some embodiments, the communication unit 145 may include a DSRC transceiver, a DSRC receiver, and other hardware or software necessary to make the vehicle 123 (or some other device, such as the server 107) a DSRC-capable device.

[0067] In some embodiments, the communication unit 145 includes a port for a direct physical connection to the network 105 or to another communication channel. For example, the communication unit 145 includes a USB, SD, CAT-5, or similar port for wired communication with the network 105. In some embodiments, the communication unit 145 includes a wireless transceiver for exchanging data with the network 105 or another communication channel and using one or more wireless communication methods, including: IEEE 802.11; IEEE 802.16, BLUETOOTH®; EN ISO 14906:2004 Electronic fee collection - Application interface; EN 11253:2004 Dedicated short-range communication - Physical layer using microwaves at 5.8 GHz (Review); EN 12795:2002 Dedicated short-range communication (DSRC) - DSRC data link layer: Media access and logical link control (Review); EN 12834:2002 Dedicated short-range communication - Application layer (Review); EN 13372:2004 Dedicated short-range communication (DSRC) - DSRC profiles for RTTT applications (Review); the communication method described in US patent application US 2016 / 0 065 355 A1, filed August 28, 2014, entitled "Full-Duplex Coordination System"; or any other suitable wireless communication method.

[0068] In some embodiments, the communication unit 145 includes a full-duplex coordination system described in U.S. patent application US 2016 / 0 065 355 A1, filed August 28, 2014, entitled “Full-Duplex Coordination System,” the entirety of which is incorporated herein by reference.

[0069] In some embodiments, communication unit 145 includes a cellular communication transceiver for sending and receiving data over a cellular communication network, including a Short Message Service (SMS), a Multimedia Messaging Service (MMS), Hypertext Transfer Protocol (HTTP), a direct data connection, WAP, email, or other suitable type of electronic communication. In some embodiments, communication unit 145 includes a wireless port and a wireless transceiver. Communication unit 145 also provides other conventional connections to network 105 for distributing files or media objects using standard network protocols, including TCP / IP, HTTP, HTTPS, and SMTP, millimeter waves, DSRC, etc.

[0070] In some embodiments, the communication unit 145 includes a DSRC radio 144. The DSRC radio 144 is a hardware unit that includes a DSRC transmitter and a DSRC receiver. The DSRC transmitter is operable to transmit and broadcast DSRC messages over the 5.9 GHz band. The DSRC receiver is operable to receive DSRC messages over the 5.9 GHz band.

[0071] In some embodiments, the DSRC radio 144 includes non-volatile memory that stores digital data that controls the frequency for broadcasting BSM messages. In some embodiments, the non-volatile memory stores a buffered version of the GPS data 185 for the vehicle 123 so that the GPS data 185 for the vehicle 123 is broadcast as an element of the BSM message periodically broadcast by the DSRC radio 144.

[0072] In some embodiments, the DSRC radio 144 includes any hardware or software necessary for the vehicle 123 to comply with the DSRC standard. In some embodiments, the DSRC-compliant GPS unit 150 is an element of the DSRC radio 144.

[0073] In some embodiments, the DSRC-compliant GPS unit 150 includes any hardware and software necessary for the vehicle 123 or the DSRC-compliant GPS unit 150 to comply with one or more of the following DSRC standards, including any derivatives or forks thereof: EN 12253:2004 Dedicated Short Range Communication - Physical Layer using Microwaves at 5.8 GHz (Review); EN 12795:2002 Dedicated Short Range Communication (DSRC) - DSRC Data Link Layer: Media Access and Logical Link Control (Review); EN 12834:2002 Dedicated Short Range Communication - Application Layer (Review); and EN 13372:2004 Dedicated Short Range Communication (DSRC) - DSRC Profiles for RTTT Applications (Review); EN ISO 14906:2004 Electronic fee collection - Application interface.

[0074] In some embodiments, the DSRC-compliant GPS unit 150 is operable to provide GPS data 185 describing the location of the vehicle 123 with lane-level accuracy.

[0075] The AR headset 198 is any conventional AR headset, glasses, or lenses. Examples of the AR headset 198 may include one or more of the following: Google™ Glass; CastAR; Moverio BT-200; Meta; Vuzix M-100; Laster SeeThru; Icis; Optinvent ORA-S; GlassUP; Atheer One; K-Glass; and Microsoft™ Hololens. The AR headset 198 is configured such that a driver of the vehicle 123 can be focused on the driving experience when operating the vehicle 123.

[0076] In some embodiments, the AR headset 198 includes one or more sensors (e.g., accelerometers) that track the orientation of the driver's head and output head position data 188 describing the orientation of the driver's head. The head position data 188 is used by the AR system 199, for example, to determine when to cause the AR headset 198 to display the AR overlay describing a point of interest that is in the driver's field of view (or optionally obscured from the driver's field of view). For example, the AR system 199 includes code and routines operable, when executed by the processor 125, to cause the AR headset 198 to display the AR overlay when the driver's head orientation indicates that they are looking at the roadway in the direction of the point of interest described by the AR overlay.For example, the AR system 199 provides the graphical data 187 to the AR headset 198.

[0077] In some embodiments, the AR headset 198 is a 3D HUD. An example of the 3D HUD is shown in Fig. 5. For example, a driver of the vehicle 123 may view the 3D HUD and the 3D HUD may display one or more virtual overlays, such as those shown in Fig. 4A to 4E are shown.

[0078] In some embodiments, the one or more virtual overlays are displayed by the AR headset 198 as three-dimensional virtual overlays that appear to the driver as having three dimensions (x, y, and z axes), where the horizontal line of the windshield of the vehicle 123 is the x-axis, the vertical line of the windshield is the y-axis, and the z-axis passes through the windshield toward the driver's head and the exterior of the vehicle 123.

[0079] The vehicle 123 may include one or more AR gloves 196. The AR glove 196 is a haptic feedback glove configured to allow the driver of the vehicle 123 to touch, grasp, and feel virtual overlays as if they existed in the real world (see, for example, the virtual overlays shown in Fig. 4A to 4E).

[0080] In some embodiments, the AR glove 196 may provide haptic feedback to the driver to inform the driver of the presence of a virtual overlay redisplayed by the AR system 199.

[0081] In some embodiments, one or more sensors of the sensor set 170 may be operable to record data describing a location of the AR glove 196 relative to some other real objects or virtual objects.

[0082] In some embodiments, the AR glove 196 includes force feedback units (hereinafter "motors") that apply torque to the rider's fingers located within the AR glove 196. These motors, when controlled by the AR system 199, are operable to dynamically change the direction and magnitude of the force caused by the rider's hand / finger movement to simulate a specific stiffness of a virtual overlay. In this way, the motors of the AR glove 196 provide slight resistance when the rider manipulates a virtual object corresponding to a soft object, such as a leather knob or handle, and provide strong resistance when the virtual object represents a denser object, such as one that would be made of metal in the real world.

[0083] In some embodiments, the AR glove 196 includes other smaller motors that provide haptic vibrations to the driver's fingertips within the AR glove 196. These haptic vibrations simulate the effect of a finger striking a touch interface or tracing a textured surface.

[0084] In some embodiments, the motors provided in the AR glove 196 are sufficiently powerful to be operable to provide physical feedback capable of physically preventing the driver's fingers from penetrating virtual objects displayed on the AR headset 198.

[0085] As described in more detail below, the AR glove 196 may include a non-volatile cache or buffer that temporarily stores data it receives from the AR system 199.

[0086] The sensor set 170 includes one or more sensors operable to measure the physical environment outside the vehicle 123 (i.e., "external sensors"). For example, the sensor set 170 includes one or more sensors that record one or more physical characteristics of the physical environment located near the vehicle 123 (i.e., the "driving environment"). The memory 127 can store environmental data 186 describing the one or more physical characteristics of the driving environment recorded by the sensor set 170.

[0087] In some embodiments, the environmental data 186 may be used by the AR system 199 to confirm or discard the GPS data 185 or other data stored in the memory 127. For example, the GPS data 185 may indicate that the vehicle 123 is near a particular landmark, and the environmental data 186 may include an image including the particular landmark, thereby confirming the accuracy of the GPS data 185.

[0088] In some embodiments, the sensor set 170 may include one or more of the following vehicle sensors: a camera; a LIDAR sensor; a radar sensor; a laser altimeter; an infrared detector; a motion detector; a thermostat; a sound detector; a carbon monoxide sensor; a carbon dioxide sensor; an oxygen sensor; a mass air flow sensor; an engine coolant temperature sensor; a throttle position sensor; a crankshaft position sensor; an automotive engine sensor; a valve timing sensor; an air-fuel ratio sensor; a blind spot sensor; a curb feeler; a defect detector; a Hall effect sensor; a manifold absolute pressure sensor; a parking sensor; a radar gun; a speedometer; a velocity sensor; a tire pressure monitoring sensor; a torque sensor; a transmission fluid temperature sensor; a turbine speed sensor (TSS); a variable reluctance sensor.Inductive sensor; a vehicle speed sensor (VSS); a water sensor; a wheel speed sensor; and any other type of automotive sensor.

[0089] In some embodiments, sensor set 170 includes one or more sensors operable to measure the physical environment within vehicle 123 (i.e., "internal sensors"). For example, sensor set 170 includes one or more sensors that record one or more physical characteristics of the physical environment located within the vehicle cabin of vehicle 123. Memory 127 may store head position data 188 describing the one or more physical characteristics recorded by sensor set 170 from within the cabin of vehicle 123.

[0090] In some embodiments, the AR system 199 includes code and routines operable when executed by the processor 125 to cause the processor 125 to perform one or more of the steps of the method 300 described below with reference to Fig. 3 is described.

[0091] In some embodiments, the AR system 199 of the vehicle 123 may be implemented using hardware, including a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In other embodiments, the AR system 199 may be implemented using a combination of hardware and software. The AR system 199 may be stored in a combination of the devices (e.g., servers or other devices) or in one of the devices.

[0092] The AR system 199 is described in more detail below with reference to Fig. 2 to 5. Server 107

[0093] In some embodiments, the server 107 is a cloud server that includes one or more of the following elements: the AR system 199; the communication unit 145; the processor 125; and the memory 127.

[0094] The following elements of the server 107 are the same or similar to those described above for the vehicle 123, so descriptions of these elements are not repeated here: the AR system 199, the processor 125; the memory 127; and the communication unit 145.

[0095] In some embodiments, the communication unit 145 of the vehicle 123 transmits one or more wireless messages (i.e., one or more "request messages") to the server 107 at regular intervals over the network 105, including digital data such as the GPS data 185 and the environmental data 186, and the AR system 199 of the server 107 determines the graphical data 187 for the vehicle 123 based on these inputs and, optionally, other locally accessible data such as the light data structure 184. The communication unit 145 of the server 107 then transmits the graphical data 187 to the communication unit 145 of the vehicle 123 over the network 105 in a manner similar to how the one or more request messages were initially received by the vehicle 123.

[0096] Referring now to Fig. 2 is a block diagram illustrating an example of a computer system 200 including an AR system 199 according to some embodiments.

[0097] In some embodiments, the computer system 200 may comprise a special-purpose computer system programmed to perform one or more steps of a method 300 described below with reference to Fig. 3 is described.

[0098] In some embodiments, computer system 200 may be an element of server 107.

[0099] In some embodiments, the computer system 200 may be an on-board vehicle computer of the vehicle 123.

[0100] In some embodiments, the computer system 200 may include an electronic control unit, a head unit, or any other processor-based computing device of the vehicle 123.

[0101] The computer system 200 may include one or more of the following elements, according to some examples: the AR system 199; the processor 125; the communication unit 145; the sensor set 170; the memory 127; the AR glove 196; and the AR headset 198. The components of the computer system 200 are communicatively coupled via a bus 120.

[0102] In the illustrated embodiment, processor 125 is communicatively coupled to bus 120 via a signal line 238. Communication unit 145 is communicatively coupled to bus 120 via a signal line 246. Sensor set 170 is communicatively coupled to bus 120 via a signal line 248. AR glove 196 is communicatively coupled to bus 120 via a signal line 241. AR headset 198 is communicatively coupled to bus 120 via a signal line 242. Memory 127 is communicatively coupled to bus 120 via a signal line 244.

[0103] The following elements of the computer system 200 were described above with reference to Fig. 1, so their descriptions are not repeated here: the processor 125; the communication unit 145; the sensor set 170; the AR glove 146; the AR headset 198; and the memory 127.

[0104] The memory 127 may store any of the information related to Fig. 1. Memory 127 may store any data necessary for computer system 200 to provide its functionality.

[0105] In some embodiments, the memory 127 stores context data 299. The context data 299 includes digital data describing information used by the AR system 199 to analyze the environment data 186 and identify how to identify and describe a point of interest in the real world based on the environment data 186. For example, the context data 299 is digital data describing how to identify an illuminated light among other lights that are not illuminated, and then determine the color of the illuminated light, and what action the driver of the vehicle 123 should take based on identifying the presence of an illuminated light of that particular color.

[0106] In one example of the context data 299, if the environmental data 186 includes digital data indicating that the illuminated light is green, then the context data 299 includes digital data describing any information necessary for the AR system 199 to determine that the driver should "GO," and so the AR system 199 generates graphical data 187 operable to cause the AR headset 198 to display an AR overlay with the word "GO" depicted in the AR overlay at a visual location of the AR headset 198 that is correlated to the real-world location of the point of interest (e.g., the green light) as viewed by the driver of the vehicle 123.

[0107] In another example of the context data 299, if the environmental data 186 includes digital data indicating that the illuminated light is yellow, then the context data 299 includes digital data describing any information necessary for the AR system 199 to determine that the driver should “SLOW DOWN,” and so the AR system 199 generates graphical data 187 operable to cause the AR headset 198 to display an AR overlay with the word “SLOW DOWN” depicted in the AR overlay at a visual location of the AR headset 198 that is correlated to the real-world location of the point of interest (e.g., the yellow light) as viewed by the driver of the vehicle 123.

[0108] In another example of the context data 299, if the environmental data 186 includes digital data indicating that the illuminated light is red and an element of a traffic signal or road sign, then the context data 299 includes digital data describing any information necessary for the AR system 199 to determine that the driver should “STOP,” and so the AR system 199 generates graphical data 187 operable to cause the AR headset 198 to display an AR overlay with the word “STOP” depicted in the AR overlay at a visual location of the AR headset 198 that is correlated to the real-world location of the point of interest (e.g., the red light of the traffic signal or road sign) as viewed by the driver of the vehicle 123.

[0109] In another example of the context data 299, if the environmental data 186 includes digital data indicating that the illuminated light is red and an element of a vehicle (i.e., a brake light), then the context data 299 includes digital data that enables the AR system 199 to know that the driver should “BRAKE,” and so the AR system 199 generates graphical data 187 operable to cause the AR headset 198 to display an AR overlay with the word “BRAKE” depicted in the AR overlay at a visual location of the AR headset 198 that is correlated to the real-world location of the point of interest (e.g., the vehicle’s red light) as viewed by the driver of the vehicle 123.It should be noted that the context data 299 distinguishes actions of a brake light and a stop light with the words “BRAKE” and “STOP” because when braking is performed due to brake lights, the driver sometimes slows down but does not stop.

[0110] In the illustrated embodiment, which is shown in Fig. 2, the AR system 199 includes a communications module 202. The communications module 202 may be software, including routines for handling communications between the AR system 199 and other components of the computer system 200. In some embodiments, the communications module 202 may be a set of instructions executable by the processor 125 to provide the functionality described below for handling communications between the AR system 199 and other components of the computer system 200.

[0111] The communication module 202 sends and receives data via the communication unit 145 to and from one or more elements of the operating environment 100. For example, the communication module 202 receives or transmits via the communication unit 145 one or more of the following elements: the light data structure 184; the GPS data 185; the environment data 186; the graphical data 187; the head position data 188; and the context data 299. The communication module 202 may send or receive any of the data or messages described above with reference to Fig. 1 or below with reference to Fig. 3, via the communication unit 145.

[0112] In some embodiments, the communication module 202 receives data from components of the AR system 199 and stores the data in the memory 127 (or a buffer or cache of the AR glove 196). For example, the communication module 202 receives any of the data described above with reference to the memory 127 from the communication unit 145 (via the network 105) and stores that data in the memory 127 (or a buffer or cache of the AR glove 196).

[0113] In some embodiments, the communications module 202 may handle communications between components of the AR system 199.

[0114] In some embodiments, the communication module 202 may be stored in the memory 127 of the computer system 200 and may be accessible and executable by the processor 125. The communication module 202 may be adapted to cooperate and communicate with the processor 125 and other components of the computer system 200 via the signal line 222.

[0115] Referring now to Fig. 3 is a flowchart of an example of a method 300 for providing AR-based assistance to a color-blind driver according to some embodiments.

[0116] One or more of the steps described herein for method 300 may be performed by one or more computer systems 200.

[0117] In step 303, the AR system identifies the location of the vehicle using the vehicle's GPS unit. The location is described by GPS data. In some embodiments, the GPS unit is a DSRC-compliant GPS unit, and the GPS data is DSRC-compliant GPS data with an accuracy of plus or minus 1.5 meters relative to the actual real-world location of the vehicle.

[0118] In step 305, the AR system generates environmental data using the vehicle's external sensors. The environmental data describes the environment outside the vehicle.

[0119] In step 307, the AR system analyzes one or more of the GPS data, the environmental data, and the context data to identify whether the driving environment includes one or more illuminated traffic lights or brake lights, and if so, these are referred to as "one or more points of interest." In some embodiments, the memory stores a light data structure (e.g., a table or other digital data structure) that includes digital data describing the latitude and longitude for different traffic lights in a geographic area. The AR system compares the GPS data with the digital data stored in the light data structure to identify whether the current geographic location for the vehicle corresponds to the location of a traffic light. In some embodiments, the AR system does not rely on the light data structure to identify all types of lights.For example, brake lights of a second vehicle can be identified by the AR system based on the environmental data alone, without reference to the digital data stored in the light data structure.

[0120] In step 308, the AR system determines graphical data for one or more points of interest. The graphical data describes one or more AR overlays configured to alleviate the driver's color blindness specifically with respect to the particular color encompassed by one or more of the points of interest.

[0121] In step 309, the AR system causes the AR system to display one or more graphic overlays that include graphic information in such a manner as to assist the color-blind driver.

[0122] Experiments show that the AR system 199 performs the method 300 in real time or near real time relative to the occurrence of an illuminated light in the driving environment of the vehicle.

[0123] Referring now to Fig. 4A to 4E illustrate examples of graphic overlays provided by the AR system according to some embodiments.

[0124] It will be Fig. 4A. A vertically oriented traffic signal 499 includes three lights: a red light 415; a yellow light 420; and a green light 425. In Fig. 4A, the environmental data 186 indicates that the green light 425 is illuminated. Optionally, a DSRC message transmitted by the vertically oriented traffic signal 499 indicates that the green light 425 is illuminated. The context data 299 describes what action the driver of the vehicle 123 should take, i.e., whether to begin driving or continue driving. Accordingly, the AR system 199 generates graphical data 187 that causes the AR headset 198 to display an AR overlay 401 for the green light 425 depicting the word "GO" or some other spelling variation of that word to describe that action (e.g., "Go," "los," etc.).

[0125] It will be Fig. 4B. The environmental data 186 indicates that the yellow light 420 is illuminated. Optionally, a DSRC message transmitted by the vertically oriented traffic signal 499 indicates that the yellow light 420 is illuminated. The context data 299 describes what action the driver of the vehicle 123 should take, that is, decelerate (i.e., slow down), in preparation for the red light being illuminated. Accordingly, the AR system 199 generates graphical data 187 that causes the AR headset 198 to display an AR overlay 402 for the yellow light 420 that displays the word "SLOW DOWN" or some other spelling variation of that word to describe that action.

[0126] It will be Fig. 4C. The environmental data 186 indicates that the red light 415 is illuminated. Optionally, a DSRC message transmitted by the vertically oriented traffic signal 499 indicates that the red light 415 is illuminated. The context data 299 describes what action the driver of the vehicle 123 should take, i.e., stop. Accordingly, the AR system 199 generates graphical data 187 that causes the AR headset 198 to display an AR overlay 403 for the red light 415 depicting the word "STOP" or some other spelling variation of that word to describe that action.

[0127] It will be Fig. 4D. A horizontally oriented traffic signal 498 includes three lights: a red light 415; a yellow light 420; and a green light 425. In Fig. 4D, the environmental data 186 indicates that the red light 415 is illuminated. Optionally, a DSRC message transmitted by the horizontally oriented traffic signal 498 indicates that the red light 415 is illuminated. The context data 299 describes what action the driver of the vehicle 123 should take, i.e., stop. Accordingly, the AR system 199 generates graphical data 187 that causes the AR headset 198 to display an AR overlay 403 for the red light 415, depicting the word "STOP" or some other spelling variation of that word to describe this action.

[0128] It will be Fig. 4E. A second vehicle is traveling in front of an ego vehicle or reference vehicle. The second vehicle can be viewed by the driver of the reference vehicle when looking through a 3D HUD. The second vehicle includes a first brake light 421 and a second brake light 422. In Fig. 4D, the environmental data 186 indicates that the first brake light 421 and the second brake light 422 are illuminated. Optionally, a DSRC message transmitted by the second vehicle indicates that the first brake light 421 and the second brake light 422 are illuminated. The context data 299 describes what action the driver of the vehicle 123 should take, i.e., that they should begin braking the reference vehicle. Accordingly, the AR system 199 generates graphical data 187 that causes the 3D HUD to display a first AR overlay 405 and a second AR overlay 404, respectively, for the first brake light 421 and the second brake light 422, each displaying the word "BRAKING" or some other spelling variation of that word to describe that action.

[0129] Referring to Fig. 5 is a block diagram illustrating an AR headset 198 in embodiments where the AR headset 198 is a 3D HUD.

[0130] In some embodiments, the 3D HUD includes a projector 1001, a movable screen 1002, a screen drive unit 1003, and an optical system (including lenses 1004, 1006, a reflector 1005, etc.). The projector 1001 may be any type of projector, such as a DMD ("Digital Mirror Device") projector or a liquid crystal projector. The projector 1001 projects an AR overlay 1008 onto the movable screen 1002. The AR overlay 1008 may include a virtual object having three dimensions and visually appearing to exist in the real world when viewed by the driver of the vehicle. For example, the driver views the AR overlay 1008 and this appears to the driver as a real-world object rather than an image of a real-world object that has three dimensions and exists in the same physical reality as the driver.

[0131] The movable screen 1002 comprises a transparent plate, and the light of the projected image passes through the movable screen 1002 to be projected onto the windshield 1007 of a vehicle (e.g., the vehicle 123). The image projected onto the windshield 1007 is perceived by a driver 1010 as if it were a real object (shown as 1011a, 1011b) existing in the three-dimensional space of the real world, as opposed to an object projected onto the windshield.

[0132] In some embodiments, the 3D HUD is capable of controlling the direction of the image relative to the driver 1010 (in other words, the image position on the windshield) by adjusting the projection position on the screen 1002. Furthermore, the screen 1002 is movable by the screen drive unit 1003 in the range between positions 1003a and 1003b. Adjusting the position of the screen 1002 can vary the depth (distance) of the projected image from the driver 1010 in the real world. In one example, the movable range of the screen 1002 (the distance between positions 1003a and 1003b) can be 5 mm, which corresponds to a range of 5 m to infinity in the real world. The use of the 3D HUD enables the driver 1010 to perceive the projected image as existing in the real world (three-dimensional space).For example, when an image is projected at the same three-dimensional position (or at least substantially the same depth) as a real object (such as a pedestrian, a vehicle, etc.), the driver does not need to adjust eye focus to view the projected image, allowing easy comprehension of the projected image while viewing the real object.

[0133] The 3D HUD, which is Fig. 5 is provided as an example. Other examples are possible. These examples include head-up displays that have lesser or greater complexity than the 3D HUD shown in Fig. 5. For example, it is expected that in the future there will be head-up displays that do not require moving parts, such as the movable screen 1002. For example, a static screen that does not move may be employed. The head-up display employed may not be a two-dimensional head-up display. In some embodiments, the AR system 199 and the graphical data 187 described above with reference to Fig. 1 to 4E are designed to be operative with such components.

[0134] In the foregoing description, for the purpose of explanation, numerous specific details have been set forth in order to provide a thorough understanding of the specification. However, it will be appreciated by those skilled in the art that the disclosure may be practiced without these specific details. In some instances, structures and devices are shown in block diagram form to avoid obscuring the description. For example, the present embodiments described above may be described primarily with reference to user interfaces and specific hardware. However, the present embodiments may be applied to any type of computer system capable of receiving data and instructions, and any peripheral device that provides services.

[0135] Reference in the specification to "some embodiments" or "some instances" means that a particular feature, structure, or characteristic described in connection with those embodiments or instances may be included in at least one embodiment of the invention. The appearance of the phrase "in some embodiments" in various places in the specification does not necessarily refer to the same embodiments.

[0136] Some sections of the detailed descriptions that follow are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art in the field of data processing to convey the substance of their work to others skilled in the art. An algorithm is considered here and generally to be a coherent sequence of steps that leads to a desired result. The steps are those that require physical manipulation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transmitted, combined, compared, and otherwise manipulated.It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, expressions, numbers, or the like.

[0137] However, it should be borne in mind that all these and similar expressions are to be linked to appropriate physical quantities and are merely convenient labels applied to these quantities.Unless otherwise stated, as will be apparent from the following discussion, it is recognized that throughout the specification, discussions using the terms including "processing" or "computing" or "calculating" or "determining" or "displaying" or the like refer to the action and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system registers and memory into other data similarly represented as physical quantities within the computer system memory or registers, or other such information storage, transmission, or display means.

[0138] The present embodiments of the specification may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purpose or may be a general-purpose computer that is selectively activated and reconfigured by a computer program stored in the computer. Such a computer program may be recorded on a computer-readable storage medium, including, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs and magnetic disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, flash memories including USB sticks with non-volatile memory, or any other type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0139] The specification may take the form of some holistic hardware embodiments, some holistic software embodiments, or some embodiments that include both hardware and software elements. In such preferred embodiments, the specification is implemented in software, which includes, for example, but is not limited to, firmware, resident software, microcode, etc.

[0140] Furthermore, the description may take the form of a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device capable of storing, communicating, propagating, or transporting the program for use by or in connection with the instruction execution system, apparatus, or device.

[0141] A data processing system suitable for storing or executing program code will include at least one processor connected directly or indirectly to storage elements through a system bus. The storage elements may include local memory used during actual execution of the program code, mass storage, and cache memory that provides temporary storage of at least some program code to reduce the number of times the code must be retrieved from the mass storage during execution.

[0142] Input / output devices or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or using I / O controllers.

[0143] Network adapters can also be integrated into the system to allow the computing system to connect to other computing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0144] Finally, the algorithms and displays presented herein are not inherently related to any particular computer or device. Various general-purpose systems may be used with programs according to the teachings described herein, or it may prove advantageous to construct specialized devices for performing the required method steps. The required structure for a variety of these systems will become apparent from the description below or above. Additionally, the specification is not described with reference to any particular programming language. It is recognized that a variety of programming languages ​​may be used to implement the teachings of the specification described herein.

[0145] The foregoing description of embodiments of the specification has been presented for purposes of illustration and description. This is not intended to be exhaustive or to limit the specification to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It is intended that the scope of the disclosure be limited not by the detailed description, but by the claims of this application. As will be understood by those skilled in the art, the specification may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Similarly, the particular naming and partitioning of the modules, routines, features, attributes, methods, and other aspects are not mandatory or significant, and the mechanisms that implement the specification or its features may have different names, partitions, or formats.Furthermore, as will be apparent to one of ordinary skill in the art, the modules, routines, features, attributes, methods, and other aspects of the disclosure may be implemented as software, hardware, firmware, or any combination of these three. Likewise, wherever a component, such as a module of the specification, is implemented as software, the component may be embodied as a stand-alone program, as a part of a larger program, as a plurality of separate programs, as a static or dynamically linked library, as a kernel-loadable module, as a device driver, or in any other known manner now or in the future known to those of ordinary skill in computer programming. In addition, the disclosure is in no way limited to embodiment in any specific programming language, or to any specific operating system or environment.Accordingly, the disclosure is intended to illustrate, and not limit, the scope of the specification as set forth in the following claims.

[0146] The disclosure includes embodiments for providing a graphic overlay depicting lane lines to a vehicle including an augmented reality ("AR") viewer. A method according to some embodiments includes determining whether environmental data indicates that a driver of a vehicle can identify lane lines on a road ahead of the vehicle. The method includes, in response to the environmental data not indicating that the driver of the vehicle can identify the lane lines, generating line data based on the environmental data and global positioning system (GPS) data. The method includes generating AR data based on the line data. The method includes providing the AR data to the AR viewer.The method includes determining, based on head position data associated with the AR viewer, to instruct the AR viewer to display a graphic overlay representing the lane lines.

[0147] The disclosure includes embodiments for providing augmented reality ("AR") vehicle assistance for drivers who are color blind. A method according to some embodiments includes identifying an illuminated light in a driving environment of a vehicle including an AR headset. The method further includes determining a vehicle action to be taken in response to identifying the illuminated light in the driving environment of the vehicle. The method further includes displaying, using the AR headset, an AR overlay that represents a word describing the vehicle action to be taken in response to identifying the illuminated light.

Claims

[1] A method for a vehicle with an augmented reality (“AR”) headset, the method comprising: Determining a location of the vehicle using a GPS unit compliant with dedicated short-range communication, DSRC, wherein the location has an accuracy of plus or minus 1.5 meters; Identifying an illuminated light in a driving environment of the vehicle based on the location of the vehicle; determining that the illuminated light is a point of interest based on a color of the illuminated light and a driver of the vehicle impaired by a type of color blindness; Determining a vehicle action to be taken in response to the illuminated light being identified in the driving environment of the vehicle and the illuminated light being the point of interest; Receiving head position data from one or more internal cameras of the vehicle, wherein the head position data describes an orientation of the driver's head; Determining, based on the head position data, that the orientation of the driver’s head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, displaying an AR overlay using the AR headset that visually depicts a word describing the vehicle action to be taken in response to identifying the illuminated light. [2] The method of claim 1, further comprising providing haptic feedback in an AR glove to inform the driver of the presence of the AR overlay newly displayed on the AR headset. [3] The method of claim 1 or 2, wherein the AR overlay visually represents a word selected from a group comprising: "STOP" for a red light of a traffic signal; "BRAKE" for a red light of a brake light; "SLOW DOWN" for a yellow light of a traffic signal; and "GO" for a green light of a traffic signal. [4] A method according to any one of claims 1 to 3, wherein the illuminated light is further identified by a basic security message, BSM, describing a location of the illuminated light. [5] The method of any one of claims 1 to 4, wherein the AR headset is a three-dimensional head-up display unit operable to display the AR overlay so that it visually appears as if it has three dimensions in the real world. [6] Method according to one of claims 1 to 5, wherein the AR headset is AR glasses. [7] The method of any one of claims 1 to 6, wherein the AR overlay consists of text for a word describing the vehicle action. [8] A method according to any one of claims 1 to 7, wherein the method is carried out in real time relative to the identification of the illuminated light. [9] System of a vehicle, with: an augmented reality (“AR”) headset; and an on-board vehicle computer system that is communicatively coupled with the AR headset, wherein the on-board vehicle computer system comprises a non-volatile memory storing computer code that, when executed by the on-board vehicle computer system, causes the on-board vehicle computer system to: to determine a location of the vehicle using a GPS unit compliant with a dedicated short-range communication (DSRC), the location having an accuracy of plus or minus 1.5 meters; identify an illuminated light in a driving environment of the vehicle based on the location of the vehicle; determine that the illuminated light is a point of interest based on a color of the illuminated light and a driver of the vehicle who is impaired by a type of color blindness; determine a vehicle action to be taken in response to the illuminated light being identified in the vehicle's driving environment and the illuminated light being the point of interest; receive head position data from one or more internal cameras of the vehicle, the head position data describing an orientation of the driver's head; determine, based on the head position data, that the orientation of the driver's head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, cause the AR headset to display an AR overlay that visually represents a word describing the vehicle action to be taken in response to identifying the illuminated light. [10] The system of claim 9, wherein the illuminated light is identified based on measurements of the driving environment recorded by one or more external cameras of the vehicle. [11] The system of claim 9 or 10, wherein the illuminated light is further identified by a basic security message, BSM, describing a location of the illuminated light. [12] The system of any one of claims 9 to 11, wherein the AR headset is a three-dimensional head-up display unit operable to display the AR overlay so that it visually appears as if it has three dimensions in the real world. [13] System according to one of claims 9 to 12, wherein the AR headset is AR glasses. [14] The system of any one of claims 9 to 13, wherein the AR overlay consists of text for a word describing the vehicle action. [15] The system of any one of claims 9 to 14, wherein the AR overlay is displayed in real time relative to the identification of the illuminated light. [16] A computer program product comprising a non-volatile memory of an on-board vehicle computer system of a vehicle storing computer-executable code that, when executed by the on-board vehicle computer system, causes the on-board vehicle computer system to: to determine a location of the vehicle using a GPS unit compliant with a dedicated short-range communication (DSRC), the location having an accuracy of plus or minus 1.5 meters; identify an illuminated light in a driving environment of the vehicle based on the location of the vehicle; determine that the illuminated light is a point of interest based on a color of the illuminated light and a driver of the vehicle who is impaired by a type of color blindness; determine a vehicle action to be taken in response to the illuminated light being identified in the vehicle's driving environment and the illuminated light being the point of interest; receive head position data from one or more internal cameras of the vehicle, the head position data describing an orientation of the driver's head; determine, based on the head position data, that the orientation of the driver's head indicates that the point of interest is within a field of view of the driver; and in response to determining that the point of interest is within the driver's field of view, display an augmented reality ("AR") overlay on an AR headset of the vehicle that visually depicts a word describing the vehicle action to be taken in response to identifying the illuminated light. [17] The computer program product of claim 16, wherein the illuminated light is identified based on measurements of the driving environment recorded by one or more on-board sensors of the vehicle. [18] A computer program product according to claim 16 or 17, wherein the illuminated light is further identified by a basic security message, BSM, describing a location of the illuminated light. [19] The computer program product of any one of claims 16 to 18, wherein the AR headset is a three-dimensional head-up display unit operable to display the AR overlay so that it visually appears as if it has three dimensions in the real world. [20] The computer program product of any of claims 16 to 19, wherein the AR overlay consists of text for a word describing the vehicle action.

Images