Road condition warning on reflective and extended head-up displays

Abstract

A system and method comprise receiving road data indicating a road hazard moving towards the vehicle, determining, based on the road data, that the road hazard is within the vehicle's lane, and determining a distance between the vehicle and the road hazard. The system and method also comprise receiving driver characteristics of the vehicle's driver from a driver tracking system and simultaneously displaying, via head-up displays, a graphical warning message alerting the vehicle's driver to the road hazard. The graphical warning message comprises an augmented reality image overlay and a virtual image, the augmented reality image overlay being distinct from the virtual image.

Description

introduction

[0001] The present invention relates to a computer-implemented method.

[0002] For example, the publication DE 10 2023 200 126 A1 describes a computer-implemented method according to the preamble of claim 1.

[0003] This disclosure relates generally to a system and method for road condition warnings on reflective and extended head-up displays. Generally, when a vehicle is in motion, its 360-degree sensing system continuously receives sensor data that identifies or indicates the vehicle's surroundings. The sensor data may, for example, indicate that the vehicle's surroundings contain an imminent hazard such as one or more other vehicles (e.g., an accident), a pothole, an animal or object, a fire, rocks, poor traction, and construction or work zones that include reduced-speed zones, construction buoys, road workers, and / or road closures. In either case, systems that alert the driver to the road hazard provide the driver with crucial time to take one or more corrective actions.

[0004] Furthermore, for safety reasons, it is crucial that any systems utilizing visual warning methods do not divert the driver's primary focus from the road ahead. To this end, systems that incorporate the driver's line of sight to place warnings for work zones and / or other road hazards can significantly reduce accidents. For example, delivering warnings using pole-to-pole display capabilities can integrate safety alerts with the driver's view of the road while minimizing distractions. Summary

[0005] An aspect of the invention provides a computer-implemented method for a road condition warning on reflective and augmented head-up displays, which, when executed on data processing hardware, causes the data processing hardware to perform operations that include receiving road data, comprising one or more sensor data detected by means of a vehicle's sensor system and third-party data, wherein the road data indicates a road hazard moving towards the vehicle, and determining, based on the road data, that the road hazard is located within a lane of the vehicle.The operations also include determining a distance between the vehicle and the road hazard, receiving driver characteristics of the vehicle's driver from a driver tracking system, and simultaneously displaying, via head-up displays, a graphical warning message that alerts the vehicle's driver to the road hazard, wherein the graphical warning message comprises an overlay of an augmented reality image and a virtual image, the augmented reality image overlay being distinct from the virtual image.

[0006] The head-up displays include an augmented reality head-up display and a blackout head-up display. Simultaneous display of the graphical warning message, alerting the driver to road hazards, via the head-up displays further includes generating an augmented reality image overlay, generating a virtual image, and simultaneously projecting the augmented reality image overlay and the virtual image onto the vehicle's windshield. This simultaneous projection of the augmented reality image overlay and the virtual image onto the windshield also includes projecting the augmented reality image overlay onto a transparent portion of the windshield and projecting the virtual image onto a blackout portion.

[0007] In addition, simultaneously projecting the augmented reality image overlay and the virtual image onto a vehicle's windshield can involve determining, based on the driver's characteristics, a location on the windshield that corresponds to the driver's line of sight, and projecting the augmented reality image overlay onto that location.

[0008] In some examples, the sensor system includes one or more cameras, radio detection and ranging (RADAR), and light detection and ranging (LIDAR). In some implementations, the operations further include receiving vehicle data, calculating the vehicle's trajectory based on that data, and determining whether the road hazard and the vehicle will interact. In these implementations, the operations may also include determining, based on the distance between the vehicle and the road hazard and the vehicle's trajectory, the time until interaction between the road hazard and the vehicle, and generating the graphical warning message based on that time.In some examples, displaying a graphical warning message via the head-up display to alert the driver to the road hazard also includes displaying the graphical warning message on the vehicle's windshield to indicate the location of the hazard. In some implementations, the graphical warning message is configured to convey one or more of the following information: the distance to the hazard, the speed of approach, the severity of the hazard, and the type of hazard.

[0009] Another, not independently claimed, aspect of the disclosure provides a system for road condition warnings on reflective and extended head-up displays, comprising data processing hardware and storage hardware associated with the data processing hardware. The storage hardware stores instructions which, when executed by the data processing hardware, cause the data processing hardware to perform operations that include receiving road data, which comprises one or more sensor data detected by a vehicle's sensor system and third-party data, wherein the road data indicates a road hazard moving toward the vehicle, and determining, based on the road data, that the road hazard is located within the vehicle's lane.The operations also include determining a distance between the vehicle and the road hazard, receiving driver characteristics of a driver of the vehicle from a driver tracking system, and simultaneously displaying, via head-up displays, a graphical warning message that alerts the driver of the vehicle to the road hazard, wherein the graphical warning message includes an augmented reality image overlay and a virtual image, the augmented reality image overlay being distinct from the virtual image.

[0010] This aspect can include one or more of the following optional features. In some implementations, the head-up displays include an augmented reality head-up display and a blackout head-up display. In these implementations, simultaneous display of the graphical warning message, alerting the driver to the road hazard, via the head-up displays can include generating the augmented reality image overlay, generating the virtual image, and simultaneously projecting the augmented reality image overlay and the virtual image onto the vehicle's windshield. The simultaneous projection of the augmented reality image overlay and the virtual image onto the vehicle's windshield can also include projecting the augmented reality image overlay onto a transparent portion of the windshield and projecting the virtual image onto a blackout portion of the windshield.Additionally or alternatively, the simultaneous projection of the augmented reality image overlay and the virtual image onto a vehicle's windshield can involve determining, based on the driver's characteristics, a location on the windshield that corresponds to the driver's line of sight, and projecting the augmented reality image overlay onto that location.

[0011] In some examples, the sensor system includes one or more cameras, radio-based detection and distance measurement (RADAR), and light-based detection and distance measurement (LIDAR). In some implementations, the operations also include receiving vehicle data, calculating a vehicle trajectory based on that data, and determining whether the road hazard and the vehicle will interact. In these implementations, the operations may further include determining, based on the distance between the vehicle and the road hazard and the vehicle's trajectory, the time until interaction between the road hazard and the vehicle, and generating the graphical warning message based on that time.In some examples, displaying a graphical warning message via the head-up display to alert the driver to the road hazard also includes displaying the graphical warning message on the vehicle's windshield to indicate the location of the hazard. In some implementations, the graphical warning message is configured to convey one or more of the following information: the distance to the hazard, the speed of approach, the severity of the hazard, and the type of hazard.

[0012] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Further aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Brief description of the drawings

[0013] The drawings described herein are for illustrative purposes only of selected configurations and are not intended to limit the scope of the present disclosure. Fig. Figure 1 is a schematic view of an exemplary system that uses reflective head-up displays to warn of an impending accident outside the line of sight. Fig. Figure 2 is a schematic view of exemplary components of the system of Fig. 1. Fig. Figure 3 is a flowchart of a road condition model for the system of Fig. 1. Fig. Figures 4A-4D are schematic views of the head-up displays of the system by Fig. 1. Fig. 5A and Fig. 5B are exemplary views of the head-up displays from Fig. 4A. Fig. 6A and Fig. Figure 6B shows exemplary views of the head-up displays from Fig. 4B. Fig. 7A and Fig. 7B are exemplary views of the head-up displays from Fig. 4C. Fig. 8A and Fig. 8B are exemplary views of the head-up displays from Fig. 4D. Fig. Figure 9 is a flowchart of an exemplary arrangement of operations for a procedure for road condition warning on reflective and extended head-up displays.

[0014] The corresponding reference numbers identify the corresponding parts in the drawings. Detailed description

[0015] With reference to the accompanying drawings, exemplary configurations are now described in more detail. These exemplary configurations are provided in such a way as to make this disclosure complete and to fully convey the scope of the disclosure to the person skilled in the art. Specific details are set forth, such as examples of certain components, devices, and methods, to enable a comprehensive understanding of the configurations of this disclosure. It is obvious to the person skilled in the art that specific details need not be used and that exemplary configurations can be embodied in many different forms.

[0016] The terminology used herein serves only to describe certain exemplary configurations and is not to be understood as restrictive. As used herein, the articles "a," "an," and "the" in the singular are intended to include the plural forms unless the context clearly indicates otherwise. The terms "comprise," "comprehensive," "inclusive," and "exhibiting" are integrative and thus specify the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The procedural steps, processes, and operations described herein are not to be interpreted as requiring that they be carried out in the order discussed or illustrated unless they are expressly identified as the order of implementation.Additional or alternative steps can be used.

[0017] When an element or layer is referred to as being "on," "interacting with," "connected with," "attached to," or "coupled with" another element or layer, it may be directly on, interacting with, connected with, attached to, or coupled with the other element or layer, or there may be intervening elements or layers. Conversely, when an element is referred to as being "directly on," "directly interacting with," "directly connected with," "directly attached to," or "directly coupled with" another element or layer, there may be no intervening elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).As used herein, the term “and / or” includes all combinations of one or more of the associated listed elements.

[0018] The terms "first," "second," "third," etc., may be used herein to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections are not intended to be restricted by these terms. These terms may only be used to distinguish one element, component, area, layer, or section from another. Terms such as "first," "second," and other numerical terms do not imply any sequence or order unless clearly evident from the context. Thus, a first element, first component, first area, first layer, or first section could be...A first section, which is referred to below, may also be called the second element, second component, second area, second layer or second section, without deviating from the teachings of the exemplary configurations.

[0019] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or comprise an application-specific integrated circuit (ASIC), a digital, analog, or mixed analog / digital discrete circuit, a digital, analog, or mixed analog / digital integrated circuit, a combinational logic circuit, a field-programmable gate array (FPGA), a processor (common, dedicated, or group) that executes code, a memory (common, dedicated, or group) that stores the code executed by a processor, other suitable hardware components that provide the described functionality, or a combination of some or all of the above, such as in a system-on-a-chip.

[0020] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or road hazards. The term "shared processor" includes a single processor that executes part or all of the code of multiple modules. The term "group processor" includes a processor that, in combination with additional processors, executes part or all of the code of one or more modules. The term "shared memory" includes a single memory that stores part or all of the code of multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores part or all of the code of one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" does not include the transient electrical and electromagnetic signals that propagate through a medium and can therefore be considered a concrete, or physical, and non-transient storage medium. Non-restrictive examples of non-transient storage include physical computer-readable media, including non-volatile memory, magnetic memory, and optical memory.

[0021] The devices and methods described in this application can be partially or completely implemented by means of one or more computer programs executed by one or more processors. The computer programs contain instructions executable by processors, which are stored on at least one non-transient, physical, computer-readable medium. The computer programs can also contain and / or access stored data.

[0022] A software application (i.e., a software resource) can refer to computer software that causes a computer device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Examples of applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0023] Non-transitory memory can be physical devices used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program status information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only or read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (which is typically used, for example, for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory.Random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and floppy disks or tapes.

[0024] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in a procedural high-level and / or object-oriented programming language and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or equipment (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor containing a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0025] Various implementations of the systems and techniques described herein may be realized in a digital electronic and / or optical circuit arrangement, an integrated circuit arrangement, specially designed or engineered ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include an implementation in one or more computer programs that are executable and / or interpretable on a programmable system containing at least one programmable processor, which may be designed or suited for special or general purposes and is coupled such that it receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits data and instructions to these.

[0026] The processes and logical sequences described in this description can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by processing input data and producing an output. The processes and logical sequences can also be executed by a special-purpose logic circuit arrangement, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing a computer program include, for example, general-purpose and specialized microprocessors, as well as any or multiple processors of any type of digital computer. Generally, a processor receives instructions, or...Instructions and data from read-only memory or random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is functionally coupled to them to receive data from or to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices, and magnetic disks, such as...Internal hard drives or removable disks, magneto-optical disks, as well as CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or integrated into a logic circuit for special purposes.

[0027] To enable user interaction, one or more aspects of the disclosure can be implemented on a computer that includes a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display), or touchscreen monitor, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, with which the user can input information into the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example by sending web pages to a web browser on a user's client device in response to requests received by the web browser.

[0028] Referring to Fig. 1 comprises a system 100, in some implementations a vehicle 10 and / or a remote system 60, which is connected to the vehicle 10 via a network 40. The vehicle 10 and / or the remote system 60 operate a road condition warning system 200 ( Fig. 2) out, and this can travel on a lane 34 ( Fig. 4A-4D). In short, and as described in more detail below, the road condition warning system 200 is configured to receive road data 20 indicating that a road hazard 32 is moving towards the vehicle 10, and, if the road hazard 32 is within the lane 34 of the vehicle 10, displays a graphical warning message 202 via head-up displays 204 to alert the driver 102 to the road hazard 32.

[0029] As used herein, a road hazard 32 may include an object located on or below the surface of the carriageway 34, such as one or more other vehicles (e.g., a crash), a pothole, an animal or object, a fire, rocks, poor traction, and the like. In some implementations, the road hazard 32 may include a work zone, which may include a reduced speed zone, construction buoys, road workers, and lane closures. In either case, alerting the driver 102 to the road hazard 32 located within the carriageway 34 provides the driver 102 with time to take one or more corrective actions. For example, if the road hazard 32 is a pothole, the graphical warning 202 may provide the driver 102 with sufficient time to change lanes to avoid the pothole.If the road hazard 32 is a work zone, the graphical warning message 202 can similarly give the driver 102 sufficient time to either reduce the vehicle's speed 10 or change lanes. Furthermore, the graphical warning messages 202 generated by the road condition warning system 200 can create additional situational awareness in automated driving modes, increasing user confidence and facilitating vehicle takeover.

[0030] In the examples shown, the road condition warning system 200 is implemented in a vehicle 10. However, the road condition warning system 200 can be implemented on other data processing or computing devices (e.g., computing devices connected to the vehicle 10), such as, without limitation, a smartphone, a tablet, a smart display, a desktop / laptop, a smartwatch, a smart device, or smart glasses / headset. The vehicle 10 comprises data processing hardware 12 and storage hardware 14, which stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations.Furthermore, the vehicle 10 includes a driver tracking system 50 configured to detect the driver's eye position while driving the vehicle 10 and to determine the driver's characteristics 52. These driver characteristics 52 may include the driver's eye position, head position, or body position. Specifically, the driver tracking system 50 may include a facial imaging camera located inside the vehicle 10 that continuously tracks the driver's eyes to determine the direction toward which the driver's eyes are focused. The facial imaging camera may be configured to capture images or video of the driver's face during a journey and extract the driver's eye position, head position, and / or body position from the images / video. In other implementations, the driver tracking system 50 includes a sensor (e.g.,an optical sensor), which is configured to determine the movement of the driver's eyes 102 by utilizing the light reflected from the cornea of ​​the eye.

[0031] The driver characteristics 52 can be used to determine the driver's line of sight 104. In other words, the driver characteristics 52 can indicate where the driver 102 is looking, which, together with the driver's peripheral vision, can form the driver's line of sight 104. As used here, the driver's line of sight 104 can generally refer to the positioning of the road hazard 32 relative to the vehicle 10 in real time, such that a vehicle occupant (e.g., the driver 102) can perceive the road hazard 32 when looking towards the front of the vehicle 10. The perception of the road hazard 32 can be based, at least in part, on where the vehicle occupant is seated inside the vehicle 10 and includes areas outside the vehicle 10 that are normally observable when the vehicle occupant's head is turned towards the front of the vehicle 10.These areas may also include areas outside the vehicle 10 that are naturally observable when the vehicle occupant's head turns from the neck to the right and to the left. An example of the driver's line of sight 104 is shown in . Fig. Figure 2 shows where the dashed arrows define the line of sight 104 and where road hazards 32 located within the line of sight 104 can be perceived by the driver 102 when looking towards the front of the vehicle 10. In some implementations, the line of sight 104 is a cone-shaped area in the direction of travel of the vehicle 10 in front of the vehicle with a field of view of 120 degrees. The line of sight 104 may also extend over a distance of approximately 800 meters from the driver 102. In other implementations, the line of sight 104 is dynamic based on the geographical area and weather conditions in which the vehicle 10 is traveling.

[0032] As in Fig. 1 and Fig. As shown in Figure 2, the vehicle 10 is configured to receive road data 20, which may include sensor data 22 and data 24 from third parties. To implement this, the vehicle 10 also includes a sensor system 16 configured to capture / receive sensor data 22. The sensor system 16 may include one or more cameras, radio-based detection and distance measurement (RADAR), and light-based detection and distance measurement (LIDAR) capable of capturing image data, as well as other external sensors of the vehicle 10. It should be noted that, while the in Fig. The sensor system 16 shown in Figure 1 is located at the front of the vehicle, but the sensor system 16 can include sensors located anywhere in the vehicle. For example, the sensor system 16 can provide 360-degree environmental sensing of the vehicle's surroundings. The data 24 from third parties can include known road conditions 34 identifying data, such as data received from Vehicle-to-Everything (V2X), cellular data, Dedicated Short-Range Communications (DSRC), a Global Positioning System (GPS), wireless communication (e.g., the network), a road database, mobile mapping applications that query road conditions via crowdsourcing, and / or beacons or data packets transmitted by work zones that identify the work zone for the vehicle 10.

[0033] The remote system 60 (e.g., server, cloud computing environment) also includes data processing hardware 62 and storage hardware 64, which stores instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. In some examples, the execution of the road condition warning system 200 is split between the vehicle 10 and the remote system 60. As referred to in Fig. 2 and Fig. As described in more detail below, the road condition warning system 200 running on the vehicle 10 and / or the remote system 60 executes a road condition model 300 that is configured to receive road data 20 indicating that a road hazard 32 is moving towards the vehicle 10 and generates the graphical warning message 202 when the road data 20 indicates that the vehicle 10 will interact with the road hazard 32 (i.e., is on the lane 34 of the vehicle 10). The vehicle interaction may include, for example, coming into contact with (i.e., meeting) the road hazard 32 and / or passing through or driving through a road hazard 32 (e.g., a work zone). In both scenarios, the safety of the driver 102, the vehicle 10, and any vehicles and / or pedestrians in the vicinity is significantly improved if the driver 102 is alerted in advance (i.e.,The graphic warning message 202) informs about the road hazard 32.

[0034] As in Fig. 1 and Fig. As shown in Figure 2, the vehicle 10 further comprises a windshield 18, which provides the road condition warning system 200 with pillar-to-pillar display options. In particular, the windshield 18 comprises a transparent section 28 and a blackout section 30. The transparent section 28 can generally refer to the part of the windshield 18 through which the driver 102 perceives areas outside the vehicle 10. The blackout section 30 can generally refer to an opaque or darkened / blackened area of ​​the windshield 18, in which, for example, a vacuum fluorescence display (VFP), a light-emitting diode (LED) display, a driver information center display, a radio display, any text device, a head-up display (HUD), a touchscreen display, a liquid crystal display (LCD), etc., is used.An instrument cluster and / or an infotainment system may be displayed.

[0035] Referring to Fig. 1-3 While the vehicle 10 is in motion, it executes the road condition model 300, which receives as input the road data 20. This data includes one or more sensor data 22 detected by the vehicle 10's sensor system 16 and data 24 from third parties. The sensor data 22 may include one or more data fragments or image data detected by the sensor system 16 and may indicate that a road hazard 32 is moving towards the vehicle 10. Additionally or alternatively, the data 24 from third parties may contain an indication of an impending road hazard 32 (e.g., a pothole or a work zone).The road condition warning system 200 can additionally receive vehicle data 26, which includes the direction of the vehicle 10, the speed of the vehicle 10, the steering angle of the vehicle 10, the acceleration of the vehicle 10, the braking of the vehicle 10, outputs from an automated driving system of the vehicle 10, and / or the current location of the vehicle 10. The road condition warning system 200, which executes the road condition model 300, then determines whether the road hazard 32 is located in a lane 34 of the vehicle 10 and, based on the driver characteristics 52 detected by the driver tracker system 50, generates the graphical warning message 202 for output to the head-up displays 204.

[0036] Referring to Fig. Figure 3 depicts the road condition model 300. Here, while the vehicle 10 is in motion, the road condition model 300 continuously receives and processes the road data 20, which includes the sensor data 22 detected by the sensor system 16 and the data 24 from third parties, the vehicle data 26, and the driver characteristics 52, in order to determine whether the graphical warning message 202 should be output to the head-up displays 204. In operation 310, the road condition model 300 receives the road data 20, which includes the sensor data 22 detected by the sensor system 16 and the data 24 from third parties, as well as the vehicle data 26 of the vehicle 10. In operation 320, the road condition model 300 then determines whether any road hazards 32 are detected that are moving in the direction of the vehicle 10.For example, the road condition model 300 can determine, based on the location of the vehicle 10 and the location of the road hazards 32, whether any road data 20 indicate that road hazards 32 are approaching the vehicle 10. If the road condition model 300 determines that a road hazard 32 is moving in the direction of the vehicle 10, the operations further include, in operation 330, calculating the distance between the road hazard 32 and the vehicle 10 and a trajectory of the vehicle 10. Here, the trajectory of the vehicle 10 can refer to a position, direction, and / or speed of the vehicle based on the current vehicle data 26. The vehicle data 26 can be measured / reported by an inertial measurement unit (IMU).In some implementations, the road state model 300 further determines during operation 330 whether the road hazard 32 and the vehicle 10 will interact. For example, the road state model 300 determines whether the vehicle 10 will come into contact with the road hazard 32 (e.g., hit it or collide with it) or pass through the road hazard 32 (e.g., a work zone).

[0037] In operation 340, the road condition model 300 determines how quickly vehicle 10 and road hazard 32 will interact. In other words, based on the distance between road hazard 32 and vehicle 10 and the trajectory of vehicle 10, the road condition model 300 determines the time until an interaction occurs between road hazard 32 and vehicle 10, at which point they will be in the same location (co-locate). In these implementations, the graphical warning message 202 can be generated based on the time until the interaction between road hazard 32 and vehicle 10. In other words, the graphical warning message 202 can be configured based on the urgency indicated by the time until the interaction between road hazard 32 and vehicle 10. Specifically, the size, highlighting, colors, gradient, etc., can be adjusted.The pattern and / or flashing of graphical warning message 202 may change depending on the urgency indicated by the time until interaction between the road hazard 32 and the vehicle 10. The urgency indicated by the time until interaction between the road hazard 32 and the vehicle 10 may include the distance between the road hazard 32 and the vehicle 10, the speed of approach by the vehicle 10, and / or the severity of the hazard 32 (e.g., crossing animals versus wildfire). Thus, graphical warning message 202 can be configured to convey the distance between the vehicle 10 and the road hazard 32, the speed of approach by the vehicle 10, the severity of the road hazard 32, and / or the type of road hazard.

[0038] As in Fig. As shown in Figure 3, if the time until the interaction between the road hazard 32 and the vehicle 10 is longer (e.g., greater than twenty (20) seconds) and therefore less urgent, the road condition model 300 can generate / display a graphical warning message 202 during an operation 350, informing the driver 102 of the impending road hazard 32. For example, the impending road hazard 32 may be outside the driver 102's line of sight 104. In this case, the graphical warning message 202 that warns the driver 102 may use smaller graphics, colors, and / or flashing elements to inform the driver 102 of the road hazard 32. If the time until the interaction between the road hazard and the vehicle is in the middle (e.g.,If the time until the collision between the road hazard 32 and the vehicle 10 is between twenty (20) seconds and five (5) seconds and is therefore imminent, the road condition model 300 can generate / display a graphical warning message 202 during an operation 360, alerting the driver 102 to the impending road hazard 32. The graphical warning message 202 warning the driver 102 can use larger graphics, brighter colors (e.g., orange), and / or flashing elements to inform the driver 102 of the approaching hazard 32. Alternatively, if the time until the collision between the road hazard 32 and the vehicle 10 is shorter (e.g., less than five (5) seconds) and therefore immediately imminent and urgent, the road condition model 300 can generate / display a graphical warning message 202 during an operation 370, warning the driver 102 of the impending road hazard 32. In this case, the graphical warning message 202, which warns the driver 102, may use larger graphics, brighter / stronger colors (e.g.Use red) and / or flashing elements to indicate that the imminent road hazard 32 is urgent.

[0039] Back on Fig. Referring to section 2, after the road condition model 300 determines that the road hazard 32 and the vehicle 10 will interact, the road condition warning system 200 generates the graphical warning message 202. Specifically, the road condition warning system 200 outputs the graphical warning message 202 to the head-up displays 204. The graphical warning message 202 is modified / configured based on the driver characteristics 52 (e.g., the driver's eye position 102). Specifically, the road condition warning system 200 generates the graphical warning message 202 based on the driver characteristics 52 and the concavity of the windshield 18 of the vehicle 10, in order to adapt the display of the graphical warning message 202 on the windshield 18 while the driver 102 scans the roadway 34.If the driver characteristics 52 indicate that the driver 102 is looking in a particular direction, the road condition warning system 200 can update the xyz positions on the windscreen 18 onto which the graphical warning message 202 is to be projected by means of the head-up displays 204. For example, if the driver characteristics 52 indicate that the driver 102 is scanning the lane 34 between right and left (e.g., laterally), the graphical warning message 202 can, based on the driver characteristics 52, shift or slide a portion of the graphical warning message 202 laterally along the windscreen 18. Similarly, if the roadway 34 describes a curve or otherwise deviates from a straight line, the driver characteristics 52 may cause the road condition warning system 200 to shift the graphical warning message 202 to align it with the driver's eye position 102 within the driver's line of sight 104.

[0040] As shown, the head-up displays 204 comprise an augmented reality head-up display 206 and a blackout head-up display 208, wherein the augmented reality head-up display 206 is configured to project images onto the transparent part 28 of the windshield 18, and the blackout head-up display 208 is configured to project images onto the blackout part 30 of the windshield 18. The graphical warning message 202 can generally include an augmented reality image overlay 210 and a virtual image 212, each projected by the respective components of the head-up displays 204.For example, the augmented reality head-up display 206 can receive the graphical warning message 202, which contains the augmented reality image overlay 210, and, based on the xyz position determined by the driver characteristics 52, project the augmented reality image overlay 210 onto the transparent part 28 of the windshield at a location on the windshield 18 within the driver's line of sight 104. In some implementations, the augmented reality image overlay 210 indicates a direction of the road hazard 32. Similarly, the blackout head-up display 208 can be configured to display images on the blackout part 30 of the windshield 18.In particular, the blackout head-up display 208 can receive the graphical warning message 202, which contains the virtual image 212, and project the virtual image 212 onto the blackout section 30 of the windshield 18, aligning the virtual image 212 laterally along the blackout section 30 with the road hazard 32 on the roadway 34. The augmented reality image overlay 210 and the virtual image 212 are projected onto the windshield 18 simultaneously.

[0041] Referring to Fig. Figures 4A-4D show exemplary components 400a-400d, which include the vehicle 10, which executes the road condition warning system 200 to warn the driver 102 that a road hazard 32 is approaching the vehicle 10. By warning the driver 102 (e.g., via the graphical warning message 202), the driver 102 is given additional time to react to / avoid the road hazard 32. It should be noted that the graphical warning message 202, projected onto the windscreen 18, can be configured / modified depending on the urgency of the time until the interaction between the road hazard 32 and the vehicle 10. For example, the graphical warning message 202 can use images of different sizes and colors and / or employ other warning techniques to attract the driver 102's attention without distracting the driver 102 from the road. Furthermore, it contains Fig. 5A-8B exemplary augmented reality image overlays 210a-210d and the virtual images 212a-212d, which are in Fig. The exemplary components 400a-400d shown in sections 4A-4D correspond to these. As described above, the augmented reality image overlay 210 and the virtual image 212 are projected simultaneously onto the windshield, and the augmented reality image overlay 210 may differ from the virtual image 212.

[0042] With reference to Fig. 4A, Fig. 5A and Fig. Based on road data 20, the road condition warning system 200 can detect that the vehicle 10 is approaching a road hazard 32 (e.g., a work zone) in the carriageway 34. The road hazard 32 may, for example, include a work zone with a reduced speed limit where the driver 102 must reduce the vehicle 10's speed to a safer level to comply with work zone regulations and to enhance the safety of workers and other drivers. In response to the detection of the road hazard 32, the road condition warning system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210a and a virtual image 212a, based on the driver characteristics 52 detected by the driver tracker system 50.As shown, the augmented reality image overlay 210a is projected onto the transparent portion 28 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210a appears to be positioned on the road in front of the vehicle 10 and includes the word "slow" to alert the driver 102 to reduce the vehicle 10's speed. Simultaneously, the virtual image 212a is projected onto the blackout portion 30 of the windshield 18 to indicate the road hazard 32. As shown, the virtual image 212a includes the inscription "work zone", a distance (e.g., 850 feet) to the road hazard 32, and a warning "reduce speed".

[0043] With particular reference to Fig. 5A can display the augmented reality image overlay 210a “slowly” in different colors (e.g., red, orange, yellow, green), sizes, or gradients depending on an urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. Similarly, as in Fig. As shown in Figure 5B, the virtual image 212a is updated / modified depending on the urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. For example, the virtual image 212a can be updated to reflect the change in the distance between the road hazard 32 and the vehicle 10 as the vehicle 10 approaches the work zone (i.e., the road hazard 32). Similarly, the virtual image 212a can contain escalating warning signals, ranging from a simple notification to the driver 102 of "reduce speed" to a warning to the driver 102 of "reduce speed ahead" to a request to the driver 102 of "reduce speed now".

[0044] With reference to Fig. 4B, Fig. 6A and Fig. 6B The road condition warning system 200 can detect, based on the road data 20, that the vehicle 10 is approaching a road hazard 32 (e.g., a work zone). The road hazard 32 may, for example, include a work zone with workers on or immediately adjacent to the roadway 34, where the driver 102 should exercise particular caution when passing the road hazard 32 (i.e., the work zone), such as slowing the vehicle 10 to a safer speed to comply with work zone regulations and to increase the safety of the workers and other drivers. In response to the detection of the road hazard 32, the road condition warning system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210b and a virtual image 212b, based on the driver characteristics 52 detected by the driver tracker system 50.As shown, the augmented reality image overlay 210b is projected onto the transparent part 28 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210b appears to be positioned on the roadway 34 in front of the vehicle 10 and includes a graphic indicating a construction site (e.g., a cone and a helmet) as well as directional arrows pointing to the right to alert the driver 102 to the worker in the work zone on the right side of the roadway 34. Simultaneously, the virtual image 212b is projected onto the blackout part 30 of the windshield 18 to indicate the road hazard 32. As shown, virtual image 212b contains a note “Work Zone”, a graphic indicating a person (i.e. the worker in the work zone) and a graphic indicating a traffic rule.

[0045] With particular reference to Fig. 6A The augmented reality image overlay 210b of a graphic indicating a construction site (e.g., a cone and a helmet) can be displayed in different colors (e.g., red, orange, yellow, green), sizes, or gradients depending on a level of urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. Additionally, the directional arrows can indicate a direction (e.g., left or right) of the road hazard 32 relative to the vehicle 10. Similarly, as in Fig. As shown in 6B, the virtual image 212b is updated / changed depending on the urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10.

[0046] With reference to Fig. 4C, Fig. 7A and Fig. 7B can detect, based on the road data 20, that the vehicle 10 is approaching a road hazard 32 (e.g., a work zone) in the carriageway 34. The road hazard 32 may, for example, include a work zone with a lane closure, where the driver 102 must move the vehicle 10 into an adjacent lane at a safer speed to comply with the work zone regulations and maintain the safety of the workers and other drivers. In response to the detection of the road hazard 32, the road condition warning system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210c and a virtual image 212c, based on the driver characteristics 52 detected by the driver tracker system 50.As shown, the augmented reality image overlay 210c is projected onto the transparent portion 28 of the windshield 18 at a location on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210c appears to be positioned on the road in front of the vehicle 10 and contains graphics indicating open (i.e., a circle) and closed (i.e., a cross) lanes to alert the driver 102 to change lanes from the closed lane to an open lane. Specifically, the open graphic is displayed to overlay the portion of the roadway 34 that contains an open lane, while the closed graphic is displayed to overlay the portion of the roadway 34 that contains the upcoming closed lane. At the same time, the virtual image 212c is projected onto the blackout part 30 of the windshield 18 to indicate the road hazard 32.As shown, virtual image 212a contains a note “Work zone”, a distance (e.g. 850 feet) to road hazard 32 and a warning “Road closure”.

[0047] With particular reference to Fig. 7A The augmented reality image overlay 210c, which contains graphics indicating open (i.e., a circle) and closed (i.e., a cross) lanes, can be displayed in different colors (e.g., red, orange, yellow, green), sizes, or gradients depending on a level of urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. Similarly, as in Fig. As shown in 6B, the virtual image 212c is updated / changed depending on the urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10.

[0048] With reference to Fig. 4D, Fig. 8A and Fig. Based on road data 20, the road condition warning system 200 can detect that the vehicle 10 is approaching a road hazard 32 (e.g., a pothole) in the roadway 34. The road hazard 32 might, for example, include a work zone with a pothole located on a lane marking, where the driver 102 might want to swerve to avoid the pothole without affecting other drivers. In response to the detection of the road hazard 32, the road condition warning system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210d and a virtual image 212d, based on the driver characteristics 52 detected by the driver tracker system 50.As shown, the augmented reality image overlay 210d is projected onto the transparent part 28 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210d appears to be positioned on the road in front of the vehicle 10 and includes a graphic of a general warning symbol as well as a right-pointing directional arrow to alert the driver 102 to the pothole (i.e., the road hazard 32) on the right side of the vehicle 10. Simultaneously, the virtual image 212d is projected onto the blackout part 30 of the windshield 18 to indicate the road hazard 32. As shown, the virtual image 212d contains a general warning symbol without directional arrows.Rather, the blackout head-up display 208 aligns the virtual image 212d laterally along the blackout portion 30 of the windscreen 18 based on the driver characteristics 52, which indicate the location on the windscreen 18 that the driver 102 is looking at. In particular, while the vehicle 10 is moving in relation to the road hazard 32 and / or while the driver 102 is scanning the roadway 34, the virtual image 212d can slide laterally along the blackout portion 30 to maintain alignment with the road hazard 32.

[0049] With particular reference to Fig. 8A The augmented reality image overlay 210d of the graphic of the general warning symbol and the directional arrow can be displayed in different colors (e.g., red, orange, yellow, green), sizes, or gradients depending on an urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. Similarly, as in Fig. As shown in Figure 8B, the virtual image 212d is updated / modified depending on the urgency defined by the distance between the road hazard 32 and the vehicle 10 and / or the speed of the vehicle 10. Optionally, the virtual image 212d can be updated / modified depending on the type of road hazard 32 approaching the vehicle 10. For example, the road condition warning system 200 can detect the type of road hazard 32 based on road data 20 via images and / or image fragments or third-party reports. In response, the road condition warning system 200 generates the graphical warning message 202, which includes the virtual image 212d containing the type of road hazard 32. The type of road hazard 32 can refer to common road hazards such as, without limitation, a crossing animal, a pothole, crossing deer, low traction, fire, rockfall, etc.

[0050] Fig. Section 9 includes a flowchart of an exemplary sequence of operations for a Procedure 900 for a road condition warning on reflective and extended head-up displays. The Procedure 900 can be described with reference to Fig. 1-8A are described. A data processing hardware (e.g., the data processing hardware 12, 62 of Fig. 1) can execute instructions stored in memory hardware (e.g., memory hardware 14, 64 of Fig. 1) are stored to perform the exemplary sequence of operations for procedure 900.

[0051] In Operation 902, Procedure 900 comprises receiving road data 20, which includes one or more sensor data 22 detected by a sensor system 16 of the vehicle 10 and data 24 from third parties. The road data 20 may indicate a road hazard 32 moving in the direction of the vehicle 10. In Operation 904, Procedure 900 also comprises determining, based on the road data 20, that the road hazard 32 is located within a lane 34 of the vehicle 10. In Operation 906, Procedure 900 further comprises determining a distance between the vehicle 10 and the road hazard 32.

[0052] Procedure 900 also includes, in operation 908, the reception of driver characteristics 52 of a driver 102 of vehicle 10 from a driver tracking system 50. In operation 910, procedure 900 further includes the simultaneous display, via head-up displays 204, of a graphical warning message 202, which alerts the driver 102 of vehicle 10 to the road hazard 32. Here, the graphical warning message 202 comprises an augmented reality image overlay 210 and a virtual image 212, wherein the augmented reality image overlay 210 differs from the virtual image 212.

[0053] Several implementations have been described. Nevertheless, it is understood that various modifications can be made without deviating from the spirit and scope of the revelation.

[0054] The foregoing description has been provided for illustrative and descriptive purposes.

Claims

[1] Computer-implemented method (900) which, when executed on data processing hardware (62), causes the data processing hardware (62) to perform operations, comprising: Receiving (902) road data (20) comprising one or more sensor data (22) detected by means of a sensor system (16) of a vehicle (10) and data (24) of third parties, wherein the road data (20) indicate a road hazard (32) moving in the direction of the vehicle (10); Determine (904), based on the road data (20), that the road hazard (32) is located within one lane (34) of the vehicle (10); Determining (906) a distance between the vehicle (10) and the road hazard (32); Receiving (908) driver characteristics (52) of a driver (102) of the vehicle (10) from a driver tracking system (50); simultaneous display (910), via head-up displays (204), of a graphic warning message (202) that alerts the driver (102) of the vehicle (10) to the road hazard (32), wherein the graphic warning message (202) comprises an augmented reality image overlay (210) and a virtual image (212), wherein the augmented reality image overlay (210) differs from the virtual image (212); the head-up displays (204) include an augmented reality head-up display (206) and a blackout head-up display (208); including the simultaneous display (910) via the head-up displays (204) of the graphic warning message (202) which alerts the driver (102) of the vehicle (10) to the road hazard (32): Creating the augmented reality image overlay (210); Creating the virtual image (212); and Simultaneous projection of the augmented reality image overlay (210) and the virtual image (212) onto a windshield (18) of the vehicle (10); characterized by , that includes a simultaneous projection of the augmented reality image overlay (210) and the virtual image (212) onto the windshield (18) of the vehicle (10), a projection of the augmented reality image overlay (210) onto a transparent part (28) of the windshield (18) and a projection of the virtual image (212) onto a blackout part (30) of the windshield (18). [2] Method (900) according to claim 1, comprising simultaneously projecting (910) the augmented reality image overlay (210) and the virtual image (212) onto a windshield (18) of the vehicle (10): Determine, based on the driver characteristics (52), a location on the windscreen (18) that corresponds to a line of sight (104) of the driver (102); and Projecting the augmented reality image overlay (210) onto the area of ​​the windscreen (18) that corresponds to the line of sight (104) of the driver (102). [3] Method (900) according to claim 1, wherein the sensor system (16) comprises one or more of the following elements: Cameras; a radio-based detection and distance measurement system (RADAR); and a light-based detection and distance measurement system (LIDAR). [4] Method (900) according to claim 1, wherein the operations further comprise: Receiving vehicle data (26); Calculate, based on the vehicle data (26), a trajectory of the vehicle (10); and Determine whether the road hazard (32) and the vehicle (10) will interact. [5] Method (900) according to claim 4, wherein the operations further comprise: Determine, based on the distance between the vehicle (10) and the road hazard (32) and the trajectory of the vehicle (10), a time until an interaction between the road hazard (32) and the vehicle (10); and Generating the graphical warning message (202) based on the time until the interaction between the road hazard (32) and the vehicle (10). [6] Method (900) according to claim 1, wherein displaying the graphic warning message (202) via the head-up displays (204) to alert the driver (102) of the vehicle (10) to the road hazard (32) comprises displaying the graphic warning message (202) on a windshield (18) of the vehicle (10) to indicate a location of the road hazard (32). [7] Method (900) according to claim 1, wherein the graphical warning message (202) is configured to transmit one or more of the following information: a distance from the road hazard (32); a speed of approaching the road hazard (32); a severity level of road hazard (32); and a type of road hazard (32).

Images