Highlighting vulnerable road users on hybrid augmented reality head-up displays

Abstract

The system and method comprise receiving sensor data detected by a vehicle's sensor system, wherein the sensor data includes one or more data fragments located outside the driver's line of sight; classifying, based on the sensor data, one or more data fragments as a vulnerable road user; and determining, based on the sensor data, the location of the vulnerable road user. The system and method also comprise assessing the degree of risk between the vulnerable road user and the vehicle and displaying, via head-up displays, a graphical warning message to alert the driver of the vehicle to the vulnerable road user. The graphical warning message comprises an augmented reality image overlay and a virtual image, the augmented reality image overlay being distinct from the virtual image.

Description

introduction

[0001] The information contained in this section serves to provide a general overview of the context of the disclosure. The work of the inventors mentioned herein, insofar as it is described in this section, as well as aspects of the description that were not part of the prior art at the time of filing, are neither expressly nor implicitly recognized as prior art with respect to the present disclosure.

[0002] The present disclosure relates generally to a system and method for highlighting vulnerable road users on hybrid augmented reality head-up displays. Generally, when a vehicle is in motion, its 360-degree sensing system continuously receives sensor data indicating the vehicle's surroundings. The sensor data may, for example, indicate that the vehicle's surroundings contain a vulnerable road user such as a pedestrian, cyclist, and / or animal. While existing head-up displays are particularly good at alerting drivers to vulnerable road users within a driver's field of vision (e.g., the driver's view while looking ahead), many vulnerable road users are located outside the driver's field of vision. Because the driver is constantly scanning the roadway, their orientation or...Bias can also influence the driver's assessment of a hazard when identifying and classifying safety risks.

[0003] Accordingly, for the safety of drivers and pedestrians, it is of particular importance to develop systems that effectively identify and classify vulnerable road users outside the driver's field of vision by combining augmented reality graphics with directional graphics to warn or alert the driver. Furthermore, delivering these hybrid graphics as warnings, leveraging pillar-to-pillar display capabilities, can integrate the warnings with the driver's view of the road while minimizing distractions. Providing automatic classification of a vulnerable road user and advance warning to the driver gives the driver additional time to take corrective action, while minimizing bias in hazard assessments.

[0004] DE 10 2023 003 826 A1 discloses a method for displaying obscured road users, wherein at least one road user obscured by an object for a vehicle's own within-the-line-of-sight detector is detected by at least one vehicle's own outside-the-line-of-sight detector in the vicinity of a vehicle, and wherein at least the current relative position of the road user with respect to the vehicle is determined. The method according to the invention is characterized in that the road user is displayed on an augmented reality display used in the vehicle in a contact analogous manner to its current location.

[0005] DE 10 2023 200 126 A1 discloses a method for displaying information on a vehicle display unit, wherein a vehicle's surrounding area is detected by a vehicle detection unit and the detected surrounding area is displayed, at least partially, on the display unit. Furthermore, hazard information concerning a condition of the vehicle and / or hazard information concerning a condition of another vehicle in the surrounding area is provided. Depending on the provided hazard information regarding the vehicle's condition and / or the provided hazard information regarding the condition of the other vehicle and the detected surrounding area, a drivable safety trajectory is generated, and the safety trajectory is displayed in an image on the display unit showing the surrounding area. Summary

[0006] One aspect of the disclosure provides a computer-implemented method for highlighting vulnerable road users on hybrid augmented reality head-up displays, which, when executed on data processing hardware, causes the data processing hardware to perform operations that include receiving sensor data detected by a vehicle's sensor system, wherein the sensor data contains one or more data fragments located outside the line of sight of the vehicle's driver, and classifying, based on the sensor data, the one or more data fragments as a vulnerable road user.The operations also include determining, based on sensor data, the location of the endangered road user, assessing the degree of danger posed by the endangered road user to the vehicle, and displaying a graphical warning message via head-up displays to alert the driver to the endangered road user. This graphical warning message comprises an augmented reality image overlay and a virtual image, the augmented reality overlay being distinct from the virtual image. The head-up displays include an augmented reality head-up display and a blackout head-up display.Simultaneous display, via the head-up displays, of the graphical warning message, which alerts the driver of the vehicle to the endangered road user, includes generating the augmented reality image overlay, generating the virtual image and simultaneously projecting the augmented reality image overlay and the virtual image onto a windshield of the vehicle, wherein simultaneous projection of the augmented reality image overlay and the virtual image onto the windshield of the vehicle includes projecting the augmented reality image overlay onto a transparent part of the windshield and projecting the virtual image onto a blackout part of the windshield.

[0007] Implementations of the disclosure may include one or more of the following optional features. In some implementations, simultaneously projecting the augmented reality image overlay and the virtual image onto a vehicle's windshield further includes determining, based on driver characteristics, a location on the windshield that corresponds to the driver's line of sight, and projecting the augmented reality image overlay onto that location.

[0008] In some examples, the sensor system includes one or more cameras, a head-on collision mitigation system, radio detection and ranging (RADAR), and light detection and ranging (LIDAR). In some implementations, the operations further include receiving vehicle data, calculating, based on the vehicle data, a vehicle trajectory and a distance between the vehicle and the endangered road user, and determining whether the endangered road user and the vehicle will interact.In these implementations, assessing the level of risk between the endangered road user and the vehicle can further include determining, based on the distance between the vehicle and the endangered road user and the vehicle's trajectory, the level of risk between the endangered road user and the vehicle, and generating the graphical warning message based on the level of risk between the endangered road user and the vehicle.

[0009] In some examples, displaying a graphical warning message via the head-up display to alert the driver to the endangered road user includes displaying the graphical warning message on the vehicle's windshield to indicate the location of the endangered road user. In some implementations, classifying one or more data fragments as the endangered road user based on sensor data includes classifying one or more data fragments as the endangered road user using a large language model.

[0010] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Further aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Brief description of the drawings

[0011] The drawings described herein are for illustrative purposes only of selected configurations and are not intended to limit the scope of the present disclosure. Fig. Figure 1 is a schematic view of an exemplary system for highlighting vulnerable road users on hybrid augmented reality head-up displays. Fig. Figure 2 is a schematic view of exemplary components of the system of Fig. 1. Fig. 3 is a flowchart of the model for vulnerable road users for the system of Fig. 1. Fig. Figure 4 is a schematic view of the system's head-up displays. Fig. 1. Fig. Figure 5 is a schematic view of the system's head-up displays. Fig. 1. Fig. Figure 6 is a schematic view of the head-up displays of the system by Fig. 1. Fig. Figure 7 is a flowchart of an exemplary arrangement of operations for a procedure for highlighting vulnerable road users on hybrid augmented reality head-up displays.

[0012] The corresponding reference numbers correspond to the relevant parts in the drawings. Detailed description

[0013] With reference to the accompanying drawings, exemplary configurations are now described in more detail. These exemplary configurations are provided in such a way as to make this disclosure complete and to fully convey the scope of the disclosure to the person skilled in the art. Specific details are set forth, such as examples of certain components, devices, and methods, to enable a comprehensive understanding of the configurations of this disclosure. It is obvious to the person skilled in the art that specific details need not be used, that exemplary configurations can be embodied in many different forms, and that the specific details and the exemplary configurations are not intended to be interpreted in such a way as to limit the scope of the disclosure.

[0014] The terminology used herein serves only to describe certain exemplary configurations and is not to be understood as restrictive. As used herein, the articles "a," "an," and "the" in the singular are intended to include the plural forms unless the context clearly indicates otherwise. The terms "comprise," "comprehensive," "inclusive," and "exhibiting" are integrative and thus specify the presence of features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The procedural steps, processes, and operations described herein are not to be interpreted as requiring that they be carried out in the order discussed or illustrated unless they are expressly identified as the order of implementation.Additional or alternative steps can be used.

[0015] When an element or layer is referred to as being "on," "interacting with," "connected with," "attached to," or "coupled with" another element or layer, it may be directly on, interacting with, connected with, attached to, or coupled with the other element or layer, or there may be intervening elements or layers. Conversely, when an element is referred to as being "directly on," "directly interacting with," "directly connected with," "directly attached to," or "directly coupled with" another element or layer, there may be no intervening elements or layers. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.).As used herein, the term “and / or” includes all combinations of one or more of the associated listed elements.

[0016] The terms "first," "second," "third," etc., may be used herein to describe different elements, components, areas, layers, and / or sections. These elements, components, areas, layers, and / or sections are not intended to be restricted by these terms. These terms may only be used to distinguish one element, component, area, layer, or section from another. Terms such as "first," "second," and other numerical terms do not imply any sequence or order unless clearly evident from the context. Thus, a first element, first component, first area, first layer, or first section could be...A first section, which is referred to below, may also be called the second element, second component, second area, second layer or second section, without deviating from the teachings of the exemplary configurations.

[0017] In this application, including the definitions below, the term "module" may be replaced by the term "circuit". The term "module" may refer to, be part of, or comprise an application-specific integrated circuit (ASIC), a digital, analog, or mixed analog / digital discrete circuit, a digital, analog, or mixed analog / digital integrated circuit, a combinational logic circuit, a field-programmable gate array (FPGA), a processor (common, dedicated, or group) that executes code, a memory (common, dedicated, or group) that stores the code executed by a processor, other suitable hardware components that provide the described functionality, or a combination of some or all of the above, such as in a system-on-a-chip.

[0018] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, and / or road hazards. The term "shared processor" includes a single processor that executes part or all of the code of multiple modules. The term "group processor" includes a processor that, in combination with additional processors, executes part or all of the code of one or more modules. The term "shared memory" includes a single memory that stores part or all of the code of multiple modules. The term "group memory" includes memory that, in combination with additional memory, stores part or all of the code of one or more modules. The term "memory" can be a subset of the term "computer-readable medium."The term "computer-readable medium" does not include the transient electrical and electromagnetic signals that propagate through a medium and can therefore be considered a concrete, or physical, and non-transient storage medium. Non-restrictive examples of non-transient storage include physical computer-readable media, including non-volatile memory, magnetic memory, and optical memory.

[0019] The devices and methods described in this application can be partially or completely implemented by means of one or more computer programs executed by one or more processors. The computer programs contain instructions executable by processors, which are stored on at least one non-transient, physical, computer-readable medium. The computer programs can also contain and / or access stored data.

[0020] A software application (i.e., a software resource) can refer to computer software that causes a computer device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Examples of applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0021] Non-transitory memory can be physical devices used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program status information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only or read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (which is typically used, for example, for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory.Random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and floppy disks or tapes.

[0022] These computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and can be implemented in a procedural high-level and / or object-oriented programming language and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or equipment (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine instructions and / or data to a programmable processor containing a machine-readable medium that receives machine instructions as a machine-readable signal.The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0023] Various implementations of the systems and techniques described herein may be realized in a digital electronic and / or optical circuit arrangement, an integrated circuit arrangement, specially designed or engineered ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include an implementation in one or more computer programs that are executable and / or interpretable on a programmable system containing at least one programmable processor, which may be designed or suited for special or general purposes and is coupled such that it receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits data and instructions to these.

[0024] The processes and logical sequences described in this description can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by processing input data and producing an output. The processes and logical sequences can also be executed by a special-purpose logic circuit arrangement, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing a computer program include, for example, general-purpose and specialized microprocessors, as well as any or multiple processors of any type of digital computer. Generally, a processor receives instructions, or...Instructions and data from read-only memory or random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is functionally coupled to them to receive data from or to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices, and magnetic disks, such as...Internal hard drives or removable disks, magneto-optical disks, as well as CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by or integrated into a logic circuit for special purposes.

[0025] To enable user interaction, one or more aspects of the disclosure can be implemented on a computer that includes a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display), or touchscreen monitor, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, with which the user can input information into the computer. Other types of devices can also be used to enable user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, verbal, or tactile input.Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example by sending web pages to a web browser on a user's client device in response to requests received by the web browser.

[0026] Referring to Fig. 1. In some implementations, a system 100 comprises a vehicle 10 and / or a remote system 60, which is connected to the vehicle 10 via a network 40. The vehicle 10 and / or the remote system 60 control a system 200 to warn of endangered road users ( Fig. 2) out, and this can travel on a lane 34 ( Fig. 4-6). In short, and as described in more detail below, the system 200 for warning vulnerable road users is configured to receive sensor data 20 containing one or more data fragments 28 that are outside the line of sight 104 of a driver 102 of the vehicle 10, and, when the system 200 for warning vulnerable road users classifies the one or more data fragments 28 as a vulnerable road user 30, displays a graphical warning message 202 via head-up displays 204 to alert the driver 102 to the vulnerable road user 30.In particular, by alerting the driver 102 to the vulnerable road user 30 located outside the driver's line of sight 104, the system provides the driver 102 with time to take corrective action, such as slowing down, stopping, and / or performing evasive maneuvers that the driver 102 would otherwise not see. Vulnerable road users 30 located outside the driver's line of sight 104 may include, for example, pedestrians, cyclists, and / or animals that are at risk from the vehicle 10. Furthermore, the graphical warning messages 202 generated by the system 200 to warn of vulnerable road users can create additional situational awareness in automated driving modes, increasing user confidence and facilitating vehicle takeover.

[0027] As used herein, a vulnerable road user 30 who is outside the driver's line of sight 104 can generally refer to the vulnerable road user 30's positioning relative to the vehicle 10 in real time, such that a vehicle occupant (e.g., the driver 102) cannot perceive the vulnerable road user 30 if the occupant is facing the front of the vehicle 10. The perception of the vulnerable road user 30 can be based, at least in part, on where the occupant is seated inside the vehicle 10 and includes areas outside the vehicle 10 that are not normally observable when the occupant's head is turned toward the front of the vehicle 10, such as the shoulder of the roadway 34.These areas may also include areas outside the vehicle 10 that are not normally observable when the vehicle occupant's head turns from neck to right and left. An example of the driver's line of sight 104 is shown in . Fig. Figure 2 shows where the dashed arrows define the line of sight 104 and where the vulnerable road user 30, who is within the line of sight 104, can be perceived by the driver 102 when the driver 102 looks towards the front of the vehicle 10. In some implementations, the line of sight 104 is a cone-shaped area in the direction of travel of the vehicle 10 in front of the vehicle with a field of view of 120 degrees. The line of sight 104 may also extend over a distance of approximately 800 meters from the driver 102. In other implementations, the line of sight 104 is dynamic based on the geographical area and weather conditions in which the vehicle 10 is traveling.

[0028] In the examples shown, the system 200 for warning vulnerable road users is implemented within a vehicle 10. However, the system 200 for warning vulnerable road users can be implemented on other data processing or computing devices (e.g., computing devices connected to the vehicle 10), such as, without limitation, a smartphone, a tablet, a smart display, a desktop / laptop, a smartwatch, a smart device, or smart glasses / headset. The vehicle 10 comprises data processing hardware 12 and storage hardware 14, which stores instructions that, when executed on the data processing hardware 12, cause the data processing hardware 12 to perform operations.Furthermore, the vehicle 10 includes a driver tracking system 50 configured to detect the driver's eye position while driving the vehicle 10 and to determine the driver's characteristics 52. These driver characteristics 52 may include the driver's eye position, head position, or body position. Specifically, the driver tracking system 50 may include a facial imaging camera located inside the vehicle 10 that continuously tracks the driver's eyes to determine the direction toward which the driver's eyes are focused. The facial imaging camera may be configured to capture images or video of the driver's face during a journey and extract the driver's eye position, head position, and / or body position from the images / video. In other implementations, the driver tracking system 50 includes a sensor (e.g.,an optical sensor) configured to determine the movement of the driver's eyes 102 by utilizing the light reflected from the cornea of ​​the eye. The driver characteristics 52 can be used to determine the driver's line of sight 104. In other words, the driver characteristics 52 can indicate where the driver 102 is looking, which, together with the driver's peripheral vision, can form the driver's line of sight 104.

[0029] As in Fig. 1 and Fig. As shown in Figure 2, the vehicle 10 is configured to receive sensor data 20 detected / captured by means of a sensor system 16. This sensor data 20 may contain one or more data fragments 28 located outside the driver's line of sight 104 102. The sensor system 16 may include one or more cameras, a frontal collision mitigation system, a radio-based detection and distance measurement system (RADAR), a light-based detection and distance measurement system (LIDAR) capable of capturing image data, and other external sensors of the vehicle 10. While the in Fig. Although the sensor system 16 shown in Figure 1 is located at the front of the vehicle, it should be noted that the sensor system 16 can include sensors located anywhere in the vehicle. For example, the sensor system 16 can perform 360-degree environmental sensing of the vehicle's surroundings 10.

[0030] The remote system 60 (e.g., server, cloud computing environment) also includes data processing hardware 62 and storage hardware 64, which stores instructions that, when executed on the data processing hardware 62, cause the data processing hardware 62 to perform operations. In some examples, the execution of system 200 to warn of endangered road users is split between the vehicle 10 and the remote system 60. As described below with reference to Fig. 2 and Fig. As described in more detail in Section 3, the system 200, running on the vehicle 10 and / or the remote system 60, executes a vulnerable road user warning model 300, which is configured to receive the sensor data 20 containing the one or more data fragments 28 located outside the driver's line of sight 104 and generates the graphical warning message 202, which warns or alerts the driver 102 if the one or more data fragments 28 correspond to a vulnerable road user 30. Because the one or more data fragments 28 of the sensor data 20 are located outside the driver's line of sight 104, the driver 102 may not be able to perceive the vulnerable road user 30 represented by the one or more data fragments 28.

[0031] As in Fig. 1 and Fig. As shown in Figure 2, the vehicle 10 further comprises a windshield 18, which provides the system 200 with pillar-to-pillar warning capabilities for endangered road users. In particular, the windshield 18 comprises a transparent section 24 and a blackout section 26. The transparent section 24 can generally refer to the part of the windshield 18 through which the driver 102 perceives the areas outside the vehicle 10. The blackout section 26 can generally refer to an opaque or darkened / blackened part of the windshield 18, in which, for example, a vacuum fluorescence display (VFP), a light-emitting diode (LED) display, a driver information center display, a radio display, any text device, a head-up display (HUD), a touchscreen display, a liquid crystal display (LCD), etc., is used.An instrument cluster and / or an infotainment system may be displayed.

[0032] Referring to Fig. 1-3 While the vehicle 10 is in motion, the vulnerable road user model 300 executes as input the sensor data 20 detected by the vehicle 10's sensor system 16. The sensor data 20 may include one or more data fragments 28 and / or additional image data detected by the sensor system 16 and may indicate that one or more data fragments 28 are moving in the direction of the vehicle 10. The vulnerable road user model 300 may additionally receive vehicle data 22, which includes the vehicle 10's direction, speed, steering angle, acceleration, braking, output from an automated driving system, and / or current location.Based on the sensor data 20 and the vehicle data 22, the system 200 generates a warning for vulnerable road users, which executes the model 300 for vulnerable road users, then the graphical warning message 202 or ignores the sensor data 20 (i.e., takes no action in respect of it).

[0033] Referring to Fig. Figure 3 depicts the Model 300 for vulnerable road users. Here, while the vehicle 10 is in motion, the Model 300 for vulnerable road users continuously receives and processes the sensor data 20, containing one or more data fragments 28, detected by the sensor system 16, the vehicle data 22, and the driver characteristics 52, to determine whether the graphical warning message 202 should be output to the head-up displays 204. In operation 310, the Model 300 for vulnerable road users receives the sensor data 20, detected by the sensor system 16, which contains one or more data fragments 28. In operation 320, the Model 300 for vulnerable road users then classifies whether any vulnerable road users 30 are detected in the one or more data fragments 28.Model 300 for vulnerable road users can, for example, include an image classifier, such as a large language model, configured to receive one or more data fragments 28 and identify one or more data fragments 28 as a vulnerable road user 30. For example, the image classifier can process one or more data fragments 28 and classify only one or more data fragments 28 as a vulnerable road user 30 if a threshold value of one or more data fragments 28 corresponds to a defined vulnerable road user 30.

[0034] In some implementations, during an operation 320, the object classifier also generates classification characteristics 322 of the endangered road user 30, such as the type of endangered road user 30, the responsibility of the endangered road user 30, and / or a trajectory assessment of the endangered road user 30. The type of endangered road user 30 can generally refer to whether the endangered road user 30 is an adult, a child, an animal, a pedestrian, a cyclist, and / or a parked vehicle. The responsibility of the endangered road user 30 can generally refer to whether the endangered road user 30 is wearing protective equipment (e.g., a helmet), is making eye contact with the vehicle 10, and / or is otherwise distracted.The trajectory assessment of the vulnerable road user 30 generally refers to whether the vulnerable road user 30 is moving on lane 34 in the direct path of vehicle 10 or is approaching vehicle 10 at an unsafe distance. As described in more detail below, the classification characteristics 322 generated for vulnerable road users by the object classifier of model 300 can be used to determine a urgency and / or hazard assessment of the vulnerable road user 30.

[0035] In an operation 330, the model 300 can generate location data 332 of the vulnerable road user 30 based on the sensor data 20 containing one or more data fragments 28. The location data 332 can include, for example, the location of the vulnerable road user 30, its trajectory, and / or its speed. The location of the vulnerable road user 30 can include, for example, on the roadway 34, in the direct path of the vehicle 10, or at the periphery of the roadway 34. The trajectory of the vulnerable road user 30 can include whether the vulnerable road user 30 is approaching the roadway 34 and / or moving along the roadway 34.

[0036] In an operation 340, the vulnerable road user model 300 can assess a hazard level between the vulnerable road user 30 and the vehicle 10 and, based on the hazard level, generate the graphical warning message 202, which alerts the driver 102 to the vulnerable road user 30. Specifically, the vulnerable road user model 300 receives the classification characteristics 322 of the vulnerable road user 30, the location data 332 of the vulnerable road user 30, the vehicle data 22, and the driver characteristics 52, and performs a hazard assessment of the vehicle 10 and the vulnerable road user 30 to generate the hazard level. For example, a pedestrian who is focused on a mobile device while approaching a crosswalk may have a high hazard level (i.e.,a cyclist may pose an increased risk of collision with the vehicle 10), whereas an adult riding a bicycle on the shoulder of the carriageway 34 and wearing a helmet may pose a low level of hazard (i.e., a low risk of collision and / or injury to the vulnerable road user 30). In some implementations, the graphical warning message 202, which alerts or warns the driver 102, can be configured based on the level of hazard. For example, the graphical warning message 202 may include graphics of varying sizes, shades and brightness of colors (e.g., green, yellow, orange, red, etc.) and / or flashing elements to inform the driver 102 of an approaching vulnerable road user 30 based on the level of hazard.

[0037] In some examples, determining the hazard level involves calculating, based on vehicle data 22, a trajectory of vehicle 10 and a distance between vehicle 10 and the vulnerable road user 30. The trajectory of vehicle 10 can refer to a position, direction, and / or speed of the vehicle based on the current vehicle data 22. The vehicle data 22 can be measured / reported, for example, by an inertial measurement unit (IMU). As part of the hazard assessment, the vulnerable road user model 300 can determine whether vehicle 10 and the vulnerable road user 30 will interact. This interaction can occur when vehicle 10 comes into physical contact with (i.e., hits) and / or passes the vulnerable road user 30.The vulnerable road user model 300 can determine that vehicle 10 may interact with vulnerable road user 30 by comparing the trajectories of vehicle 10 and vulnerable road user 30 and determining whether their respective trajectories will intersect or approach each other. In both scenarios, the safety of vulnerable road user 30, driver 102, vehicle 10, and any vehicles and / or pedestrians in the vicinity is significantly improved if driver 102 is informed in advance (i.e., via graphical warning message 202) about vulnerable road user 30.

[0038] Furthermore, the hazard assessment includes determining how quickly vehicle 10 and the vulnerable road user 30 will interact. In other words, model 300 for vulnerable road users determines, based on the distance between the vulnerable road user 30 and vehicle 10, as well as the trajectory of vehicle 10, the time until an interaction between the vulnerable road user 30 and vehicle 10 occurs, at which point the vehicle 10 and the vulnerable road user 30 are in the same location (i.e., co-locate). In these implementations, the hazard level, and thus the graphical warning message 202, can be generated based on the time until the interaction between the vulnerable road user 30 and vehicle 10.In other words, the graphical warning message 202 can be configured based on the level of danger, which is indicated by the time until the interaction between the endangered road user 30 and the vehicle 10. The size, highlighting, colors, gradient, and / or flashing of the graphical warning message 202 can change depending on the urgency indicated by the time until the interaction between the endangered road user 30 and the vehicle 10.

[0039] Back on Fig. Referring to section 2, after the model 300 assesses the degree of risk between the vulnerable road user 30 and the vehicle 10, the system 200 for warning vulnerable road users generates the graphical warning message 202. Specifically, the system 200 for warning vulnerable road users outputs the graphical warning message 202 to the head-up displays 204. The graphical warning message 202 is further modified / configured based on the driver characteristics 52 (e.g., the driver's eye position 102). In particular, the system 200 generates the graphic warning message 202 for warning of endangered road users on the basis of the driver characteristics 52 and a concavity of the windshield 18 of the vehicle 10, in order to adapt the display of the graphic warning message 202 on the windshield 18 while the driver 102 scans or searches the roadway 34.If the driver characteristics 52 indicate that the driver 102 is looking in a particular direction, the system 200 can update the xyz positions on the windscreen 18 onto which the graphical warning message 202 is to be projected by means of the head-up displays 204 to warn of endangered road users. For example, if the driver characteristics 52 indicate that the driver 102 is scanning the lane 34 between right and left (e.g., laterally), the graphical warning message 202 can, based on the driver characteristics 52, shift or slide a portion of the graphical warning message 202 laterally along the windscreen 18.Similarly, if the roadway 34 describes a curve or otherwise deviates from a straight line, the driver characteristics 52 may cause the system 200 to warn of endangered road users to shift the graphical warning message 202 to align it with the eye position of the driver 102 within the line of sight 104 of the driver 102.

[0040] As shown, the head-up displays 204 comprise an augmented reality head-up display 206 and a blackout head-up display 208, wherein the augmented reality head-up display 206 is configured to project images onto the transparent part 24 of the windshield 18, and the blackout head-up display 208 is configured to project images onto the blackout part 26 of the windshield 18. The graphical warning message 202 can generally comprise an augmented reality image overlay 210 and a virtual image 212, each projected by the respective components of the head-up displays 204.For example, the augmented reality head-up display 206 can receive the graphical warning message 202, which contains the augmented reality image overlay 210, and, based on the xyz position determined by the driver characteristics 52, project the augmented reality image overlay 210 onto the transparent part 24 of the windshield 18 at a location on the windshield 18 that corresponds to the driver's line of sight 104. In some implementations, the augmented reality image overlay 210 indicates a direction of the endangered road user 30. Similarly, the blackout head-up display 208 can be configured to display images on the blackout part 26 of the windshield 18.In particular, the blackout head-up display 208 can receive the graphical warning message 202, which contains the virtual image 212, and project the virtual image 212 onto the blackout section 26 of the windshield 18, aligning the virtual image 212 laterally along the blackout section 26 with the endangered road user 30 on the roadway 34. In some implementations, the augmented reality image overlay 210 and the virtual image 212 are projected onto the windshield 18 simultaneously.

[0041] Referring to Fig. Figures 4-6 illustrate exemplary components 400-600, including the vehicle 10, which executes the system 200 to warn of vulnerable road users in order to alert the driver 102 to a vulnerable road user 30. By warning the driver 102 (e.g., via the graphical warning message 202), the driver 102 is given additional time to react to / avoid the vulnerable road user 30. As described above, it should be noted that the graphical warning message 202 projected onto the windshield 18 can be configured / modified based on the estimated degree of hazard between the vulnerable road user 30 and the vehicle 10. For example, the graphical warning message 202 can use images of different sizes, colors and / or other warning techniques to attract the attention of the driver 102 without distracting the driver 102 from the road.In particular, the augmented reality image overlay 210 and the virtual image 212 can be projected onto the windscreen simultaneously, and the augmented reality image overlay 210 can differ from the virtual image 212.

[0042] With reference to Fig. 4. Based on sensor data 20 containing one or more data fragments 28, the system 200 for warning vulnerable road users can detect that a vulnerable road user 30 is on the shoulder of the carriageway 34 (i.e., outside the driver's line of sight 104). The vulnerable road user 30 may, for example, include the classification characteristics 322 of an adult wearing a helmet while riding a bicycle on the shoulder of the carriageway 34. Additionally, the location data 332 may indicate that the vulnerable road user 30 is traveling along the side of the carriageway 34 and is unlikely to cross the path of the vehicle 10. In response to the detection of the vulnerable road user 30, the system 200 for warning vulnerable road users can assess the level of danger between the vulnerable road user 30 and the vehicle 10 as medium.Subsequently, the system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210a and a virtual image 212a, based on the average level of risk and the driver characteristics 52 detected by means of the driver tracker system 50, to warn of endangered road users.

[0043] As shown, the augmented reality image overlay 210a is projected onto the transparent part 24 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210a appears to be positioned on the roadway 34 in front of the vehicle 10 and contains a large arrow alerting the driver 102 to the endangered road user 30 on the right side of the vehicle 10. Based on the medium level of danger, the augmented reality image overlay 210a containing the large arrow can be shaded in a warning color (e.g., orange). However, the augmented reality image overlay 210a can also contain different colors, sizes, or gradients depending on the medium level of danger between the endangered road user 30 and the vehicle 10.Simultaneously, the virtual image 212a is projected onto the blackout section 26 of the windshield 18 to indicate the endangered road user 30. As shown, the virtual image 212a contains a graphic of a bicycle to identify the endangered road user 30. Like the augmented reality image overlay 210a, the virtual image 212a can be shaded between the endangered road user 30 and the vehicle 10 in a warning color (e.g., orange) or any other color (e.g., red, yellow, green), size, or gradient, depending on the degree of danger.

[0044] With reference to Fig. 5 The system 200 can detect, based on sensor data 20 containing one or more data fragments 28, that vulnerable road users 30a, 30b are near a zebra crossing on the roadway 34. In this case, the driver 102 of the vehicle 10 may be in the process of turning left into the roadway where the vulnerable road users 30a, 30b are located, so that the vulnerable road users 30a, 30b are outside the driver 102's line of sight 104. In this example, the vulnerable road user 30a may exhibit the classification characteristics 322 of an adult pedestrian making eye contact with the vehicle 10. Furthermore, the location data 332 can indicate that the endangered road user 30a is crossing the carriageway 34 and that the trajectory of the endangered road user 30a will intersect the trajectory of vehicle 10.Based on the classification characteristics 322 and the location data 332, the system 200 for warning vulnerable road users can assess the hazard level between vulnerable road user 30a and vehicle 10 as high. In contrast, vulnerable road user 30b may have the classification characteristics 322 of an adult pedestrian looking at a mobile device (i.e., inattentive), but the location data 332 indicates that they are on a sidewalk (i.e., not on the roadway) and are currently standing still. In this case, the system 200 for warning vulnerable road users can assess the hazard level between vulnerable road user 30b and vehicle 10 as low.Subsequently, the system 200 generates the graphical warning message 202, which includes an augmented reality image overlay 210b and virtual images 212b1, 212b2, to warn of endangered road users based on the high degree of danger of the endangered road user 30a and the low degree of danger of the endangered road user 30b as well as the driver characteristics 52 detected by means of the driver tracker system 50.

[0045] As shown, the augmented reality image overlay 210b is projected onto the transparent part 24 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210b appears to be positioned on the roadway 34 in front of the vehicle 10 and includes a circular platform below the endangered road user 30b, alerting the driver 102 to the endangered road user 30 into which the driver 102 steers the vehicle 10. Based on the high level of danger, the augmented reality image overlay 210b can be shaded in a color for an urgent warning (e.g., red). However, the augmented reality image overlay 210b can contain different colors, sizes or gradients based on the average level of danger between the endangered road user 30 and the vehicle 10.Since the endangered road user 30b has a low level of risk, the graphical warning message 202 does not need to contain an augmented reality image overlay 210.

[0046] Simultaneously, virtual images 212b1, 212b2 are projected onto the blackout section 26 of the windshield 18 to indicate the vulnerable road users 30a, 30b. As shown, virtual images 212b1, 212b2 each contain a graphic of a pedestrian to identify the vulnerable road users 30a, 30b, with each virtual image 212b1, 212b2 aligned with its respective vulnerable road user along a longitudinal axis of the blackout section 26 of the windshield. Like the augmented reality image overlay 210b, virtual images 212b1, 212b2 can have different colors (e.g., red, orange, yellow, green), sizes, and / or gradients depending on the level of danger between the vulnerable road users 30a, 30b and the vehicle 10.For example, virtual image 212b1 can be shaded red to indicate the high level of danger between the endangered road user 30a and the vehicle 10, while virtual image 212b2, which corresponds to the endangered road user 30b, can be shaded yellow to indicate the low level of danger between the endangered road user 30b and the vehicle 10.

[0047] With reference to Fig. 6. While driver 102 continues driving vehicle 10 to turn left onto the road, vulnerable road users 30a and 30b are present. Vulnerable road user 30a may enter the line of sight 104 of driver 102 of vehicle 10, while vulnerable road user 30b remains outside the line of sight 104 of driver 102. Although vulnerable road user 30a is within the line of sight 104 of driver 102, the system 200 can still generate the graphical warning message 202 for warning vulnerable road users. For example, the location data 332 of vulnerable road user 30a may indicate that vulnerable road user 30a is too close to vehicle 10 (e.g.,within 10 meters), so that a circular platform under the feet of the endangered road user 30a is not visible to the driver 102 despite a high degree of danger between the endangered road user 30a and the vehicle 10. Here, the graphical warning message 202 generated by the system 200 to warn of endangered road users can contain an augmented reality image overlay 210c.

[0048] As shown, the augmented reality image overlay 210c is projected onto the transparent part 24 of the windshield 18 at a point on the windshield 18 within the driver's line of sight 104, so that the augmented reality image overlay 210c appears to be positioned directly above the endangered road user 30a in front of the vehicle 10. Here, the augmented reality image overlay 210c includes a box that overlays the endangered road user 30a. Based on the high level of danger, the augmented reality image overlay 210b can be shaded in one color for a priority warning (e.g., red). However, based on the medium level of danger, the augmented reality image overlay 210b can have different colors, sizes, or gradients between the endangered road user 30a and the vehicle 10.

[0049] Fig. Section 7 includes a flowchart of an exemplary sequence of operations for a Procedure 700 for highlighting vulnerable road users on hybrid augmented reality head-up displays. The Procedure 700 can be described with reference to Fig. 1-6 are described. A data processing hardware (e.g., the data processing hardware 12, 62 of Fig. 1) can execute instructions placed on memory hardware (e.g., memory hardware 14, 64 of Fig. 1) are stored to perform the exemplary sequence of operations for procedure 700.

[0050] In Operation 702, Procedure 700 comprises receiving sensor data 20 detected by a sensor system 16 of a vehicle 10. The sensor data 20 specify one or more data fragments 28 that are located outside the line of sight 104 of a driver 102 of the vehicle 10. In Operation 704, Procedure 700 also comprises classifying, based on the sensor data 20, the one or more data fragments 28 as a vulnerable road user 30. In Operation 706, Procedure 700 further comprises determining, based on the sensor data 20, the location of the vulnerable road user 30.

[0051] Procedure 700 also includes, in operation 708, the assessment of the hazard level of the endangered road user 30 for the vehicle 10. In operation 710, procedure 700 further includes the display, via head-up displays 204, of a graphical warning message 202, which alerts the driver 102 of the vehicle 10 to the endangered road user 30. Here, the graphical warning message 202 comprises an augmented reality image overlay 210 and a virtual image 212, whereby the augmented reality image overlay 210 differs from the virtual image 212.

[0052] Several implementations have been described. Nevertheless, it is understood that various modifications can be made without deviating from the spirit and scope of the revelation. Accordingly, other implementations fall within the scope of the following claims.

[0053] The foregoing description has been provided for illustrative and descriptive purposes. It is not intended to be exhaustive or complete, nor does it limit the disclosure. Individual elements or features of a particular configuration are generally not restricted to that particular configuration but are interchangeable and may be used in a selected configuration, even if not specifically illustrated or described. They may also be varied in various ways. Such variations are not to be considered a deviation from the disclosure, and all such modifications are to be included within the scope of the disclosure.

Claims

[1] A computer-implemented method which, when executed on data processing hardware (12), causes the data processing hardware (12) to perform operations, comprising: Receiving sensor data (20) detected by means of a sensor system (16) of a vehicle (10), wherein the sensor data (20) contains one or more data fragments that are outside the line of sight of a driver of the vehicle (10); Classify, based on the sensor data (20), one or more data fragments as a vulnerable road user; Determine, based on the sensor data (20), location data of the endangered road user; Assessing the degree of risk to the vehicle posed by the endangered road user (10); and Displaying, via head-up displays, a graphical warning message that alerts the driver of the vehicle (10) to the endangered road user, wherein the graphical warning message comprises an augmented reality image overlay and a virtual image, wherein the augmented reality image overlay differs from the virtual image, wherein the head-up displays comprise an augmented reality head-up display (206) and a blackout head-up display (208), wherein simultaneously displaying, via the head-up displays, the graphical warning message that alerts the driver of the vehicle (10) to the endangered road user comprises: Creating the augmented reality image overlay; Creating the virtual image; and Simultaneous projection of the augmented reality image overlay and the virtual image onto a windshield (18) of the vehicle (10), wherein simultaneous projection of the augmented reality image overlay and the virtual image onto the windshield (18) of the vehicle (10) includes projecting the augmented reality image overlay onto a transparent part (24) of the windshield (18) and projecting the virtual image onto a blackout part (26) of the windshield (18). [2] Method according to claim 1, comprising simultaneously projecting the augmented reality image overlay and the virtual image onto a windshield (18) of the vehicle (10): Determine, based on driver characteristics, a location on the windscreen (18) that corresponds to a line of sight (104) of the driver; and Projecting the augmented reality image overlay onto the area of ​​the windscreen (18) that corresponds to the driver's line of sight (104). [3] Method according to claim 1, wherein the sensor system (16) comprises one or more of the following elements: Cameras; a system for mitigating a head-on collision; a radio-based detection and distance measurement system (RADAR); and a light-based detection and distance measurement system (LIDAR). [4] The method of claim 1, wherein the operations further comprise: Receiving vehicle data; Calculate, based on the vehicle data, a trajectory of the vehicle (10) and a distance between the vehicle (10) and the endangered road user; and Determine whether the vulnerable road user and the vehicle (10) will interact. [5] Method according to claim 4, comprising an assessment of the degree of danger between the endangered road user and the vehicle (10): Determine, based on the distance between the vehicle (10) and the endangered road user and the trajectory of the vehicle (10), the degree of danger between the endangered road user and the vehicle (10); and Generating the graphical warning message based on the degree of risk between the endangered road user and the vehicle (10). [6] Method according to claim 1, wherein displaying the graphic warning message via the head-up displays, which alerts the driver of the vehicle (10) to the endangered road user, comprises displaying the graphic warning message on a windscreen (18) of the vehicle (10) to indicate a location of the endangered road user.

Images