Risk notification route planner

Abstract

A system for providing an alert to an occupant within a vehicle includes a system controller in communication with a plurality of on-board sensors, other vehicles, and an infrastructure system; an augmented reality head-up display system; and a full windshield display system, each system adapted to project an image onto an inner surface of a windshield of the vehicle. The system controller is adapted to: detect at least one approaching vehicle, calculate a distance of the at least one approaching vehicle from the intersection, a speed and a trajectory of the at least one approaching vehicle, determine whether the at least one approaching vehicle constitutes a threat, and classify the at least one approaching vehicle as a threat vehicle, and providing the warning message and the enhancement graph by projecting the warning message and the enhancement graph onto the inner surface of the windshield using at least one of the AR-HUD system and the full windshield display system.

Description

Technical Field

[0001] The present disclosure relates to a system and method for providing a warning to an occupant in a vehicle when the vehicle is at an intersection to inform the occupant of the presence of a threatening vehicle. Background Art

[0002] Vehicles are equipped with many sensors to monitor the environment around the vehicle and provide warnings when objects are approaching, etc. Vehicles are also equipped with rearview mirrors to enable occupants in the vehicle to see objects near the vehicle that may be approaching the vehicle. Current sensor systems and rearview mirrors in vehicles are not suitable for actively monitoring other vehicles approaching intersections to probabilistically determine whether such approaching vehicles pose a threat to the vehicle, notify occupants in the vehicle of the threat, and display warning messages and enhanced graphics on window surfaces in the vehicle to warn occupants, highlight threatening vehicles, and provide route suggestions to avoid threatening vehicles.

[0003] Therefore, while current systems and methods achieve their intended purposes, a need exists for a new and improved system and method for providing a warning to an occupant of a vehicle when the vehicle is at an intersection to inform the occupant of the presence of a threatening vehicle. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for issuing a warning to an occupant in a vehicle when the vehicle is located at an intersection to inform the occupant of the presence of a threatening vehicle includes: using a system controller that communicates with multiple on-board sensors, wirelessly communicates with other vehicles, and wirelessly communicates with infrastructure systems to detect at least one approaching vehicle moving toward the intersection, using the system controller to calculate the distance of the at least one approaching vehicle from the intersection, the speed and trajectory of the at least one approaching vehicle, using the system controller to determine whether the at least one approaching vehicle poses a threat, and if the at least one approaching vehicle poses a threat, classifying the at least one approaching vehicle as a threatening vehicle, providing a warning message and an enhanced graphic on the inner surface of the windshield of the vehicle by at least one of the following: using an augmented reality head-up display system (AR-HUD) to project the warning message and the enhanced graphic onto the inner surface of the windshield, changing the color of a route guidance graphic currently displayed by the AR-HUD, and using a full windshield display system to project the warning message and the enhanced graphic onto the inner surface of the windshield.

[0005] According to another aspect, providing warning messages and enhanced graphics onto the inner surface of a windshield of a vehicle further includes: determining, using a system controller, the warning messages and enhanced graphics to be provided; and determining, using the system controller, whether the warning messages and enhanced graphics are within the field of view (FOV) of the AR-HUD system, projecting the warning messages and enhanced graphics onto the inner surface of the windshield using the AR-HUD system when the warning messages and enhanced graphics are within the FOV of the AR-HUD system, and projecting the warning messages and enhanced graphics onto the inner surface of the windshield using the full windshield display system when the warning messages and enhanced graphics are not within the FOV of the AR-HUD system.

[0006] According to another aspect, determining, using a system controller, whether at least one approaching vehicle poses a threat and, if the at least one approaching vehicle poses a threat, classifying the at least one approaching vehicle as a threat vehicle further comprises calculating, using a machine learning algorithm, a probability of the at least one approaching vehicle colliding with the vehicle, and classifying the at least one approaching vehicle as a threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a first predetermined threshold.

[0007] According to another aspect, providing a warning message and enhanced graphics onto an interior surface of a windshield of a vehicle further comprises providing a warning message including text information and providing enhanced graphics including at least one of a trajectory arrow, a highlighted threatening vehicle, and a warning icon.

[0008] According to another aspect, determining, using a system controller, the warning message and enhanced graphics to be provided further comprises: predicting, using a machine learning algorithm, a route of each of at least one approaching vehicle classified as a threat vehicle, and providing the warning message and enhanced graphics onto an inner surface of a windshield of the vehicle further comprises: projecting, using a full windshield display system, an enhanced graphic onto an inner surface of a windshield of the vehicle, the enhanced graphic comprising the predicted route of each of the at least one approaching vehicle classified as a threat vehicle.

[0009] According to another aspect, determining, using a system controller, the warning message and enhanced graphics to be provided further comprises: calculating, using a system controller and a machine learning algorithm, a suggested route for the vehicle, wherein, if the vehicle follows the suggested route, a probability of any one of at least one approaching vehicle classified as a threatening vehicle colliding with the vehicle is less than a predetermined threshold, and providing the warning message and enhanced graphics onto an inner surface of a windshield of the vehicle further comprises: projecting, using a full windshield display system, enhanced graphics including the suggested route for the vehicle onto an inner surface of a windshield of the vehicle.

[0010] According to another aspect, when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a first predetermined threshold, classifying at least one approaching vehicle as a threat vehicle further includes: classifying at least one approaching vehicle as a low risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds the first predetermined threshold; classifying at least one approaching vehicle as a medium risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a second predetermined threshold; and classifying at least one approaching vehicle as a high risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a third predetermined threshold.

[0011] According to another aspect, determining, with the system controller, the warning message and enhanced graphics to be provided further comprises determining, with the system controller, the warning message and enhanced graphics to be provided for each of at least one approaching vehicle classified as a threat vehicle based on classifying each threat vehicle as one of a low risk threat vehicle, a medium risk threat vehicle, or a high risk threat vehicle.

[0012] According to another aspect, the method further includes providing a secondary warning message and a secondary enhanced graphic on an inner surface of at least one of a side window and a rear window of the vehicle by projecting the secondary warning message and the secondary enhanced graphic onto an inner surface of at least one of a side window and a rear window of the vehicle using a full windshield display system.

[0013] According to another aspect, the method further includes providing information related to the vehicle's intended route to an entity outside the vehicle by projecting the information related to the vehicle's intended route onto an inner surface of at least one of a side window of the vehicle and a rear window of the vehicle using a full windshield display system.

[0014] According to several aspects of the present disclosure, a system for providing a warning to an occupant in a vehicle when the vehicle is located at an intersection to inform the occupant of the presence of a threatening vehicle includes: a system controller that communicates with a plurality of onboard sensors and has a wireless communication module adapted to allow the system controller to communicate with other vehicles and infrastructure systems; an augmented reality head-up display system (AR-HUD) that communicates with the system controller and is adapted to project an image onto an inner surface of a windshield of the vehicle; and a full windshield display system that is adapted to project an image onto an inner surface of a windshield of the vehicle, the system controller being adapted to: communicate with the plurality of onboard sensors and communicate with other vehicles and infrastructure systems via the wireless communication module; The invention also provides a method for detecting at least one approaching vehicle moving toward an intersection by communicating with the at least one approaching vehicle, calculating a distance of the at least one approaching vehicle from the intersection, a speed and a trajectory of the at least one approaching vehicle, determining whether the at least one approaching vehicle poses a threat, and if the at least one approaching vehicle poses a threat, classifying the at least one approaching vehicle as a threat vehicle, and providing a warning message and an enhanced graphic onto an inner surface of a windshield of the vehicle by at least one of: projecting the warning message and the enhanced graphic onto the inner surface of the windshield using an AR-HUD system, changing a color of a route guidance graphic currently displayed by the AR-HUD, and projecting the warning message and the enhanced graphic onto the inner surface of the windshield using a full windshield display system.

[0015] According to another aspect, when warning messages and enhanced graphics are provided onto the inner surface of a windshield of the vehicle, the system controller is further adapted to: determine what warning messages and enhanced graphics will be provided, determine whether the warning messages and enhanced graphics are within the field of view (FOV) of the AR-HUD system, project the warning messages and enhanced graphics onto the inner surface of the windshield using the AR-HUD system when the warning messages and enhanced graphics are within the FOV of the AR-HUD system, and project the warning messages and enhanced graphics onto the inner surface of the windshield using the full windshield display system when the warning messages and enhanced graphics are not within the FOV of the AR-HUD system.

[0016] According to another aspect, when determining whether an approaching vehicle poses a threat and classifying at least one approaching vehicle as a threat vehicle if at least one approaching vehicle poses a threat, the system controller is further adapted to: calculate a probability of the at least one approaching vehicle colliding with the vehicle using a machine learning algorithm, and classify the at least one approaching vehicle as a threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a first predetermined threshold.

[0017] According to another aspect, when providing the warning message and enhanced graphics onto the inner surface of the vehicle's windshield, the system controller is further adapted to: provide a warning message including text information, and provide enhanced graphics including at least one of: a trajectory arrow, a highlighted threatening vehicle, and a warning icon.

[0018] According to another aspect, when determining what warning message and enhanced graphics to provide, the system controller is further adapted to utilize a machine learning algorithm to predict a route for each of at least one approaching vehicle classified as a threat vehicle, and when providing the warning message and enhanced graphics onto an interior surface of a windshield of the vehicle, the system controller is further adapted to utilize a full windshield display system to project an enhanced graphic including the predicted route for each of at least one approaching vehicle classified as a threat vehicle onto an interior surface of a windshield of the vehicle.

[0019] According to another aspect, when determining what warning message and enhanced graphics to provide, the system controller is further adapted to calculate a suggested route for the vehicle using a machine learning algorithm, wherein if the vehicle follows the suggested route, the probability of any of at least one approaching vehicle classified as a threatening vehicle colliding with the vehicle is less than a predetermined threshold, and when providing the warning message and enhanced graphics onto an inner surface of a windshield of the vehicle, the system controller is further adapted to project the enhanced graphics including the suggested route for the vehicle onto the inner surface of the windshield of the vehicle using a full windshield display system.

[0020] According to another aspect, when classifying at least one approaching vehicle as a threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a first predetermined threshold, the system controller is further adapted to: classify at least one approaching vehicle as a low risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a first predetermined threshold, classify at least one approaching vehicle as a medium risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a second predetermined threshold, and classify at least one approaching vehicle as a high risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a third predetermined threshold, and when determining what warning message and enhanced graphics to provide, the system controller is further adapted to determine what warning message and enhanced graphics to provide for each of the at least one approaching vehicles classified as threat vehicles based on each threat vehicle being classified as one of a low risk threat vehicle, a medium risk threat vehicle, or a high risk threat vehicle.

[0021] According to another aspect, the system controller is further adapted to provide a secondary warning message and secondary enhanced graphics onto an inner surface of at least one of the side windows and rear windows of the vehicle by projecting the secondary warning message and secondary enhanced graphics onto an inner surface of at least one of the side windows and rear windows of the vehicle utilizing a full windshield display system.

[0022] According to another aspect, the system controller is further adapted to provide information related to the vehicle's intended route to an entity outside the vehicle by projecting the information related to the vehicle's intended route onto an inner surface of at least one of a side window of the vehicle and a rear window of the vehicle utilizing a full windshield display system.

[0023] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0025] Figure 1 is a schematic diagram of a vehicle having a system for warning an occupant of a hazardous object before the occupant exits the vehicle according to an exemplary embodiment;

[0026] Figure 2 is a schematic diagram of a system according to an exemplary embodiment;

[0027] Figure 3A is a schematic diagram of an intersection where a vehicle is about to turn left and sensors within the vehicle and infrastructure systems detect the presence of an approaching vehicle;

[0028] Figure 3B yes Figure 3A A schematic diagram of the intersection shown, wherein an approaching vehicle is classified as a threatening vehicle;

[0029] Figure 4 is a schematic diagram of an augmented reality head-up display system according to an exemplary embodiment;

[0030] Figure 5 is a schematic diagram of an inner surface of a windshield of a vehicle;

[0031] Figure 6 is a schematic diagram of a full windshield display system according to an exemplary embodiment;

[0032] Figure 7 is a schematic diagram of a substrate for a vehicle windshield according to an exemplary embodiment;

[0033] Figure 8 yes Figure 7 The enlarged part, such as Figure 7 Marked as " Figure 8 " is shown in the circle;

[0034] Fig. 9is a schematic diagram of the inner surface of a windshield of a vehicle including a do not proceed warning message, highlighting of an approaching vehicle, and a predicted path of the approaching vehicle;

[0035] Fig.10 A schematic diagram of the inside surface of a vehicle's windshield that includes cautionary messages, highlighting of approaching vehicles, and a suggested route for the vehicle.

[0036] Fig.11 is a schematic diagram of the inner surface of a windshield of a vehicle including a warning message indicating that it is OK to continue driving, a predicted route of an approaching vehicle, and a suggested route for the vehicle;

[0037] Fig.12 is a schematic diagram of the interior surface of a vehicle windshield including a warning message indicating that the vehicle should wait for an approaching vehicle to pass before continuing to move, a warning icon, a predicted route of the approaching vehicle, and a suggested route for the vehicle;

[0038] Fig.13 is a schematic top view of an intersection showing a suggested route for a vehicle without a trailer compared to a suggested route for a vehicle with a trailer;

[0039] Fig.14A is a schematic top view of a vehicle approaching a sharp right turn;

[0040] Fig. 14B yes Fig.14A A schematic top view is shown with an approaching vehicle;

[0041] Fig.15A is a perspective view of a vehicle having arrows displayed externally on the front windshield and side windows;

[0042] Fig. 15B is a perspective view of the vehicle displaying an arrow on the exterior of the rear window; and

[0043] Fig.16 is a flowchart illustrating a method of issuing a warning to an occupant in a vehicle to inform the occupant of a threatening vehicle when the vehicle is located at an intersection according to an exemplary embodiment.

[0044] The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized, for example, to show details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways. DETAILED DESCRIPTION

[0045] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. In addition, it is not intended to be constrained by any express or implied theory presented in the aforementioned technical field, background technology, summary of the invention or the following specific embodiments. It should be understood that throughout the drawings, corresponding reference numerals represent similar or corresponding components and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or groups) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0046] As used herein, the term "vehicle" is not limited to automobiles. Although the present technology is described primarily in conjunction with automobiles herein, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as in connection with aircraft, ships, other vehicles, and consumer electronic components.

[0047] According to an exemplary embodiment, Figure 1 A vehicle 10 is shown with an associated system 11 for providing a warning to an occupant 56 within the vehicle 10 when the vehicle 10 is located at an intersection to inform the occupant 56 of the presence of a threatening vehicle. Generally speaking, the system 11 works in conjunction with other systems within the vehicle 10 to display various information and infotainment content to the occupant. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a vehicle frame. The front wheels 16 and the rear wheels 18 are each rotatably connected to the chassis 12 near a respective corner of the body 14.

[0048] In various embodiments, the vehicle 10 is an autonomous vehicle, and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport occupants from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the driving mode-specific performance of the autonomous driving system in various aspects of dynamic driving tasks even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in various aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage. The novel aspects of the present disclosure are also applicable to non-autonomous vehicles.

[0049] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a wireless communication module 36. In an embodiment where the vehicle 10 is an electric vehicle, there may be no transmission system 22. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transfer power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmission. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include friction brakes, wire control brakes, regenerative braking systems (e.g., motors), and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0050] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, light detection and ranging systems (LiDAR), global positioning systems, optical cameras, thermal cameras, ultrasonic sensors, and / or other sensors. The camera may include two or more digital cameras spaced a selected distance from each other, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image or map. The plurality of sensing devices 40a-40n are used to determine information about the environment surrounding the vehicle 10. In an exemplary embodiment, the plurality of sensing devices 40a-40n include a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the plurality of sensing devices 40a-40n also include sensors for determining information about the environment surrounding the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a photographic and / or video camera positioned to observe the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40a-40n is capable of measuring distances in the environment surrounding the vehicle 10.

[0051] In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n measure distance using an image processing algorithm configured to process images from the camera and determine the distance between objects. In another non-limiting example, the plurality of vehicle sensing devices 40a-40n include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of sensing devices 40a-40n is fixed to the interior of the vehicle 10, for example, fixed in the roof lining of the vehicle 10 and visible through the windshield of the vehicle 10. In another example, at least one of the plurality of sensing devices 40a-40n is a camera fixed to the exterior of the vehicle 10 (e.g., fixed on the roof of the vehicle 10) that has a field of view of the environment surrounding the vehicle 10 and is suitable for collecting information (images) related to the environment outside the vehicle 10. It should be understood that various additional types of sensing devices, such as light detection and ranging sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42 a - 42 n that control one or more features of the vehicle 10 , such as, but not limited to, the propulsion system 20 , the transmission system 22 , the steering system 24 , and the braking system 26 .

[0052] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of multiple processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrical PROM), EEPROM (electrically erasable PROM), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data, some of which represents executable instructions used by the controller 34 in controlling the vehicle 10.

[0053] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and based on the logic, calculations, methods and / or algorithms, generate control signals to the actuator system 30 to automatically control components of the vehicle 10. Although Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals 10 that automatically control autonomous vehicle functions.

[0054] In various embodiments, one or more instructions of the vehicle controller 34 are embodied in the trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that solve the kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions implement a graph-based approach and use a customized cost function to handle different road scenarios for cities and highways.

[0055] The wireless communication module 36 is configured to wirelessly transmit information to and from other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. DSRC channels refer to one-way or two-way short-range to medium-range wireless communication channels designed specifically for automotive use and a set of corresponding protocols and standards.

[0056] The vehicle controller 34 is a non-general purpose electronic control device having a pre-programmed digital computer or processor, memory or non-transitory computer readable medium, and a transceiver [or input / output port] for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc. Computer readable media include any type of media that can be accessed by a computer, such as read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media excludes wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer readable media include media that can permanently store data and media that can store and subsequently rewrite data, such as rewritable optical disks or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0057] Figure 2 A schematic diagram of the system 11 is shown. The system 11 includes a system controller 34A in communication with a plurality of sensing devices (onboard sensors) 40a-40n. The system controller 34A may be the vehicle controller 34, or the system controller 34A may be a separate controller in communication with the vehicle controller 34. In addition to the plurality of onboard sensors 40a-40n, the system controller 34A is in communication with an occupant monitoring system 50. The system 11 also includes an augmented reality head-up display system (AR-HUD) 52 in communication with the system controller 34A and adapted to project images onto an inner surface 66 of a windshield 68 of the vehicle 10, and a full windshield display system 54 adapted to project images onto an inner surface 66 of a windshield 68 of the vehicle 10.

[0058] like Figure 3AAs shown, the vehicle 10 is located at an intersection and is preparing to turn left. In an exemplary embodiment, the system controller 34A is adapted to detect at least one approaching vehicle 60 moving toward the intersection via communication with a plurality of onboard sensors 40a-40n and wireless communication with other vehicles and infrastructure systems 58 (as shown, a camera system) via the wireless communication module 36. As shown, the system controller 34A identifies four vehicles 60A, 60B, 60C, 60D approaching the intersection.

[0059] The system controller 34A is also adapted to calculate the distance, speed, and trajectory of each identified approaching vehicle 60 from the intersection. Figure 3A For each of the four identified approaching vehicles 60A, 60B, 60C, 60D shown, the system controller calculates the distance of the approaching vehicle 60 from the intersection, the speed at which the approaching vehicle 60 is traveling, and the predicted path (trajectory) of the approaching vehicle 60. The system controller 34A can use a front camera as one of the multiple sensors 40a-40n in conjunction with a computer vision algorithm to estimate the distance and depth. Distance estimation is possible by using an optical device or by using a monocular depth estimation (MDE) algorithm. Speed ​​can be calculated by dividing the difference in the calculated distance to the intersection of two consecutive frames by the time span between the two consecutive frames. Speed ​​and trajectory information of the approaching vehicle 60 can also be obtained through radar and light detection and ranging sensors included in the multiple on-board sensors 40a-40n and through wireless communications, through the wireless communication module 36, with other vehicles, infrastructure systems 58, etc.

[0060] The system controller 34A is adapted to use the calculated information of the distance, speed, and trajectory of each approaching vehicle 60A, 60B, 60C, 60D to determine whether at least one approaching vehicle 60 is a threat, and if so, to classify at least one approaching vehicle 60 as a threat vehicle. In an exemplary embodiment, when determining whether an approaching vehicle 60 is a threat, the system controller 34A is further adapted to utilize a machine learning algorithm 62 to probabilistically calculate the probability of at least one approaching vehicle 60 colliding with the vehicle 10 if the vehicle 10 performs its planned maneuver (here, a left turn). The system controller 34A determines the planned maneuver of the vehicle 10 based on data collected from the onboard system (left turn signal on, GPS indicating a left turn).

[0061] The database 64 in communication with the system controller 34A is adapted to store data related to past events that classified the approaching vehicle 60 as a threatening vehicle, as well as environmental aspects (weather, temperature, precipitation) and vehicle aspects during such past events. The machine learning algorithm 62 is adapted to predict the probability of at least one approaching vehicle 60 colliding with the vehicle 10 based on real-time data collected by the plurality of onboard sensors 40a-40n, analysis of captured images using computer vision and image recognition techniques, and data from past occurrences in the database 64.

[0062] Various techniques are used to extract meaningful features from sensor readings and data, including time series analysis, frequency domain analysis, and spatiotemporal patterns. The machine learning algorithm 62 includes a machine learning model, which can be, but is not limited to, one of a long short-term memory (LSTM), a gated recurrent unit (GRU), a decision tree, a random forest, a support vector machine (SVM), a neural network (NN), a K nearest neighbor (KNN), a gradient boosting, and a recurrent neural network (RNN). The machine learning model is trained using data collected from multiple different vehicles. A diverse dataset is collected from vehicles equipped with sensors such as GPS, accelerometers, cameras, radars, and light detection and ranging. The data covers a variety of driving scenarios, including urban, highway, and off-road driving. Before the data is fed into the machine learning model, a preprocessing step is required to remove noise, handle missing values, and standardize features. The basic step in driving behavior classification is to extract relevant features from the raw data. As described above, various techniques are used to extract meaningful features from sensor readings, including time series analysis, frequency domain analysis, and spatiotemporal patterns. Different types of machine learning algorithms can be used for probabilistic identification of threatening vehicles. The machine learning model is trained on the labeled dataset and evaluated using various performance metrics such as accuracy, precision, recall, F1 score, and confusion matrix. The hyperparameters of the model are tuned to obtain the best results. The machine learning model is further trained on the training data and will learn to map the input features to the corresponding probabilities.

[0063] Data from events in which the system controller 34A classifies an approaching vehicle 60 as a threatening vehicle is saved to the database 64 and used to continuously update the machine learning algorithm 62. Thus, the system controller 34A uses the machine learning algorithm 62 and machine learning techniques to predict the probability of at least one approaching vehicle 60 colliding with the vehicle 10 based on analysis of real-time data collected by the plurality of onboard sensors 40a-40n and transmitted via wireless communications with other vehicles and infrastructure systems 58, analysis of captured images using computer vision and image recognition techniques, and data from past occurrences in the database 64.

[0064] The system controller 34A is adapted to classify the at least one approaching vehicle 60 as a threat vehicle when the probability of the at least one approaching vehicle 60 colliding with the vehicle 10 exceeds a first predetermined threshold. In an exemplary embodiment, the system controller 34A is adapted to classify the at least one approaching vehicle 60 as a "low risk threat vehicle" when the probability of the at least one approaching vehicle 60 colliding with the vehicle 10 exceeds a first predetermined threshold, classify the at least one approaching vehicle 60 as a "medium risk threat vehicle" when the probability of the at least one approaching vehicle 60 colliding with the vehicle 10 exceeds a second predetermined threshold, and classify the at least one approaching vehicle 60 as a "high risk threat vehicle" when the probability of the at least one approaching vehicle 60 colliding with the vehicle 10 exceeds a third predetermined threshold.

[0065] For example, refer again to Figure 3A , the first predetermined threshold is 5%, the second predetermined threshold is 25%, and the third predetermined threshold is 50%. The system controller identifies five approaching vehicles 60 and refers to Figure 3B , calculates the probability of each approaching vehicle 60 colliding with the vehicle 10 if the vehicle 10 continues the planned left turn. The system controller 34A calculates that the first approaching vehicle 60A has a 26% probability of colliding with the vehicle 10, and therefore classifies the first approaching vehicle 60A as a medium risk threat vehicle. The system controller 34A calculates that the second approaching vehicle 60B has a 60% probability of colliding with the vehicle 10, and therefore classifies the second approaching vehicle 60B as a high risk threat vehicle. The system controller 34A calculates that the third approaching vehicle 60C has a 30% probability of colliding with the vehicle 10, and therefore classifies the third approaching vehicle 60B as a medium risk threat vehicle. The system controller 34A calculates that there is a 6% probability that the fourth approaching vehicle 60D will collide with the vehicle 10, and therefore classifies the fourth approaching vehicle 60B as a medium risk threat vehicle. The system controller 34A calculates that the fifth approaching vehicle 60E has a 4% probability of colliding with the vehicle 10, and therefore does not classify the fifth approaching vehicle 60E as a threat vehicle.

[0066] For each identified approaching vehicle 60 classified as a threat vehicle (60A, 60B, 60C, 60D) by the system controller 34A, the system controller 34A is also adapted to provide a warning message 70 and an enhanced graphic 72 on the inner surface 66 of the windshield 68 of the vehicle by projecting at least one of the warning message 70 and the enhanced graphic 72 onto the inner surface 66 of the windshield 68 utilizing the AR-HUD system 52, and projecting the warning message 70 and the enhanced graphic 72 onto the inner surface 66 of the windshield 68 utilizing the full windshield display system 54.

[0067] refer to Figure 4In an exemplary embodiment, the AR-HUD system 52 includes a projection module 74 that includes at least one light source 76 that is suitable for projecting an image onto the inner surface 66 of the windshield 68 of the vehicle 10. As described herein, the at least one light source 76 includes a laser, however, it should be understood that the at least one light source 76 can be other known types of light sources used in head-up display systems. In an exemplary embodiment, the projection module 74 is a holographic projection module and includes an exit pupil replicator. The holographic image is projected into the exit pupil replicator, then propagates within the exit pupil replicator and is extracted multiple times before being projected upward onto the inner surface 66 of the windshield 68. The multiple recycling of light within the exit pupil replicator expands the pupil so that the observer can see the holographic image from the expanded eye box.

[0068] The spatial light modulator is located between the projection module 74 and the exit pupil replicator. The spatial light modulator is adapted to receive light from the laser 76, diffract the laser light with the encoded hologram, and transmit the diffracted laser light to the exit pupil replicator. As shown and described herein, the head-up system is a holographic head-up display system, however, it should be understood that the novel features of the present disclosure can be applied to other head-up display configurations.

[0069] The driver monitoring system 50 is adapted to track the position of the driver's eyes 82. The plurality of onboard sensors 40a-40n include at least one non-visual sensor 78 adapted to detect objects within the environment surrounding the vehicle 10, such as an approaching vehicle 60, and at least one image capture device 80 adapted to capture images of the environment surrounding the vehicle 10.

[0070] The driver monitoring system 50 includes one or more sensors, such as cameras, for determining the position of the head of the driver of the vehicle 10 and the direction or gaze position of the driver's eyes 82. The at least one non-visual sensor 78 includes one or more sensors that are suitable for detecting the position, speed, and direction of travel of objects in the environment surrounding the vehicle 10. As non-limiting examples, the at least one non-visual sensor 78 may include one or more sensors, such as radar, light detection and ranging sensors, and infrared sensors, however, it should be understood that other sensors that use non-visual technology to detect the presence of objects may also be used. At least one image capture device 80 may include a camera that obtains periodic or sequential images of the environment surrounding the vehicle 10 and the objects therein.

[0071] The system controller 34A is in electronic communication with at least one laser 76, at least one non-visual sensor 78, and at least one image capture device 80. The system controller 34A uses the at least one non-visual sensor 78 and the at least one image capture device 80 to identify the approaching vehicle 60. The at least one laser 76 of the holographic projection module 74 is adapted to project an image upwardly onto the inner surface 66 of the windshield 68 of the vehicle 10, as indicated by arrow 84. Figure 5 , the projection module 74 is capable of projecting an image located in a near image plane 86 as viewed by an occupant within the vehicle 10 , and projecting an image located in a far image plane 88 as viewed by the driver. Figure 5 is an exemplary interior view of windshield 68 , wherein the environment surrounding vehicle 10 is visible through windshield 68 .

[0072] The system controller 34A instructs the projection module 74 of the augmented reality head-up display system 52 to project cluster content information 90 onto the windshield 18 within the near-field image plane 44. The cluster content information 90 informs the driver of the vehicle 10 of driving conditions, such as, but not limited to, vehicle speed, speed limit, gear, fuel level, current location, and navigation instructions. Figure 5 In the example shown, cluster content information 90 includes vehicle speed and navigation direction. System controller 34A instructs projection module 74 to project warning message 70 and enhanced graphics 72 onto windshield 68 within far image plane 88, wherein warning message 70 and / or enhanced graphics 72 are superimposed on windshield 68 at a location where warning message 70 and / or enhanced graphics 72 appear to the driver to be located on a road surface 92 in front of vehicle 10. Far image plane 88 contains an image overlaid on road 92 that is visible through windshield 68. Figure 5 In the illustrated embodiment, the far image plane 88 covers only a portion of the entire plane of the windshield 68 , however, it should be understood that in another embodiment, the far image plane 88 may cover the entire plane of the windshield 68 not occupied by the near image plane 86 .

[0073] refer to Figure 6 In an exemplary embodiment, the full windshield display system 54 includes a mixed augmented reality head-up display system in communication with the system controller 34A. Other options are suitable for the full windshield display 54, such as, as non-limiting examples, black area Pepper's ghost display, fiber-based projection system, micro-LED transparent display, and holographic transparent display. The mixed augmented reality head-up display of the exemplary embodiment described herein includes a digital light projector (DLP) 94, which is located in the vehicle 10 and is suitable for projecting warning messages 70 and / or augmented graphics 72 onto the inner surface 66 of the windshield 68 of the vehicle 10.

[0074] The DLP 94 is capable of projecting two-dimensional and three-dimensional images depicting objects such as pedestrians, buildings, and other vehicles in the environment surrounding the vehicle 10. In an exemplary embodiment, the DLP 94 includes a light source 96 adapted to project excitation light 98, a focusing lens 100 adapted to focus the excitation light 98 from the light source 96, a color filter 102 (color wheel) adapted to separate the focused excitation light 98 into red, green, and blue light, a shaping lens 104 adapted to focus the excitation light 98 that passes through the color filter 102, a digital micromirror device (DMD) 106 adapted to redirect the excitation light 98; and a projection lens 108 adapted to receive the excitation light 98 from the digital micromirror device (DMD) 106 and project the excitation light 98 onto the inner surface 66 of the windshield 68.

[0075] The system controller 34A includes a DLP engine 110. The DLP engine 110 includes a programmable processor that includes programming to monitor various inputs and determine what information is suitable for display. The DLP engine 110 can communicate directly with various systems and components, or the DLP engine 110 can communicate via a LAN / CAN system instead or in addition. The DLP engine 110 utilizes information about the operating environment of the vehicle 10 derived from multiple inputs. Various sensing devices 40a-40n collect and provide information. The sensing devices 40a-40n include cameras or image capture devices that capture periodic or sequential images representing views of the vehicle 10. The windshield 68 is equipped with features that can display images projected thereon while remaining transparent or substantially transparent, so that the occupants 56 of the vehicle 10 can clearly observe the exterior of the vehicle 10 through the windshield 68. It should be understood that, as shown, the windshield 68 is located at the front of the vehicle 10, and other surfaces within the vehicle 10 can also be used for projection, including side windows and rear windows. Additionally, the field of view on the front windshield 68 may continue as a continuous image onto the front vehicle "A-pillars" and onto the side windows.

[0076] The DLP engine 110 includes display software or programming to translate requests to display information from the DLP engine 110 in the form of a graphical representation describing the information. The DLP engine 110 includes programming to compensate for the curved and tilted surface of the windshield 68 and any other surface onto which the warning message 70 and / or enhanced graphics 72 are to be projected. The DLP engine 110 controls the light source 96, which includes a laser or projector device that produces the excitation light 98 to project the warning message 70 and / or enhanced graphics 72.

[0077] The occupant monitoring system 50 includes sensors known in the art to estimate the head position of the occupant (driver) 56 and further estimate the eye direction or gaze position of the occupant 56. Based on the output of the occupant monitoring system 50 and the input data about the position information of the environment surrounding the tracking vehicle 10, the DLP engine 110 can accurately position the warning message 70 and / or enhanced graphics 72 so that the occupant 56 sees the warning message 70 and / or enhanced graphics 72 overlaid with a visual image through the windshield 68.

[0078] The occupant monitoring system 50 includes eye sensing and head sensing devices that allow for estimation of eye position, allowing for registration of an image on the windshield 56 so that the image corresponds to the field of view of the occupant 56. In addition, the occupant monitoring system 50 allows the system 11 to locate the position of the occupant's eye ellipse 112. The eye ellipse 112 is a three-dimensional graphical depiction of a multivariate normal distribution that is used to model the distribution of eye positions of the occupant 56 within the vehicle 10. The eye ellipse 112 is represented by two three-dimensional ellipses, one for the right eye and one for the left eye.

[0079] It should be understood that estimation of head and eye position can be achieved by a variety of methods. For example, in a process similar to adjusting a rearview mirror, the operator can use a calibration routine upon entering the vehicle to align the graphic with the detected object. In another embodiment, the longitudinal seat position in the vehicle 10 can be used to estimate the position of the driver's head. In another embodiment, manual adjustment of one or more rearview mirrors can be used to estimate the position of the operator's eyes. It should be understood that a combination of methods such as seat position and mirror adjustment angle can be utilized to estimate the operator's head position with improved accuracy. Many methods are contemplated to achieve precise registration of the graphic on the windshield 68 with the position of the occupant's eye ellipse 112, and the present disclosure is not intended to be limited to the specific embodiments described herein.

[0080] The windshield 68 serves as a medium through which relevant features can be viewed, and also as a display device on which warning messages 70 and / or enhanced graphics 72 can be displayed. The windshield 68 is both transparent and capable of displaying the image projected by the activation light 98. Figure 7 and Figure 8 , the occupant 56 is able to see any object (e.g., cube 114) through the substrate 116 located on the windshield 68. The substrate 116 can be transparent or substantially transparent. When the occupant 56 sees any object 114 through the substrate 116, the occupant 56 can also see the image created at the substrate 116 (e.g., circle 118 and triangle 120). The substrate 116 can be a portion of the windshield 68, a glass substrate, a plastic substrate, a polymer substrate, or other transparent (or substantially transparent) medium that will be understood by those of ordinary skill in the art. Other substrates can supplement the substrate 116 to provide coloring, substrate protection, light filtering (e.g., filtering external ultraviolet light), and other functions.

[0081] Figure 8 The illumination of a transparent display illuminated with excitation light 98 (e.g., ultraviolet light) from a light source 96 is depicted. The substrate 116 receives the excitation light 98 from the light source 96. The received excitation light 98 may be absorbed by the luminescent material 122 at the substrate 116. When the luminescent material 122 receives the excitation light 98, the luminescent material 122 emits visible light. Thus, an image (e.g., a circle 118 and a triangle 120) may be generated at the substrate 116 by selectively illuminating the substrate 116 with the excitation light 98.

[0082] In an exemplary embodiment, the luminescent material 122 includes a transparent phosphor embedded in the substrate 116. The transparent phosphor is a luminescent particle that emits fluorescence in response to excitation by the excitation light 98. In an exemplary embodiment, the transparent phosphor is a red, green and blue (RGB) phosphor that allows full color. The use of monochromatic and / or dual-color phosphors is also within the scope of the present invention. When the excitation light 98 is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light 98 is provided by the light source 96. The use of substrate 116 and luminescent material 122 to display graphics is discussed in more detail in U.S. application No. 17 / 749,464, entitled “HYBRID AUGMENTED REALITY HEAD-UP DISPLAY FOR CREATING AN EDGE-TO-EDGE AUGMENTED REALITY VIEW” filed on May 20, 2022, the entire contents of which are now incorporated herein by reference.

[0083] According to an embodiment of the present disclosure, the excitation light 98 may be ultraviolet light. If the excitation light 98 is ultraviolet light, when the luminescent material 122 emits visible light in response to the ultraviolet light, a down-conversion physical phenomenon occurs. Specifically, ultraviolet light has a shorter wavelength and higher energy than visible light. Therefore, when the luminescent material 122 absorbs ultraviolet light and emits lower energy visible light, the ultraviolet light is down-converted into visible light because its energy level decreases when the ultraviolet light is converted into visible light.

[0084] like Figure 6As shown, the excitation light 98 is output by the light source 96 of the digital light processing projector (DLP) 94. In an exemplary embodiment, the light source 96 is a micromirror array (MMA) projector (e.g., a digital light processing (DLP) projector 94). In the DLP 94, the warning message 70 and the enhanced graphics 72 are generated by the tiny mirrors arranged in a matrix on the semiconductor chip in the DMD 106. The MMA projector that outputs ultraviolet light can be similar to the MMA projector that outputs visible light, except that the color filter 102 has a filter specifically for the ultraviolet spectrum. In other embodiments, the DLP 94 is a liquid crystal display (LCD) projector. In some embodiments, the DLP 94 can be a liquid crystal on silicon (LCOS) projector. In an embodiment, the DLP 94 can be an analog projector (e.g., a slide projector or a movie film projector). Those of ordinary skill in the art will understand other types of projectors that can be used to project ultraviolet light onto the substrate 116.

[0085] In an exemplary embodiment, when providing the warning message 70 and the augmented graphic 72 to the interior surface 66 of the windshield 68 of the vehicle 10, the system controller 34A is also adapted to determine what warning message 70 and augmented graphic 72 to provide, and to determine whether the warning message 70 and augmented graphic 72 are within the field of view (FOV) of the AR-HUD system 52. Referring again to Figure 5 , the FOV of the AR-HUD system 52 is the area in which the AR-HUD system 52 can display images. Figure 5 In the example shown, the FOV of the far image plane 88 is limited to the area shown by 88. In this example, the AR-HUD system 52 is only able to provide images within the area indicated by the dashed line of the far image plane 88.

[0086] If the size and properties of the warning message 70 and the augmented graphics 72 allow the warning message 70 and the augmented graphics 72 to be within the FOV of the AR-HUD system 72, the system controller 34A is adapted to project the warning message 70 and the augmented graphics 72 onto the inner surface 66 of the windshield 68 via the AR-HUD system 52. If the size and properties of the warning message 70 and the augmented graphics 72 are such that the warning message 70 and the augmented graphics 72 are not within the FOV of the AR-HUD system 52, the system controller 34A is adapted to project the warning message 70 and the augmented graphics 72 onto the inner surface 66 of the windshield 68 via the full windshield display system 54, because the full windshield display system 54 essentially has an infinite FOV and can project images onto the entire windshield 68, the side windows 124, or the rear window 126.

[0087] When providing the warning message 70 and the enhanced graphics 72 to the interior surface 66 of the windshield 68 of the vehicle 10, the system controller 34A is further adapted to provide the warning message 70 including text information and to provide the enhanced graphics 72 including at least one of a trajectory arrow, a highlighted threatening vehicle, and a warning icon. Fig. 9 , the vehicle 10 is located at the intersection, and based on the navigation settings, the system controller 34A knows that the vehicle 10 intends to go straight through the intersection. The system controller 34A identifies the approaching vehicle 60 and calculates based on the speed and trajectory of the approaching vehicle 60 that if the vehicle 10 proceeds through the intersection at this moment, there is a greater than 50% chance (a third predetermined threshold) that the approaching vehicle 60 will collide with the vehicle 10. Therefore, the system controller 34A provides a text warning message 70 that means "DO NOT PROCEED!" and provides an enhanced graphic 72 that includes a warning icon next to an arrow 72A indicating the intended route of the vehicle 10. In addition, the enhanced graphic 72 includes a highlight 72B of the approaching vehicle 60 to draw the attention of the occupant 56 to the approaching vehicle 60.

[0088] In an exemplary embodiment, when determining what warning message 70 and enhanced graphic 72 to provide, the system controller 34A is further adapted to determine what warning message 70 and enhanced graphic 72 to provide for each of the at least one approaching vehicle 60 that is classified as a threat vehicle based on each threat vehicle being classified as one of a low risk threat vehicle, a medium risk threat vehicle, or a high risk threat vehicle. For example, Fig. 9 As shown, the calculated collision probability is greater than 50%, so the system controller 34A classifies the approaching vehicle 60 as a high risk threat vehicle. Fig. 9 In addition, the text message 70 and enhanced graphics 72, 72A, 72B for high-risk threat vehicles can be provided in a color such as red to further convey to the occupants that there is a high risk of continuing to move forward.

[0089] refer to Fig.10 In another example, the calculated collision probability is less than 10%, so the system controller 34A classifies the approaching vehicle 60 as a low risk threat vehicle. This may be due to circumstances such as the approaching vehicle 60 is still far from the intersection and moving very slowly, or the approaching vehicle 60 may be stopped at the intersection waiting to turn left and the vehicle 10 has the right of way. Fig.10 The text warning message 70 "Drive Carefully" and the enhanced graphics 72, 72C, 72D in the low-risk threat vehicle may be provided in a color such as yellow to convey to the occupants that there is a risk to continue, but the enhanced graphics 72, 72C, 72D are less effective. Fig. 9 The red message 70 and enhanced graphics 72 , 72A, 72B shown for high risk threat vehicles are less impactful.

[0090] refer to Fig.11 In another example, the calculated collision probability is less than 2%, so the system controller 34A does not classify the approaching vehicle 60 as a threatening vehicle. Fig.11 The text warning message 70 and enhanced graphics 72 in the embodiment include the message "OK TO TURN RIGHT" 70A, and the enhanced graphics 72 include a planned / suggested route arrow 72E and a predicted approaching vehicle route arrow 72F, indicating that it is safe for the vehicle 10 to make the planned right turn because the approaching vehicle 60 does not pose a risk to the maneuver. In addition, the text message 70A and enhanced graphics 72E, 72F provided herein may be provided in a color such as green to convey to the occupant 56 that there is little or no risk to proceed. In other embodiments, when no threatening vehicle is identified, the system controller 34A simply maintains the displayed planned / suggested route arrow 72E, or changes the color of the displayed planned / suggested route arrow 72E to green, and does not provide an explicit message instructing the driver to perform an action.

[0091] The system controller 34A will also take into account when the vehicle has additional length, such as a semi-truck trailer combination or a vehicle 10 towing a trailer 128. Fig.12 , in the Fig.11 In the same scenario shown, when the approaching vehicle 60 in the rightmost lane approaches from the right, the system controller 34A does not identify the approaching vehicle 60 as a threatening vehicle because the predicted path of the vehicle 10 (eg, Fig.13 130 in the figure) will not intersect the predicted path 132 of the approaching vehicle 60. However, if the vehicle 10 is towing a trailer 128, the vehicle 10 will need to make a wide right turn due to the extra length, such as Fig.13 134 in the figure, which may intersect the predicted path 132 of the approaching vehicle 60. In this case, for the planned maneuver of the vehicle 10 towing the trailer 128, the system controller 34A identifies and classifies the approaching vehicle 60 as a high risk threat vehicle and provides a text warning message 70C "Wait for the vehicle to pass" and enhanced graphics 72, including a warning icon next to the right turn arrow 72G, and highlighting 72H the approaching vehicle 60 and the predicted path 72I of the approaching vehicle 60. In addition, the text message 70C and enhanced graphics 72G, 72H, 72I for the high risk threat vehicle can be provided in a color such as red to further convey to the occupant 56 that there is a high risk to continue.

[0092] Reference again Fig.11 and Fig.12In another exemplary embodiment, when determining what warning message 70 and enhanced graphics 72 to provide, the system controller 34A is further adapted to utilize the machine learning algorithm 62 to predict the route 72F, 72I of each of at least one approaching vehicle classified as a threatening vehicle, and when providing the warning message 70 and enhanced graphics 72, the system controller 34A is further adapted to utilize the full windshield display system 54 to project the enhanced graphics 72 including the predicted route 72F, 72I for each of at least one approaching vehicle 60 classified as a threatening vehicle onto the inner surface 66 of the windshield 68 of the vehicle 10.

[0093] In another exemplary embodiment, when determining what warning message 70 and enhanced graphics 72 to provide, the system controller 34A is further adapted to utilize a machine learning algorithm to calculate a suggested route 72E for the vehicle 10. The suggested route provides a route based on the nature and environment of the road and the presence of any approaching vehicles 60, wherein if there are any approaching vehicles 60, then if the vehicle 10 follows the suggested route 72E, then the probability of any of the at least one approaching vehicle 60 colliding with the vehicle 10 is less than a first predetermined threshold, and when providing the warning message 70 and enhanced graphics 72 onto the inner surface 66 of the windshield 68 of the vehicle 10, the system controller 34A is further adapted to project the enhanced graphics 72 including the suggested route 72E for the vehicle 10 onto the inner surface 66 of the windshield 68 of the vehicle.

[0094] For example, refer to Fig.14A , the vehicle 10 approaches a sharp right turn, wherein many times, it is easy for a vehicle making such a sharp turn to miscalculate the steering required for the maneuver and hit the edge of the road. The system controller 34A using crowdsourced telemetry information and data from the database 64 / machine learning algorithm 62 can plan a nominal path 136 that will cause the vehicle 10 to travel on a sufficiently wide turning path to avoid hitting the edge of the road and provide a warning message 70 and / or enhanced graphics 72 to the occupants 56 in the vehicle 10 to illustrate the nominal path 136. And when the system controller 34A detects that the vehicle 10 deviates from the nominal path 136, the warning message 70 and / or enhanced graphics 72 are provided. Reference Fig. 14B , the presence of an approaching vehicle 60 that may be classified as a threat vehicle 60 will cause the system controller 34A to change the nominal path to reduce the threat risk. The vehicle 10 and the possible threat vehicle 60 may receive a warning message 70 of the possible conflict, as well as an enhanced graphic 72 showing the altered route 138 to reduce the risk of conflict.

[0095] Referring again to the above scenario, in which vehicle 10 is towing trailer 128. Fig.14A and Fig. 14BIn the illustrated sharp turn, given the additional length of the vehicle 10 and trailer 128, the system controller 34A may determine that the vehicle 10 and trailer 128 can only successfully make the sharp turn by taking a very wide radius path, and such a path is not possible in the presence of the approaching vehicle 60. In this case, rather than providing an altered route 138, the system controller 34A provides a warning message 70 indicating that the vehicle 10 should stop and wait until the approaching vehicle 60 has passed before making the right sharp turn.

[0096] In another exemplary embodiment, the system controller 34A is further adapted to provide the secondary warning message 170 and the secondary enhanced graphics 172 onto the inner surface of at least one of the side windows 124 of the vehicle 10 and the rear window 126 of the vehicle. The secondary warning message 170 and the secondary enhanced graphics 172 are projected onto the inner surface of at least one of the side windows 124 of the vehicle 10 and the rear window 126 of the vehicle 10 by utilizing the full windshield display system 54. The full windshield display system 54 described above may be used to activate phosphors within the side windows 124 and the rear window 126 of the vehicle 10, which are visible to entities external to the vehicle 10 and to the occupants 56 within the vehicle 10. Thus, the full windshield display system 54 may be used to present the warning message 170 and / or the enhanced graphics 172 to nearby or adjacent vehicles to convey information related to the intended route of the vehicle 10.

[0097] refer to Fig.15A , the full windshield display system 54 presents arrows 172A on the front windshield 68 and side windows 124 to provide information about the intended route of the vehicle 10 in potentially confusing situations, such as at busy intersections or detours and when there is no line of sight of approaching vehicles 60, wherein information about such approaching vehicles 60 is provided via wireless communications with infrastructure and other vehicles. Fig. 15B , the full windshield display system 54 presents arrows 170 , 172 on the rear window 126 to provide backup guidance or threat awareness to nearby vehicles.

[0098] The system controller 34A includes a graphics processor that processes information from the system controller 34A and determines appropriate warning messages 70, 170 and / or enhanced graphics 72, 172 to overlay on the image on the windshield 68, side windows 124 or rear window 126 for display to the occupant 56 or external entities.

[0099] refer to Fig.16The method 200 for providing a warning to an occupant 56 in the vehicle 10 when the vehicle 10 is located at an intersection to inform the occupant 56 of the presence of a threatening vehicle 60 includes: starting at box 202, using a system controller 34A in communication with a plurality of onboard sensors 40a-40n, detecting wireless communications with other vehicles and wireless communications with an infrastructure system 58, at least one approaching vehicle 60 moving toward the intersection, moving to box 204, using the system controller 34A to calculate a distance to the intersection, a speed and a trajectory of the at least one approaching vehicle 60, moving to box 206, using the system controller 34A to determine the at least one approaching vehicle 60 Whether it constitutes a threat, and if at least one approaching vehicle 60 constitutes a threat, at least one approaching vehicle 60 is classified as a threatening vehicle, and moves to box 207, providing a warning message 70 and an enhanced graphic 72 to the inner surface 66 of the windshield 68 of the vehicle 10 by at least one of the following methods: moving to box 208, projecting the warning message 70 and the enhanced graphic 72 onto the inner surface 66 of the windshield 68 using an augmented reality head-up display system (AR-HUD) 52, and moving to box 210, projecting the warning message 70 and the enhanced graphic 72 onto the inner surface 66 of the windshield 68 using the full windshield display system 54.

[0100] In an exemplary embodiment, providing the warning message 70 and the enhanced graphics 72 onto the interior surface 66 of the windshield 68 of the vehicle 10 further includes: moving to box 212; determining, using the system controller 34A, the warning message 70 and the enhanced graphics 72 to be provided; moving to box 214; determining, using the system controller 34A, whether the warning message 70 and the enhanced graphics 72 will be within the field of view (FOV) 88 of the AR-HUD system 52; and moving to box 208, when the warning message 70 and the enhanced graphics 72 are within the FOV 88 of the AR-HUD system 52, projecting the warning message 70 and the enhanced graphics 72 onto the interior surface 66 of the windshield 68 using the AR-HUD system 52, and, moving to box 210, when the warning message 70 and the enhanced graphics 72 are not within the FOV 88 of the AR-HUD system 52, projecting the warning message 70 and the enhanced graphics 72 onto the interior surface 66 of the windshield 68 using the full windshield display system 54.

[0101] In another exemplary embodiment, determining whether at least one approaching vehicle 60 poses a threat using the system controller 34A, and if at least one approaching vehicle 60 poses a threat, classifying at least one approaching vehicle 60 as a threat vehicle at box 206 also includes: moving to box 216, calculating the probability of at least one approaching vehicle 60 colliding with the vehicle 10 using the machine learning algorithm 62, and moving to box 220, identifying the vehicle 60 as a threat vehicle when the probability of at least one approaching vehicle 60 colliding with the vehicle 10 exceeds a first predetermined threshold at box 218, and moving to box 222, not classifying the approaching vehicle 60 as a threat vehicle if the probability of at least one approaching vehicle 60 colliding with the vehicle 10 does not exceed the first predetermined threshold at box 218.

[0102] In an exemplary embodiment, at box 207, providing the warning message 70 and enhanced graphics 72 to the inner surface 66 of the windshield 68 of the vehicle 10 also includes: providing the warning message 70 including text information, and providing the enhanced graphics 72 including at least one of a trajectory arrow, highlighting of the threatening vehicle, and a warning icon.

[0103] In an exemplary embodiment, at box 212, determining, using the system controller 34A, the warning message 70 and enhanced graphics 72 to be provided further includes: moving to box 224, predicting, using the machine learning algorithm 62, a path for each of the at least one approaching vehicle 60 that is classified as a threat vehicle. And at box 210, using the full windshield display system 54, providing the warning message 70 and enhanced graphics 72 onto the interior surface 66 of the windshield 68 of the vehicle 10 further includes: using the full windshield display system 54, projecting the enhanced graphics 72 onto the interior surface 66 of the windshield 68 of the vehicle 10, the enhanced graphics 72 including the predicted path for each of the at least one approaching vehicle 60 that is classified as a threat vehicle.

[0104] In another exemplary embodiment, at box 212, determining, using the system controller 34A, the warning message 70 and enhanced graphics 72 to be provided further includes: moving to box 226, calculating, using the system controller 34A and the machine learning algorithm 62, a suggested route 72E for the vehicle 10, wherein if the vehicle 10 follows the suggested route 72E, the probability of any of the at least one approaching vehicle 60 classified as a threat vehicle colliding with the vehicle 10 is less than a first preset threshold. At box 210, providing, using the full windshield display system 54, the warning message 70 and enhanced graphics 72 onto the interior surface 66 of the windshield 68 of the vehicle 10 further includes: projecting, using the full windshield display system 54, the enhanced graphics 72 including the suggested route 72E for the vehicle 10 onto the interior surface 66 of the windshield 68 of the vehicle 10.

[0105] In an exemplary embodiment, when the probability of at least one approaching vehicle 10 colliding with vehicle 10 exceeds a first predetermined threshold at box 218, classifying at least one approaching vehicle 60 as a threat vehicle at box 220 further includes: moving to box 230, when the probability of at least one approaching vehicle 60 colliding with vehicle 10 exceeds the first predetermined threshold at box 218 and does not exceed a second predetermined threshold at box 228, classifying at least one approaching vehicle 10 as a low risk threat vehicle, moving to box 234, when the probability of at least one approaching vehicle 60 colliding with vehicle 10 exceeds the second predetermined threshold at box 228 and does not exceed a third predetermined threshold at box 232, classifying at least one approaching vehicle 60 as a medium risk threat vehicle, and, moving to box 236, when the probability of at least one approaching vehicle 60 colliding with vehicle 10 exceeds the third predetermined threshold at box 232, classifying at least one approaching vehicle 60 as a high risk threat vehicle.

[0106] In an exemplary embodiment, at box 212, determining, using the system controller 34A, the warning message 70 and enhanced graphics 72 to be provided also includes: determining, using the system controller 34A, based on the classification of each threat vehicle as one of a low risk threat vehicle, a medium risk threat vehicle, or a high risk threat vehicle, the warning message 70 and enhanced graphics 72 to be provided for each of the at least one approaching vehicle 60 classified as a threat vehicle.

[0107] In an exemplary embodiment, the method 200 also includes moving from frame 212 to frame 238 to provide the secondary warning message 170 and the secondary enhanced graphic 172 to the inner surface of at least one of the side windows 124 of the vehicle 10 and the rear window 126 of the vehicle 10 by utilizing the full windshield display system 54 to project the secondary warning message 170 and the secondary enhanced graphic 172 onto the inner surface of at least one of the side windows 124 of the vehicle 10 and the rear window 126 of the vehicle 10.

[0108] In another exemplary embodiment, the method 200 also includes moving from frame 238 to frame 240 to provide information related to the scheduled route of the vehicle 10 to an entity outside the vehicle 10 by utilizing the full windshield display system 54 to project the information related to the scheduled route of the vehicle 10 onto the inner surface of at least one of the side windows 124 of the vehicle 10 and the rear window 126 of the vehicle 10.

[0109] The disclosed system and method have the following advantages: when the vehicle is located at an intersection, an occupant in the vehicle is warned of the presence of a threatening vehicle and guidance is provided on continuing with a planned maneuver based on the risk level of the identified threatening vehicle.

[0110] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to be within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A method for issuing a warning to an occupant in a vehicle when the vehicle is at an intersection, for informing the occupant of the presence of a threatening vehicle, comprising: detecting, using a system controller in communication with at least one of a plurality of onboard sensors, in wireless communication with other vehicles, and in wireless communication with infrastructure systems, at least one approaching vehicle moving toward the intersection; calculating, using the system controller, a distance of the at least one approaching vehicle from the intersection, a speed and a trajectory of the at least one approaching vehicle; determining, with the system controller, whether the at least one approaching vehicle poses a threat, and if the at least one approaching vehicle poses a threat, classifying the at least one approaching vehicle as a threatening vehicle; The warning message and enhanced graphics are provided on the inner surface of the windshield of the vehicle by at least one of: Projecting the warning message and the augmented graphics onto an inner surface of the windshield using an augmented reality head-up display system (AR-HUD); Changing the color of the route guidance graphic currently displayed by the AR-HUD; as well as The warning message and the enhanced graphics are projected onto an inner surface of the windshield using a full windshield display system.

2. The method according to claim 1, wherein: Providing the warning message and the enhanced graphics on the interior surface of the windshield of the vehicle further comprises: determining, using the system controller, warning messages and enhanced graphics to be provided; Determining, using the system controller, whether the warning message and the augmented graphics are within the field of view (FOV) of the AR-HUD system; When the warning message and the augmented graphics are within the FOV of the AR-HUD system, projecting the warning message and the augmented graphics onto the inner surface of the windshield using the AR-HUD system; and When the warning message and the augmented graphics are not within the FOV of the AR-HUD system, the warning message and the augmented graphics are projected onto the inner surface of the windshield using the full windshield display system.

3. The method according to claim 2, wherein: Determining, with the system controller, whether the at least one approaching vehicle poses a threat, and if the at least one approaching vehicle poses a threat, classifying the at least one approaching vehicle as a threat vehicle further comprises: calculating, using a machine learning algorithm, a probability that the at least one approaching vehicle will collide with the vehicle; and The at least one approaching vehicle is classified as a threatening vehicle when a probability that the at least one approaching vehicle will collide with the vehicle exceeds a first predetermined threshold.

4. The method according to claim 3, wherein: Providing the warning message and the enhanced graphics on the interior surface of the windshield of the vehicle further comprises: providing a warning message including text information; and Provide enhanced graphics including at least one of the following: Track arrows; Highlighting threat vehicles; and Warning icon.

5. The method according to claim 3, wherein: Determining, using the system controller, the warning message and enhanced graphics to provide further comprises: using the machine learning algorithm to predict a path of each of the at least one approaching vehicle classified as a threatening vehicle; Providing the warning message and enhanced graphics onto the inner surface of the windshield of the vehicle also includes: projecting the enhanced graphics onto the inner surface of the windshield of the vehicle using the full windshield display system, the enhanced graphics including the predicted path of each of the at least one approaching vehicle classified as a threat vehicle.

6. The method according to claim 3, wherein: Determining, using the system controller, the warning message and enhanced graphics to provide further comprises: calculating, using the system controller and the machine learning algorithm, a suggested route for the vehicle, wherein if the vehicle follows the suggested route, the probability that any of the at least one approaching vehicle classified as a threat vehicle will collide with the vehicle is less than the predetermined threshold; and Providing the warning message and enhanced graphics onto the inner surface of the windshield of the vehicle further comprises projecting the enhanced graphics including the suggested route of the vehicle onto the inner surface of the windshield of the vehicle using the full windshield display system.

7. The method according to claim 3, wherein: When the probability that the at least one approaching vehicle collides with the vehicle exceeds the first predetermined threshold, classifying the at least one approaching vehicle as a threatening vehicle further comprises: classifying the at least one approaching vehicle as a low-risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds the first predetermined threshold; classifying the at least one approaching vehicle as a medium risk threat vehicle when the probability of the at least one approaching vehicle colliding with the vehicle exceeds a second predetermined threshold; and When the probability that the at least one approaching vehicle collides with the vehicle exceeds a third predetermined threshold, the at least one approaching vehicle is classified as a high risk threat vehicle.

8. The method according to claim 7, wherein: Determining, with the system controller, the warning message and the enhanced graphic to be provided further comprises determining, with the system controller, the warning message and the enhanced graphic to be provided for each of the at least one approaching vehicle classified as a threat vehicle based on classifying each threat vehicle as one of a low risk threat vehicle, a medium risk threat vehicle, or a high risk threat vehicle.

9. The method according to claim 3, wherein: It also includes projecting the secondary warning message and the secondary enhanced graphic onto the inner surface of at least one of the side windows and the rear window of the vehicle by utilizing the full windshield display system, thereby providing the secondary warning message and the secondary enhanced graphic on the inner surface of at least one of the side windows and the rear window of the vehicle.

10. The method according to claim 9, wherein: It also includes providing information related to the vehicle's intended route to entities outside the vehicle by utilizing the full windshield display system to project the information related to the vehicle's intended route onto the inner surface of at least one of the vehicle's side windows and the vehicle's rear window.

Images