Reminding vehicle occupant with autonomous vehicle action

Abstract

A system for providing information to a vehicle occupant of a vehicle includes an automated driving system, a first display configured to provide information to the vehicle occupant of the vehicle, and a controller in electrical communication with the automated driving system and the first display. The controller is programmed to identify a planned change in driving behavior initiated by the automated driving system. The controller is also programmed to identify traffic sign interpretation errors using the automated driving system. The controller is further programmed to notify, using the first display, a vehicle occupant of a plan change in driving behavior and a traffic sign interpretation error in response to identifying at least one of: a plan change in driving behavior and a traffic sign interpretation error.

Description

Technical Field

[0001] The present disclosure relates to an augmented reality heads-up display for displaying information to vehicle occupants. Background Art

[0002] Augmented reality (AR) involves augmenting the real world with virtual elements that are displayed in three-dimensional space and allow real-time interaction with the user. A head-up display (HUD) displays information such as vehicle speed and navigation instructions directly on the vehicle's windshield within the occupant's forward field of view. The head-up display thus provides information to the occupant without the occupant having to take their eyes off the road. One possible implementation of augmented reality is an augmented reality head-up display (AR-HUD) for vehicles. By overlaying images on the windshield, the AR-HUD enhances the occupant's view of the environment outside the vehicle, increasing situational awareness. However, current AR-HUDs may not provide occupants with additional information about the operation of vehicle systems and components.

[0003] Therefore, while current augmented reality heads-up displays achieve their intended purpose, there is a need in the art for an improved method of providing information to vehicle occupants. Summary of the invention

[0004] According to several aspects, a system for providing information to a vehicle occupant of a vehicle is provided. The system may include an automated driving system, a first display configured to provide information to a vehicle occupant of the vehicle, and a controller in electrical communication with the automated driving system and the first display. The controller is programmed to identify a planned change in driving behavior initiated by the automated driving system. The controller is also programmed to identify a traffic sign misinterpretation using the automated driving system. The controller is also programmed to notify the vehicle occupant using the first display in response to identifying at least one of: a planned change in driving behavior and a traffic sign misinterpretation.

[0005] In another aspect of the present disclosure, in order to identify a planned change in driving behavior, the controller is further programmed to monitor a path planning algorithm of the automated driving system for a planned change in driving behavior. The planned change in driving behavior includes at least one of: an emergency stop, sudden acceleration, lane change, and merging.

[0006] In another aspect of the present disclosure, the system further includes a plurality of vehicle sensors in electrical communication with the controller. To identify a traffic sign interpretation error, the controller is further programmed to detect a traffic sign of interest using the plurality of vehicle sensors. To identify a traffic sign interpretation error, the controller is further programmed to search a database of known traffic signs for the traffic sign of interest. To identify a traffic sign interpretation error, the controller is further programmed to identify a traffic sign interpretation error in response to determining that the traffic sign of interest is not found within the database of known traffic signs.

[0007] In another aspect of the present disclosure, to notify the vehicle occupant, the controller is further programmed to generate a graphic based at least in part on at least one of: a planned change in driving behavior and a traffic sign interpretation error. To notify the vehicle occupant, the controller is further programmed to display the graphic to the vehicle occupant using the first display.

[0008] In another aspect of the present disclosure, to generate the graphics, the controller is further programmed to generate a warning graphic in response to identifying a planned change in driving behavior. The warning graphic is configured to alert a vehicle occupant to the planned change in driving behavior. To generate the graphics, the controller is further programmed to generate a confirmation graphic in response to identifying a traffic sign interpretation error. The confirmation graphic is configured to prompt the vehicle occupant to resolve the traffic sign interpretation error.

[0009] In another aspect of the present disclosure, to generate the warning graphic, the controller is further programmed to use an explainable artificial intelligence (XAI) algorithm to determine an explanation for the planned change in driving behavior. To generate the warning graphic, the controller is further programmed to identify an object of interest in the vehicle's surroundings that causes the automated driving system to initiate the planned change in driving behavior based at least in part on the explanation. To generate the warning graphic, the controller is further programmed to generate the warning graphic. From the perspective of the vehicle occupant, the warning graphic appears overlaid on the object of interest.

[0010] In another aspect of the present disclosure, the first display is an augmented reality head-up display (AR-HUD) system in electrical communication with the controller. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. To display the graphic, the controller is also programmed to use the occupant position tracking device to determine the position of the vehicle occupant of the vehicle. To display the graphic, the controller is also programmed to calculate the size, shape, and position of the graphic based on the position of the vehicle occupant. To display the graphic, the controller is also programmed to display the graphic on the front windshield of the vehicle using the augmented reality head-up display system based on the size, shape, and position.

[0011] In another aspect of the present disclosure, the first display also includes a transparent front windshield display (TWD) system that is electrically in communication with the controller. The transparent front windshield display system includes a transparent phosphor embedded in the front windshield of the vehicle and a TWD projector. In order to display the graphics, the controller is also programmed to calculate the size, shape and position of the graphics. In order to display the graphics, the controller is also programmed to use the TWD system to display the graphics on the front windshield of the vehicle according to the size, shape and position.

[0012] In another aspect of the present disclosure, the system further includes a second display in electrical communication with the controller. The second display is configured to provide information to a remote vehicle occupant. The controller is further programmed to notify the remote vehicle occupant using the second display in response to identifying a planned change in driving behavior.

[0013] In another aspect of the present disclosure, the system further includes a haptic feedback device in electrical communication with the controller.The controller is further programmed to notify a vehicle occupant using the haptic feedback device in response to identifying the planned change in driving behavior.

[0014] According to several aspects, a method for providing information to a vehicle occupant of a vehicle is provided. The method may include identifying a planned change in driving behavior initiated by an automated driving system. The method may also include using the automated driving system to identify a traffic sign misinterpretation. The method may also include determining a notification mode in response to identifying at least one of: a planned change in driving behavior and a traffic sign misinterpretation. The notification mode includes one of: a visual notification mode and a tactile notification mode. The method may also include notifying the vehicle occupant using an augmented reality head-up display (AR-HUD) system in response to determining that the notification mode is a visual notification mode. The method may also include notifying the vehicle occupant using a tactile feedback device in response to determining that the notification mode is a tactile notification mode.

[0015] In another aspect of the present disclosure, identifying a planned change in driving behavior may further include monitoring a path planning algorithm of the autonomous driving system for a planned change in driving behavior. The planned change in driving behavior includes at least one of: an emergency stop, sudden acceleration, lane change, and merging.

[0016] In another aspect of the present disclosure, identifying a traffic sign interpretation error may further include detecting a traffic sign of interest using a plurality of vehicle sensors. Identifying a traffic sign interpretation error may further include searching a database of known traffic signs for the traffic sign of interest. Identifying a traffic sign interpretation error may further include identifying a traffic sign interpretation error in response to determining that the traffic sign of interest is not found within the database of known traffic signs.

[0017] In another aspect of the present disclosure, determining the notification mode may further include determining a gaze direction of the vehicle occupant using an occupant position tracking device. The gaze direction includes a front windshield gaze direction and a non-front windshield gaze direction. Determining the notification mode may further include determining the notification mode as a visual notification mode in response to determining that the gaze direction is a front windshield gaze direction. Determining the notification mode may further include determining the notification mode as a tactile notification mode in response to determining that the gaze direction is a non-front windshield gaze direction.

[0018] In another aspect of the present disclosure, notifying a vehicle occupant using an AR-HUD system may also include generating a warning graphic in response to identifying a planned change in driving behavior. Generating a warning graphic may also include using an explainable artificial intelligence (XAI) algorithm to determine an explanation for the planned change in driving behavior. Generating a warning graphic may also include identifying an object of interest in the vehicle's surroundings based at least in part on the explanation, which causes the autonomous driving system to initiate a planned change in driving behavior. Generating a warning graphic may also include generating a warning graphic, wherein, from the perspective of the vehicle occupant, the warning graphic is presented as an overlay on the object of interest. Notifying a vehicle occupant using an AR-HUD system may also include generating a confirmation graphic in response to identifying a traffic sign interpretation error. The confirmation graphic is configured to prompt the vehicle occupant to resolve the traffic sign interpretation error.

[0019] In another aspect of the present disclosure, notifying a vehicle occupant using the AR-HUD system may further include determining a location of a vehicle occupant of the vehicle using an occupant location tracking device. Notifying a vehicle occupant using the AR-HUD system may further include calculating a size, shape, and position of a warning graphic based on the location of the vehicle occupant. Notifying a vehicle occupant using the AR-HUD system may further include displaying at least one of the following: displaying a warning graphic and a confirmation graphic on a front windshield of the vehicle based on the size, shape, and position using the AR-HUD system.

[0020] In another aspect of the present disclosure, the method further includes notifying the remote vehicle occupant using the second display in response to identifying the planned change in driving behavior. The second display is configured to provide information to the remote vehicle occupant.

[0021] According to several aspects, a system for providing information to a vehicle occupant of a vehicle is provided, which system may include a plurality of vehicle sensors, an autonomous driving system, a first display configured to provide information to the vehicle occupant of the vehicle, and a second display configured to provide information to a remote vehicle occupant of a remote vehicle. The second display is a rear display fixed to a rear windshield of the vehicle. The system may also include a controller in electrical communication with the plurality of vehicle sensors, the autonomous driving system, the first display, and the second display. The controller is also programmed to identify a planned change in driving behavior initiated by the autonomous driving system. The controller is also programmed to identify a traffic sign misinterpretation using the autonomous driving system. The controller is also programmed to notify the vehicle occupant using the first display in response to identifying at least one of the following: a planned change in driving behavior and a traffic sign misinterpretation. The controller is also programmed to notify the remote vehicle occupant using the second display in response to identifying a planned change in driving behavior.

[0022] In another aspect of the present disclosure, to notify the vehicle occupant using the first display, the controller is further programmed to use an explainable artificial intelligence (XAI) algorithm to determine an explanation for the planned change in driving behavior. To notify the vehicle occupant using the first display, the controller is further programmed to identify an object of interest in the vehicle's surroundings that causes the automated driving system to initiate the planned change in driving behavior based at least in part on the explanation. To notify the vehicle occupant using the first display, the controller is further programmed to generate a warning graphic. The warning graphic indicates the object of interest. To notify the vehicle occupant using the first display, the controller is further programmed to display the warning graphic to the vehicle occupant using the first display.

[0023] In another aspect of the present disclosure, the first display is an augmented reality head-up display (AR-HUD) system in electrical communication with the controller. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. To display the warning graphic, the controller is also programmed to use the occupant position tracking device to determine the position of the vehicle occupant of the vehicle. To display the warning graphic, the controller is also programmed to calculate the size, shape and position of the warning graphic based on the position of the vehicle occupant. To display the warning graphic, the controller is also programmed to display the warning graphic on the front windshield of the vehicle using the augmented reality head-up display system based on the size, shape and position.

[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0026] Figure 1 is a schematic diagram of a system for providing information to a vehicle occupant of a vehicle according to an exemplary embodiment;

[0027] Figure 2 is a schematic diagram of an AR-HUD system for use by an exemplary occupant according to an exemplary embodiment;

[0028] Figure 3 is a schematic front view of a dual focal plane augmented reality display according to an exemplary embodiment, highlighting a second image plane of the dual focal plane augmented reality display;

[0029] Figure 4 is a schematic diagram of a second image plane of a dual focal plane augmented reality display according to an exemplary embodiment;

[0030] Figure 5is a schematic diagram of an exemplary use of a rear display according to an exemplary embodiment;

[0031] Figure 6 is a flow chart of a method for providing information to a vehicle occupant of a vehicle according to an exemplary embodiment;

[0032] Figure 7 is a schematic diagram of a first view of a vehicle interior from the perspective of a vehicle occupant according to an exemplary embodiment; and

[0033] Figure 8 is a schematic diagram of a second view of a vehicle interior from the perspective of a vehicle occupant according to an exemplary embodiment. DETAILED DESCRIPTION

[0034] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0035] The vehicle autonomous driving system can evaluate the environment around the vehicle and make adjustments to the planned path in response to changing conditions (such as the movement of a remote vehicle). In some cases, the vehicle sensors can detect anomalies such as shadows, glare, reflections, foreign objects on the road, water on the road, etc. Detection of such anomalies may result in adjustments to the planned path. Therefore, the present disclosure provides a new and improved system and method for providing information to vehicle occupants of a vehicle, which informs the vehicle occupants about planned changes in driving behavior, giving the vehicle occupants an opportunity to take action in response to the planned changes in driving behavior.

[0036] refer to Figure 1 , a system for providing information to a vehicle occupant of a vehicle is shown, generally indicated by reference numeral 10. The system 10 is shown with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be understood that the vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a controller 14, a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent front windshield display (TWD) system 20, a human-machine interface (HMI) 22, an autonomous driving system 24, a haptic feedback device 26, and a rear display 28.

[0037] The controller 14 is used to implement a method 100 for providing information to a vehicle occupant of a vehicle, as will be described below. The controller 14 includes at least one processor 30 and a non-transitory computer-readable storage device or medium 32. The processor 30 can be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of multiple processors associated with the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, a combination thereof, or generally a device for executing instructions. The computer-readable storage device or medium 32 can include volatile and non-volatile storage in, for example, read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the processor 30 is powered off. The computer readable storage device or medium 32 may be implemented using a plurality of storage devices, such as a PROM (Programmable Read Only Memory), an EPROM (Electrical PROM), an EEPROM (Electrically Erasable PROM), a flash memory, or other electrical, magnetic, optical memory or combination storage device capable of storing data, some of which represents executable instructions used by the controller 14 to control various systems of the vehicle 12. The controller 14 may also be comprised of multiple controllers in electrical communication with each other. The controller 14 may be interconnected with additional systems and / or controllers of the vehicle 12, thereby allowing the controller 14 to access data, such as the speed, acceleration, braking, and steering angle of the vehicle 12.

[0038] The controller 14 is in electrical communication with a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent front windshield display (TWD) system 20, a human-machine interface (HMI) 22, an autonomous driving system 24, a haptic feedback device 26, and a rear display 28. In an exemplary embodiment, electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a serial peripheral interface (SPI) network, etc. It should be understood that various additional wired and wireless technologies and communication protocols for communicating with the controller 14 are within the scope of the present disclosure.

[0039] The plurality of vehicle sensors 16 are used to acquire information related to the vehicle 12 . In the exemplary embodiment, the plurality of vehicle sensors 16 include at least a camera system 34 , a vehicle communication system 36 , and a global navigation satellite system (GNSS) 38 .

[0040] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors for determining performance data regarding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a brake position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.

[0041] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment within the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of a seat occupancy sensor, a cabin air temperature sensor, a cabin motion detection sensor, a cabin camera, a cabin microphone, etc.

[0042] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors for determining information about the environment 40 surrounding the vehicle 12. In a non-limiting example, the plurality of vehicle sensors 16 also include at least one of an ambient air temperature sensor, an atmospheric pressure sensor, and / or a photo and / or video camera positioned to view the environment 40 in front of the vehicle 12.

[0043] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is a perception sensor capable of perceiving objects in the environment 40 surrounding the vehicle 12 and / or measuring distances. In a non-limiting example, the plurality of vehicle sensors 16 include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is fixed to the interior of the vehicle 12, for example, fixed to the roof lining of the vehicle 12, with a field of view through the front windshield 42 of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed to the exterior of the vehicle 12, for example, fixed to the roof of the vehicle 12, with a field of view of the environment 40 surrounding the vehicle 12. It should be understood that various additional types of perception sensors, such as LiDAR sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors are within the scope of the present disclosure. The plurality of vehicle sensors 16 are in electrical communication with the controller 14, as described above.

[0044] The camera system 34 is a perception sensor for capturing images and / or video of the environment 40 surrounding the vehicle 12. In an exemplary embodiment, the camera system 34 includes a photo and / or video camera positioned to view the environment 40 surrounding the vehicle 12. In a non-limiting example, the camera system 34 includes a camera fixed to the interior of the vehicle 12 (e.g., in the headliner of the vehicle 12) that has a field of view through the front windshield 42. In another non-limiting example, the camera system 34 includes a camera fixed to the exterior of the vehicle 12 (e.g., fixed to the roof of the vehicle 12) that has a field of view of the environment 40 in front of the vehicle 12.

[0045] In another exemplary embodiment, the camera system 34 is a surround view camera system that includes a plurality of cameras (also referred to as satellite cameras) arranged to provide a view of the environment 40 adjacent to all sides of the vehicle 12. In a non-limiting example, the camera system 34 includes a front camera (e.g., mounted in the front grille of the vehicle 12), a rear camera (e.g., mounted on the rear tailgate of the vehicle 12), and two side cameras (e.g., mounted below each of the two side view mirrors of the vehicle 12). In another non-limiting example, the camera system 34 also includes an additional rear view camera mounted near the center high mounted brake light of the vehicle 12.

[0046] It should be understood that camera systems with additional cameras and / or additional mounting locations are within the scope of the present disclosure. It should also be understood that cameras with various sensor types, including, for example, charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors, are within the scope of the present disclosure. In addition, cameras with various lens types, including, for example, wide angle lenses and / or narrow angle lenses, are also within the scope of the present disclosure.

[0047] GNSS 38 is used to determine the geographic location of vehicle 12. In an exemplary embodiment, GNSS 38 is a global positioning system (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from a plurality of satellites, and the GPS controller calculates the geographic location of vehicle 12 based on the signals received by the GPS receiver antenna. In an exemplary embodiment, GNSS 38 also includes a map. The map includes information about infrastructure, such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, the geographic location of vehicle 12 is contextualized using map information. In a non-limiting example, a map is retrieved from a remote source using a wireless connection. In another non-limiting example, the map is stored in a database of GNSS 38. It should be understood that various additional types of satellite-based radio navigation systems, such as global positioning system (GPS), Galileo, GLONASS, and BeiDou navigation satellite system (BDS) are within the scope of the present disclosure. It should be appreciated that the GNSS 38 may be integrated with the controller 14 (eg, located on the same circuit board as the controller 14 or otherwise be a part of the controller 14) without departing from the scope of the present disclosure.

[0048] refer to Figure 2 , a system diagram of an AR-HUD system 18 for use with an exemplary vehicle occupant 44 is shown. Within the scope of the present disclosure, in a non-limiting example, the vehicle occupant 44 includes a driver, a passenger, and / or any other person in the vehicle 12. The AR-HUD system 18 is used to display AR-HUD graphics 46 (i.e., notification symbols that provide visual information to the vehicle occupant 44) on the front windshield 42 of the vehicle 12. The AR-HUD system 18 includes an AR-HUD projector 48 and an occupant position tracking device 50. The AR-HUD system 18 is in electrical communication with the controller 14, as described above.

[0049] The AR-HUD projector 48 is used to project the AR-HUD graphics 46 onto the front windshield 42 of the vehicle 12. It should be understood that various devices designed for projecting images, including, for example, optical collimators, laser projectors, digital optical projectors (DLP), etc., are within the scope of the present disclosure.

[0050] The occupant position tracking device 50 is used to determine the position of the vehicle occupant 44 in the vehicle 12. For example, the occupant position tracking device 50 can track the position of the head 52a or the eyes 52b of the vehicle occupant 44. The position of the vehicle occupant 44 in the vehicle 12 from the occupant position tracking device 50 is used to position the AR-HUD graphics 46 on the front windshield 42 of the vehicle 12. In an exemplary embodiment, the occupant position tracking device 50 is one or more cameras disposed in the vehicle 12.

[0051] To operate the AR-HUD system 18, the controller 14 includes a plurality of software modules, including a system manager 54. During operation of the system 18, the system manager 54 receives at least a first input 56, a second input 58, and a third input 60. The first input 56 indicates the position of the vehicle 12 in space (i.e., the geographic location of the vehicle 12), the second input 58 indicates the position of the vehicle occupant 44 in the vehicle 12 (e.g., the position of the eyes and / or head of the vehicle occupant 44 in the vehicle 12), and the third input 60 is data related to a planned change in driving behavior and / or a traffic sign interpretation error, as will be discussed in more detail below. The first input 56 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering, and the data is collected from a plurality of vehicle sensors 16. The second input 58 is received from the occupant position tracking device 50. The third input 60 is data related to a planned change in driving behavior and / or a traffic sign interpretation error. The system manager 54 is configured to determine (e.g., calculate) the type, size, shape, and color of the AR-HUD graphics 46 to be displayed using the AR-HUD projector 48 based on the first input 56 (i.e., the vehicle position in the environment 40), the second input 58 (e.g., the position of the eyes 52b and / or the head 52a of the vehicle occupant 44 in the vehicle 12), and the third input 60 (i.e., data related to the planned change in driving behavior and / or traffic sign interpretation error). The system manager 54 instructs the image engine 62 (which is a software module or integrated circuit of the AR-HUD projector 48 or the controller 14) to display the AR-HUD graphics 46 using the AR-HUD projector 48. The image engine 62 displays the AR-HUD graphics 46 on the front windshield 42 of the vehicle 12 using the AR-HUD projector 48 based on the type, size, shape, and color of the AR-HUD graphics 46 determined by the system manager 54. The AR-HUD graphics 46 are projected onto the front windshield 42 by the AR-HUD projector 48 to display the AR-HUD graphics 46 along the road surface 64 .

[0052] In an exemplary embodiment of the present disclosure, the AR-HUD system 18 is a dual focal plane AR-HUD system. Figure 3 and Figure 4 And continue to refer to Figure 2 , the AR-HUD system 18 has a first image plane 66 and a second image plane 68. The first image plane 66 shows the view of the outside world, and the second image plane 68 is reserved for displaying the AR-HUD graphics 46. The second image plane 68 spans multiple lanes and the AR-HUD graphics 46 appear at a farther position on the road surface 64 relative to the first image plane 66. For example, Figure 3 and Figure 4 As shown, the second image plane 68 covers the left lane 70, the center lane 72, and the right lane 74. As a non-limiting example, in the center lane 72, the second image plane 68 starts at a first predetermined distance D1 (e.g., twenty-five meters) from the vehicle 12 and ends at a second predetermined distance D2 (e.g., ninety meters) from the vehicle 12. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to help the vehicle occupant 44 see the AR-HUD graphics 46 displayed using the AR-HUD projector 48. In the left lane 70 and the right lane 74, the second image plane 68 is bounded by a sloped boundary that starts at the first predetermined distance D1 from the vehicle 12 and ends at a third predetermined distance D3 (e.g., fifty meters) from the vehicle 12. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to help the vehicle occupant 44 see the AR-HUD graphics 46 displayed using the AR-HUD projector 48. As used herein, the term "dual focal plane AR-HUD" refers to an AR-HUD system that presents images in a first image plane and a second image plane, wherein the first image plane is located at a different position than the second image plane. It is desirable to configure the AR-HUD system 18 as a dual focal plane AR-HUD to facilitate manipulation of the AR-HUD graphics 46 over the field of view of the outside world. For example, by using a dual focal plane AR-HUD, the size, position, and characteristics of the AR-HUD graphics 46 may be changed based on, for example, the position of the eye 52b of the vehicle occupant 44.

[0053] The TWD system 20 is used to display images on the front windshield 42 of the vehicle 12. In an exemplary embodiment, the AR-HUD system 18 can display AR-HUD graphics 46 in predefined areas of the front windshield 42 (for example, in the first image plane 66 and the second image plane 68). The TWD system 20 can display TWD graphics (not shown) in any area of ​​the front windshield 42. Therefore, by jointly operating the AR-HUD system 18 and the TWD system 20, the controller 14 can display graphics in any area of ​​the front windshield 42. In an exemplary embodiment, the TWD system 20 includes a transparent phosphor (not shown) embedded in the front windshield 42 and a TWD projector 76 ( Figure 1 ). The TWD system 20 is in electrical communication with the controller 14, as described above.

[0054] The transparent phosphor is a luminescent particle that fluoresces in response to being excited by the TWD projector 76. In an exemplary embodiment, the transparent phosphor is a red, green and blue (RGB) phosphor, allowing full-color operation of the TWD system 20. The use of monochromatic and / or bichromatic phosphors is also within the scope of the present disclosure. When the excitation light is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light can be, for example, violet light (ranging from about 380 to 450 nanometers) and / or ultraviolet light in the visible spectrum.

[0055] The TWD projector 76 is used to excite the transparent phosphor in a predetermined pattern to produce a TWD pattern on the front windshield 42. In an exemplary embodiment, the TWD projector 76 is a violet / ultraviolet laser projector disposed near the headliner of the vehicle 12. The TWD projector 76 includes three lasers, each of which is configured to excite one of the red, green, or blue transparent phosphors.

[0056] In an exemplary embodiment, in addition to the AR-HUD system 18 and the TWD system 20, an HMI 22 is used to display information to the vehicle occupants 44 of the vehicle 12. In another exemplary embodiment, an HMI 22 is used instead of the AR-HUD system 18 and / or the TWD system 20 to display information to the vehicle occupants 44 of the vehicle 12. In the aforementioned exemplary embodiments, the HMI 22 is a display system that is located in the field of view of the vehicle occupants 44 and is capable of displaying text, graphics and / or images. It should be understood that HMI display systems including LCD displays, LED displays, etc. are within the scope of the present disclosure. Additional exemplary embodiments in which the HMI 22 is disposed in a rearview mirror are also within the scope of the present disclosure. As described above, the HMI 22 is in electrical communication with the controller 14.

[0057] The automated driving system 24 is used to provide assistance to the vehicle occupants 44 to increase occupant awareness and / or control the behavior of the vehicle 12. Within the scope of the present disclosure, the automated driving system 24 encompasses systems that provide any level of assistance to the vehicle occupants 44 (e.g., blind spot warning, lane departure warning, etc.) as well as systems that are capable of automatically driving the vehicle 12 under some or all conditions (e.g., automatic lane keeping, adaptive cruise control, fully automated driving, etc.). It should be understood that all levels of driving automation defined by, for example, SAE J3016 (i.e., SAE Level 0, SAE Level 1, SAE Level 2, SAE Level 3, SAE Level 4, and SAE Level 5) are within the scope of the present disclosure.

[0058] In an exemplary embodiment, the automated driving system 24 is configured to detect and / or receive information about the environment 40 surrounding the vehicle 12 and process the information to provide assistance to the vehicle occupants 44. In some embodiments, the automated driving system 24 is a software module executed on the controller 14. In other embodiments, the automated driving system 24 includes a separate automated driving system controller similar to the controller 14 that is capable of processing information about the environment 40 surrounding the vehicle 12. In an exemplary embodiment, the automated driving system 24 can operate in a manual operating mode, a partially automated operating mode, and a fully automated operating mode.

[0059] Within the scope of the present disclosure, manual operation mode means that the autonomous driving system 24 provides warnings or notifications to the vehicle occupants 44, but does not directly intervene or control the vehicle 12. In a non-limiting example, the autonomous driving system 24 receives information from the plurality of vehicle sensors 16. Using techniques such as computer vision, the autonomous driving system 24 understands the environment 40 surrounding the vehicle 12 and provides assistance to the vehicle occupants 44. For example, if the autonomous driving system 24 identifies that the vehicle 12 may collide with a remote vehicle based on data from the plurality of vehicle sensors 16, the autonomous driving system 24 may issue a warning to the vehicle occupants 44 using the AR-HUD system 18, the TWD system 20, and / or the HMI 22.

[0060] Within the scope of the present disclosure, a partially automated mode of operation means that the autonomous driving system 24 issues a warning or notification to the vehicle occupant 44 and may directly intervene or control the vehicle 12 in certain circumstances. In a non-limiting example, the autonomous driving system 24 also communicates electrically with components of the vehicle 12, such as the braking system, propulsion system, and / or steering system of the vehicle 12, so that the autonomous driving system 24 can control the behavior of the vehicle 12. In a non-limiting example, the autonomous driving system 24 may control the behavior of the vehicle 12 by using the brakes of the vehicle 12 to avoid an imminent collision. In another non-limiting example, the autonomous driving system 24 may control the steering system of the vehicle 12 to provide an automatic lane keeping feature. In another non-limiting example, the autonomous driving system 24 may control the braking system, propulsion system, and steering system of the vehicle 12 to temporarily drive the vehicle 12 toward a predetermined destination. However, occupant intervention may be required at any time. In an exemplary embodiment, the autonomous driving system 24 may include additional components, such as an eye tracking device configured to monitor the occupant's attention level and ensure that the occupant is ready to take over control of the vehicle 12.

[0061] Within the scope of the present disclosure, a fully automated mode of operation means that the automated driving system 24 uses data from the plurality of vehicle sensors 16 to understand the environment 40 and controls the vehicle 12 to drive the vehicle 12 to a predetermined destination without control or intervention by a vehicle occupant 44 .

[0062] The automated driving system 24 operates using a path planning algorithm that is configured to generate a safe, efficient trajectory for the vehicle 12 to navigate the environment around the vehicle 12. In an exemplary embodiment, the path planning algorithm is a machine learning algorithm that is trained to output control signals for the vehicle 12 based on input data collected from the plurality of vehicle sensors 16. In another exemplary embodiment, the path planning algorithm is a deterministic algorithm that has been programmed to output control signals for the vehicle 12 based on data collected from the plurality of vehicle sensors 16.

[0063] In a non-limiting example, the path planning algorithm generates a series of waypoints or a continuous path that the vehicle 12 should follow to reach the destination while complying with rules, regulations, and safety constraints. The series of waypoints or the continuous path is generated based at least in part on the detailed map and the current state of the vehicle 12 (i.e., the position, speed, and direction of the vehicle 12). The detailed map includes, for example, information about lane boundaries, road geometry, speed limits, traffic signs, and / or other relevant features. In an exemplary embodiment, the detailed map is stored in the medium 32 of the controller 14 and / or on a remote database or server.

[0064] In another exemplary embodiment, the path planning algorithm performs perception and mapping tasks to interpret data collected from the plurality of vehicle sensors 16 and create, update, and / or augment a detailed map. In a non-limiting example, the path planning algorithm continuously receives data about the environment 40 surrounding the vehicle 12 from the plurality of vehicle sensors 16 and interprets the data. In a non-limiting example, interpreting the data includes performing computer vision tasks such as segmentation, classification, localization, etc. of objects perceived by the plurality of vehicle sensors 16.

[0065] In an exemplary embodiment, the results of the one or more computer vision tasks include a perception confidence value. The perception confidence value represents the confidence level in interpreting the data received from the plurality of vehicle sensors 16. In this example, if the exemplary object perceived by the plurality of vehicle sensors 16 is classified as a tree by the computer vision task, the associated perception confidence value represents the confidence that the exemplary object is actually a tree. In a non-limiting example, a perception confidence value of 1 represents a very high confidence, and a perception confidence value of 0 represents a very low confidence.

[0066] As described above, the path planning algorithm continually receives additional data about the environment 40 surrounding the vehicle 12 from the plurality of vehicle sensors 16. Accordingly, the path planning algorithm may adjust a sequence of waypoints or a continuous path based on changing conditions in the environment 40 surrounding the vehicle 12. For example, changes and / or unexpected behavior of other road users (i.e., remote vehicle occupants), such as emergency stops, turns, lane changes, etc., may cause the path planning algorithm to make adjustments to a sequence of waypoints or a continuous path.

[0067] In another example, the intrusion of foreign objects on the road, such as unexpected pedestrians, wild animals, wreckage caused by motor vehicle accidents, etc., may cause the path planning algorithm to adjust a sequence of waypoints or a continuous path. Adjustments to a sequence of waypoints or a continuous path are referred to herein as "planned changes in driving behavior." Therefore, within the scope of the present disclosure, a planned change in driving behavior is any adjustment made by the path planning algorithm to a sequence of waypoints or a continuous path. In a non-limiting example, the planned change in driving behavior includes at least one of an emergency stop, sudden acceleration, lane change, and / or merging.

[0068] It should be understood that the autonomous driving system 24 may include any software and / or hardware modules configured to operate in a manual operating mode, a partially automated operating mode, or a fully automated operating mode as described above.

[0069] The haptic feedback device 26 is used to provide haptic feedback to the vehicle occupant 44. Within the scope of the present disclosure, haptic feedback includes force, vibration, or movement provided to the vehicle occupant 44 to notify the vehicle occupant 44. In an exemplary embodiment, the haptic feedback device 26 includes one or more actuators disposed within the vehicle 12. Within the scope of the present disclosure, the one or more actuators include, for example, a linear resonant actuator (LRA), an eccentric rotating mass (ERM) actuator, and / or a piezoelectric actuator. In a non-limiting example, the one or more actuators are disposed within an element of the vehicle 12 that the vehicle occupant 44 contacts when operating the vehicle 12, such as a seat of the vehicle 12, a steering wheel of the vehicle 12, etc.

[0070] In a non-limiting example, one or more actuators may be positioned so that selective actuation of the one or more actuators provides additional information about the notification to the vehicle occupant 44. For example, one or more actuators may be located in multiple areas of a seat of the vehicle 12 so that tactile feedback may be generated in multiple areas of the seat through selective actuation of the one or more actuators. Thus, tactile feedback originating from each of the multiple areas of the seat may indicate various types of notifications to the vehicle occupant 44. As described above, one or more actuators of the tactile feedback device 26 are in electrical communication with the controller 14.

[0071] The rear display 28 is used to provide information to remote vehicle occupants of remote vehicles behind the vehicle 12. In an exemplary embodiment, the rear display 28 is fixed to a rear windshield 78 of the vehicle 12. In a non-limiting example, the rear display 28 is an electronic display, such as a liquid crystal display (LCD), a light emitting diode (LED) display, a plasma display, an electroluminescent display, etc. In another non-limiting example, the rear windshield 78 includes a transparent phosphor that is configured to emit light when selectively excited by a laser projector, similar to the TWD system 20 discussed above.

[0072] Reference Figure 5 , a schematic diagram showing an exemplary use of the rear display 28 is shown. Figure 5 As shown, the rear display 28 can be used to provide information, such as a message 80, to a remote vehicle occupant of a remote vehicle behind the vehicle 12. In a non-limiting example, the message 80 includes the text "Caution Slow Down" to notify the remote vehicle occupant that the vehicle 12 is slowing down. It should be understood that the rear display 28 can be used to display any message, graphic, image, icon, etc. within the present disclosure. The rear display 28 can also adjust the color, size and / or position of the message 80 within the scope of the present disclosure. As described above, the rear display 28 is in electrical communication with the controller 14.

[0073] Reference Figure 6 , a flow chart of a method 100 for providing information to a vehicle occupant 44 of a vehicle 12 is shown. The method 100 begins at box 102 and proceeds to boxes 104, 106, and 108.

[0074] At box 104, the controller 14 monitors the path planning algorithm of the autonomous driving system 24 for planned changes in driving behavior. As described above, a planned change in driving behavior is any adjustment to a sequence of waypoints or a continuous path made by the path planning algorithm of the autonomous driving system 24, including emergency stops, sudden accelerations, lane changes, and / or lane merges. If a planned change in driving behavior is not identified at box 104, the method 100 continues to enter a standby state at box 110. If a planned change in driving behavior is identified at box 104, the method 100 proceeds to boxes 112 and 114.

[0075] At block 112, the controller 14 displays the message 80 to one or more remote vehicle occupants of one or more remote vehicles behind the vehicle 12 using the rear display 28. In an exemplary embodiment, the message 80 is configured to notify one or more remote vehicle occupants of the one or more remote vehicles of the planned change in driving behavior identified at block 104. For example, if the planned change in driving behavior includes an emergency stop, the message 80 may indicate "Slow down carefully," such as Figure 5It should be understood that the message 80 may include any text, graphics, images, icons, etc. within the present disclosure. After block 112, the method 100 continues at block 110 to enter a standby state.

[0076] At box 114, the controller 14 determines a notification method. Within the scope of the present disclosure, the notification method is the method by which the controller 14 will notify the vehicle occupant 44 of the planned change in driving behavior. Within the scope of the present disclosure, the notification method includes one of the following: a visual notification method and a tactile notification method, as will be discussed in more detail below.

[0077] The notification manner is based at least in part on the gaze direction of the vehicle occupant 44. In the scope of the present disclosure, the gaze direction of the vehicle occupant 44 is the direction in which the vehicle occupant 44 is currently looking. In the scope of the present disclosure, the gaze direction includes one of: a front windshield gaze direction (i.e., the vehicle occupant 44 is looking at the front windshield 42) and a non-front windshield gaze direction (i.e., the vehicle occupant 44 is looking at anywhere beside the front windshield 42). In an exemplary embodiment, the occupant position tracking device 50 is used to determine the angle of the head 52a of the vehicle occupant 44, and the gaze direction is determined based at least in part on the angle of the head 52a of the vehicle occupant 44. In another exemplary embodiment, the occupant position tracking device 50 is used to perform eye tracking on the eyes 52b of the vehicle occupant 44, and the gaze direction 44 is determined based at least in part on the eye tracking of the eyes 52b of the vehicle occupant. If the gaze direction is determined to be a front windshield gaze direction, the method 100 proceeds to frame 116. If the gaze direction is determined to be a non-front windshield gaze direction, the method 100 proceeds to frame 118.

[0078] Reference Figure 7 , a schematic diagram of a first view 82a of the interior of the vehicle 12 from the perspective of a vehicle occupant 44 is shown. The first view 82a includes a warning graphic 84. The warning graphic 84 includes a warning location indicator 86a and a text warning 86b, as will be discussed in more detail below. The warning location indicator 86a is positioned to highlight foreign objects 88 (e.g., tire marks) on the road, as will be discussed in more detail below. The warning graphic 84 can be displayed using the AR-HUD system 18, the TWD system 20, or a combination of the AR-HUD system 18 and the TWD system 20. It should also be understood that the warning graphic 84 can be displayed on the HMI 22 without departing from the scope of the present disclosure.

[0079] Reference again Figure 6 And continue to refer to Figure 7, at box 116, the controller 14 determines an explanation for the planned change in driving behavior. Within the scope of the present disclosure, the explanation is the reasoning behind the planned change in driving behavior. The explanation includes objects in the environment 40 that affect the planned change in driving behavior. For example, in the exemplary scenario, multiple vehicle sensors 16 mistakenly identify a foreign object 88 on the road as a pedestrian and therefore initiate a planned change in driving behavior (e.g., an emergency stop). Therefore, in the exemplary scenario, the explanation for the planned change in driving behavior is the perceived pedestrian.

[0080] The explanation also includes an explanation confidence value. The explanation confidence value indicates the confidence of the reasoning behind the planned change in driving behavior. A high explanation confidence value (e.g., an explanation confidence value of 1) indicates that the reasoning behind the planned change in driving behavior is likely correct.

[0081] In an exemplary embodiment, an explainable artificial intelligence (XAI) algorithm is used to determine an explanation for the planned change in driving behavior. In an exemplary embodiment, an explainable artificial intelligence (XAI) algorithm is a machine learning algorithm that is used to provide human-understandable explanations for decisions made by the path planning algorithm of the autonomous driving system 24. In a non-limiting example, the path planning algorithm of the autonomous driving system 24 is a deep learning algorithm (e.g., a neural network algorithm), and the XAI algorithm is applied to the path planning algorithm to determine an explanation for the planned change in driving behavior. The functionality of the exemplary XAI algorithm is discussed in more detail in U.S. Publication No. US20220067511A1, entitled “METHODS, SYSTEMS, AND APPARATUSES FOR USER-UNDERSTANDABLE EXPLAINABLE LEARNING MODELS” filed on September 17, 2020, the entire contents of which are incorporated into this disclosure by reference.

[0082] In an exemplary embodiment, the interpretation confidence value is determined based at least in part on the perceived confidence values ​​of objects in the environment 40 that affect the planned change in driving behavior. In another exemplary embodiment, the interpretation confidence value is further determined based at least in part on the XAI confidence value generated by the XAI algorithm. After frame 116, the method 100 proceeds to frame 120.

[0083] At box 120, the controller 14 identifies an object of interest in the environment 40 surrounding the vehicle 12. In an exemplary embodiment, the object of interest is an object and / or area in the environment 40 that causes the automated driving system 24 to initiate a planned change in driving behavior. Figure 7In the first view 82a shown, a foreign object 88 is identified as an object of interest, and because the interpretation indicates that the foreign object 88 has been identified as a pedestrian, a planned change in driving behavior is initiated.

[0084] It should be understood that in some examples, the object of interest may include an area of ​​the environment 40 surrounding the vehicle 12, rather than a specific object such as the foreign object 88. For example, the object that causes the automated driving system 24 to initiate a planned change in driving behavior may be in motion, so it may be out of the field of view of the vehicle occupant 44. In another example, the object that causes the automated driving system 24 to initiate a planned change in driving behavior may be a temporary condition, such as a shadow, reflection, glare, etc. Therefore, in such an example, the object of interest is considered to be the area of ​​the environment 40 surrounding the vehicle 12, including the last known position of the object that causes the automated driving system 24 to initiate a planned change in driving behavior. After box 120, the method 100 proceeds to box 122.

[0085] At box 122, the controller 14 generates the warning graphic 84. In the exemplary embodiment, the warning graphic 84 includes a warning location indicator 86a and a textual warning 86b. From the perspective of the vehicle occupant 44, the warning location indicator 86a appears overlaid on the object of interest (e.g., foreign object 88), such as Figure 7 82a in the first view 82a. In an exemplary embodiment, at least one of the size, shape and color of the warning position indicator 86a is used to visually indicate the interpretation confidence value of the interpretation. For example, if the interpretation confidence value is less than or equal to 0.33, the warning position indicator 86a is colored red. If the interpretation confidence value is greater than 0.33 and less than or equal to 0.66, the warning position indicator 86a is colored orange. If the interpretation confidence value is greater than 0.66, the warning position indicator 86a is colored green.

[0086] In an exemplary embodiment, the textual warning 86b includes the reasoning behind the planned change in driving behavior. Figure 7 In the example shown, the textual warning 86b may include the text “Stop due to perceived pedestrian in the road.” Thus, based on the textual warning 86b and the location and color of the warning location indicator 86a, the vehicle occupant 44 may take action to disable and / or override the automated driving system 24, such as depressing the accelerator pedal of the vehicle 12. It should be appreciated that the above-discussed and Figure 7 The appearance and composition of the illustrated warning graphic 84 are merely exemplary in nature, and any graphic configured to warn the vehicle occupant 44 and / or provide the vehicle occupant 44 with the reasoning behind the planned change in driving behavior is within the scope of the present disclosure. After frame 122 , the method 100 proceeds to frame 124 .

[0087] At box 124, the controller 14 uses at least one of the AR-HUD system 18, the TWD system 20, and the HMI 22 to display the warning graphic 84 to the vehicle occupant 44. To display the warning graphic 84, the controller 14 first determines the location of the vehicle occupant 44 using the occupant position tracking device 50, as described above. After determining the location of the vehicle occupant 44, the controller 14 calculates the size, shape, and position of the AR-HUD graphic 46 (i.e., the warning graphic 84), as described above. The position of the AR-HUD graphic 46 is selected so that from the perspective of the vehicle occupant 44, the warning position indicator 86a appears to be overlaid on the object of interest (e.g., the foreign object 88), as described above. Figure 7 As shown in the first view 82a.

[0088] After calculating the size, shape, and position of the AR-HUD graphic 46 (i.e., the warning graphic 84), the controller 14 uses at least one of the AR-HUD system 18 and the TWD system 20 to display the AR-HUD graphic 46, as described above. In an exemplary embodiment, the controller 14 also uses the HMI 22 to provide an augmented view of the environment 40 around the vehicle 12, including the AR-HUD graphic 46. After frame 124, the method 100 continues at frame 110 to enter the standby state.

[0089] At box 118, the controller 14 uses the haptic feedback device 26 to provide haptic feedback to the vehicle occupant 44, notifying the vehicle occupant 44 about the planned change in driving behavior. In an exemplary embodiment, at box 118, the controller 14 also determines an explanation for the planned change in driving behavior, as discussed above with reference to box 116. In a non-limiting example, one or more aspects of the haptic feedback, such as intensity, location, duration, etc., are adjusted based at least in part on the explanation, for example, based on the explanation confidence value. After box 118, the method 100 continues at box 110 to enter a standby state.

[0090] refer to Figure 8 , a schematic diagram of a second view 82b of the interior of the vehicle 12 from the perspective of the vehicle occupant 44 is shown. The second view 82b includes a confirmation graphic 90. The confirmation graphic 90 includes a sign position indicator 92a and a text confirmation message 92b, as will be discussed in more detail below. The sign position indicator 92a is positioned to highlight a traffic sign 94 of interest near the road, as will be discussed in more detail below. Figure 8 As shown, the traffic sign of interest 94 is partially obscured. The confirmation graphic 90 may be displayed using the AR-HUD system 18, the TWD system 20, or a combination of the AR-HUD system 18 and the TWD system 20. It should also be understood that the confirmation graphic 90 may be displayed on the HMI 22 without departing from the scope of the present disclosure.

[0091] Reference again Figure 6 And continue to refer to Figure 8 At box 106, the controller 14 uses the plurality of vehicle sensors 16 to detect a traffic sign of interest 94. Within the scope of the present disclosure, a traffic sign of interest 94 includes any traffic sign in the environment 40 surrounding the vehicle 12. After detecting the traffic sign of interest 94, the controller 14 uses the plurality of vehicle sensors 16 to identify the traffic sign of interest. For example, the traffic sign of interest 94 may be identified as a speed limit sign. Figure 8 In the example shown in the second view 82 b in FIG. 8 , the traffic sign 94 of interest is obscured and thus can be identified as a speed limit sign indicating a speed limit of five miles per hour. After frame 106 , the method 100 proceeds to frame 126 .

[0092] At box 126, the controller 14 searches for the traffic sign of interest 94 in a database of known traffic signs. In an exemplary embodiment, the database of known traffic signs is stored in the medium 32 of the controller 14. In another exemplary embodiment, the database of known traffic signs is stored on a remote server system and accessed using the vehicle communication system 36. If the traffic sign of interest 94 is found in the database, the method 100 continues at box 110 and enters a standby state. Figure 8 In the example shown in the second view 82b in FIG, the traffic sign 94 of interest is not found in the database because five miles per hour is not a valid speed limit sign type stored in the database. If the traffic sign 94 of interest is not found in the database, it is determined that a traffic sign interpretation error has been identified, and the method 100 proceeds to box 128, as will be discussed in more detail below.

[0093] Within the scope of the present disclosure, a traffic sign interpretation error is any failure of the plurality of vehicle sensors 16 and / or the autonomous driving system 24 to interpret a traffic sign. For example, if a traffic sign is obscured, dirty, vandalized, damaged, or otherwise defective, the plurality of vehicle sensors 16 and / or the autonomous driving system 24 may not be able to interpret the traffic sign, resulting in a traffic sign interpretation error. Additionally, a traffic sign interpretation error may also result from a lack of traffic signs. For example, if the vehicle 12 approaches a four-way intersection with no visible and / or interpretable traffic signs, the controller 14 may identify a traffic sign interpretation error.

[0094] At box 108, the controller 14 uses the plurality of vehicle sensors 16 and the automated driving system 24 to monitor the environment 40 surrounding the vehicle 12 for a lack of traffic signs. If the environment 40 is identified as lacking traffic signs, for example, if the environment 40 includes features such as intersections, junctions, end lanes, narrowing lanes, etc., without identifying appropriate traffic signs for these features, then a traffic sign interpretation error is identified and the method 100 proceeds to box 128. If no traffic signs are found to be lacking in the environment 40, the method 100 continues to enter a standby state at box 110.

[0095] At frame 128, the controller 14 determines a notification mode. The notification mode is based at least in part on the gaze direction of the vehicle occupant 44. In an exemplary embodiment, the occupant position tracking device 50 is used to determine the angle of the head 52a of the vehicle occupant 44, and the gaze direction is determined based at least in part on the angle of the head 52a of the vehicle occupant 44. In another exemplary embodiment, the occupant position tracking device 50 is used to perform eye tracking on the eyes 52b of the vehicle occupant 44, and the gaze direction is determined based at least in part on the eye tracking of the eyes 52b of the vehicle occupant 44. If the gaze direction is determined to be a front windshield gaze direction, the method 100 proceeds to frame 130. If the gaze direction is determined to be a non-front windshield gaze direction, the method 100 proceeds to frame 132.

[0096] refer to Figure 8 And continue to refer to Figure 6 At box 130, the controller 14 generates the confirmation graphic 90, as described above. The confirmation graphic 90 includes a sign position indicator 92a and a text confirmation message 92b. The sign position indicator 92a is positioned to highlight the traffic sign 94 of interest. If the environment 40 is identified as lacking traffic signs, as discussed above with reference to box 108, the sign position indicator 92a can be omitted from the confirmation graphic 90. The text confirmation message 92b is configured to prompt the vehicle occupant 44 to resolve the traffic sign interpretation error. In an exemplary embodiment, the text confirmation message 92b includes a message such as "The speed limit is interpreted as 5MPH, is this correct?" or "Traffic signs are missing, please explain the correct action."

[0097] In an exemplary embodiment, the vehicle occupant 44 provides additional information and / or confirmation to the vehicle 12 to resolve the traffic sign interpretation error by using, for example, voice commands received by a microphone, gesture commands received by an interior camera, interacting with physical buttons and / or the HMI 22, etc. In another exemplary embodiment, the automated driving system 24 is configured to resolve the traffic sign interpretation error using an error handling algorithm (e.g., a machine learning algorithm). After frame 130, the method 100 proceeds to frame 134.

[0098] At box 134, the controller 14 uses at least one of the AR-HUD system 18, the TWD system 20, and the HMI 22 to display the confirmation graphic 90 to the vehicle occupant 44. To display the confirmation graphic 90, the controller 14 first determines the location of the vehicle occupant 44 using the occupant position tracking device 50, as described above. After determining the location of the vehicle occupant 44, the controller 14 calculates the size, shape, and position of the AR-HUD graphic 46 (i.e., the confirmation graphic 90), as described above. In some examples, as described above, the position of the AR-HUD graphic 46 is selected so that from the perspective of the vehicle occupant 44, the sign position indicator 92a appears as an overlay on the traffic sign of interest 94, such as Figure 8 As shown in the second view 82b.

[0099] After calculating the size, shape, and position of the AR-HUD graphics 46 (i.e., confirming the graphics 90), the controller 14 uses at least one of the AR-HUD system 18 and the TWD system 20 to display the AR-HUD graphics 46, as described above. In an exemplary embodiment, the controller 14 also uses the HMI 22 to provide an augmented view of the environment 40 around the vehicle 12, including the AR-HUD graphics 46. After frame 134, the method 100 continues at frame 110 to enter the standby state.

[0100] At block 132, the controller 14 uses the haptic feedback device 26 to provide haptic feedback to the vehicle occupant 44 to notify the vehicle occupant 44 of the traffic sign interpretation error. In a non-limiting example, one or more aspects of the haptic feedback, such as intensity, location, duration, etc., are adjusted based at least in part on the traffic sign interpretation error, such as based on the nature of the traffic sign interpretation error (i.e., occluded traffic sign vs. absent traffic sign). After block 132, the method 100 continues at block 110 to enter a standby state.

[0101] In an exemplary embodiment, the controller 14 repeatedly exits the standby state 110 and restarts the method 100 at block 102. In a non-limiting example, the controller 14 exits the standby state 110 and restarts the method 100 based on a timer, such as every three hundred milliseconds.

[0102] The system 10 and method 100 of the present disclosure have several advantages. By providing a warning notification and / or tactile feedback to the vehicle occupant 44 regarding a planned change in driving behavior, the system 10 increases occupant awareness, allowing the vehicle occupant 44 an opportunity to correct or prevent the planned change in driving behavior. Using XAI, the reasoning behind the planned change in driving behavior is provided to the vehicle occupant 44, enabling the vehicle occupant 44 to evaluate whether to intervene to correct or prevent the planned change in driving behavior. By providing notifications to the remote vehicle occupant using the rear display 28, the remote vehicle occupant is informed of the planned change in driving behavior, thereby increasing the awareness of the remote vehicle occupant. By providing a confirmation notification to the vehicle occupant 44 regarding a traffic sign interpretation error, the vehicle occupant 44 can provide additional information to the autonomous driving system 18, thereby improving the operation of the autonomous driving system 18.

[0103] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A system for providing information to a vehicle occupant of a vehicle, comprising: Autonomous driving systems; a first display configured to provide information to a vehicle occupant of the vehicle; as well as a controller in electrical communication with the automated driving system and the first display, wherein the controller is programmed to: identifying a planned change in driving behavior initiated by the automated driving system; identifying a traffic sign interpretation error using the automated driving system; as well as The vehicle occupant is notified using the first display in response to identifying at least one of: the planned change in driving behavior and the traffic sign interpretation error.

2. The system according to claim 1, wherein: In order to identify a planned change in the driving behavior, the controller is further programmed to: A path planning algorithm of the automated driving system is monitored for the planned change in driving behavior, wherein the planned change in driving behavior includes at least one of: an emergency stop, sudden acceleration, a lane change, and a merging lane.

3. The system of claim 1 further comprising a plurality of vehicle sensors in electrical communication with the controller, wherein: To identify the traffic sign interpretation error, the controller is further programmed to: detecting a traffic sign of interest using the plurality of vehicle sensors; searching a database of known traffic signs for the traffic sign of interest; as well as In response to determining that the traffic sign of interest is not found within the database of known traffic signs, the traffic sign interpretation error is identified.

4. The system according to claim 1, wherein: To notify the vehicle occupants, the controller is also programmed to: generating a graph based at least in part on at least one of: the planned change in driving behavior and the traffic sign interpretation error; as well as The graphic is displayed to the vehicle occupant using the first display.

5. The system according to claim 4, wherein: To generate the graph, the controller is also programmed to: generating a warning graphic in response to identifying the planned change in driving behavior, wherein the warning graphic is configured to warn the vehicle occupant of the planned change in driving behavior; and A confirmation graphic is generated in response to identifying the traffic sign interpretation error, wherein the confirmation graphic is configured to prompt the vehicle occupant to resolve the traffic sign interpretation error.

6. The system according to claim 5, wherein: In order to generate the warning graphic, the controller is also programmed to: determining an explanation for the planned change in driving behavior using an explainable artificial intelligence (XAI) algorithm; identifying an object of interest in the vehicle's surroundings, the object causing the automated driving system to initiate a planned change in the driving behavior based at least in part on an interpretation; as well as The warning graphic is generated, wherein the warning graphic appears overlaid on the object of interest from the perspective of the vehicle occupant.

7. The system according to claim 4, wherein: The first display is an augmented reality head-up display (AR-HUD) system in electrical communication with the controller, wherein the AR-HUD system includes an occupant position tracking device and an AR-HUD projector, and wherein, to display the graphics, the controller is further programmed to: determining a position of a vehicle occupant of the vehicle using the occupant position tracking device; calculating a size, shape and position of the graphic based on the position of the vehicle occupant; and The graphic is displayed on a front windshield of the vehicle based on the size, shape, and location using the augmented reality head-up display system.

8. The system according to claim 7, wherein: The first display further comprises a transparent windshield display (TWD) system in electrical communication with the controller, wherein the transparent windshield display system comprises a transparent phosphor and a TWD projector embedded in the windshield of the vehicle, and wherein, to display the graphics, the controller is further programmed to: calculating the size, shape and position of the graphic; and Using the TWD system, the graphic is displayed on the front windshield of the vehicle based on the size, shape, and location.

9. The system of claim 1 further comprising a second display in electrical communication with the controller, wherein: The second display is configured to provide information to a remote vehicle occupant, and wherein the controller is further programmed to: The remote vehicle occupant is notified using the second display in response to identifying the planned change in driving behavior.

10. The system of claim 1 further comprising a haptic feedback device in electrical communication with the controller, and wherein: The controller is also programmed to: The vehicle occupant is notified using the haptic feedback device in response to identifying the planned change in driving behavior.

Images