Displaying automated driving system status for occupant perception
Through the combination of autonomous driving system and vehicle sensors, the AR-HUD system displays the automated confidence level graph of the target position on the vehicle windshield, solving the problem that the prior art cannot provide vehicle system operation information, and improving the environmental perception of occupants and Operational assistance.
Patent Information
- Application Number
- CN202311407555.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-31
- Filing Date
- 2023-10-26
- Publication Date
- 2024-12-03
AI Technical Summary
Existing augmented reality head-up displays (AR-HUDs) are unable to provide vehicle occupants with additional information on the operation of vehicle systems and components, limiting occupants’ environmental perception and operational assistance.
Through the autonomous driving system, multiple vehicle sensors and augmented reality head-up display (AR-HUD) system, the controller uses multiple measurements of the vehicle's surroundings to determine the automation confidence level of the target position, and generate different graphics based on this level to notify The occupant, displayed on the windshield, provides visual assistance.
It improves the occupants' perception of the vehicle system and environment, enhances the occupants' operation assistance, provides the status information of the autonomous driving system visually, and improves the occupants' perception and comfort.
Smart Images

Figure CN119058738A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an augmented reality heads-up display for displaying information to an occupant of a vehicle. Background Art
[0002] Augmented reality (AR) involves augmenting the real world with virtual elements that are displayed in three-dimensional space and allow real-time interaction with the user. A head-up display (HUD) displays information such as vehicle speed and navigation instructions directly on the vehicle windshield in the occupant's forward field of view. The head-up display thus provides information to the occupant without the occupant having to take their eyes off the road. One possible implementation of augmented reality is an augmented reality head-up display (AR-HUD) for a vehicle. By superimposing images on the windshield, the AR-HUD enhances the occupant's view of the vehicle's external environment, resulting in greater situational awareness. However, current AR-HUDs are unable to provide occupants with additional information about the operation of vehicle systems and components.
[0003] Therefore, while current augmented reality heads-up displays achieve their intended purposes, there is a need in the art for an improved method for providing information to vehicle occupants. Summary of the invention
[0004] According to several aspects, a system for providing information to an occupant of a vehicle is provided. The system includes an automated driving system, a display configured to provide information to the occupant, and a controller in electrical communication with the automated driving system and the display. The controller is programmed to determine, using the automated driving system, an automation confidence level for a target location. The target location is a location in an environment surrounding the vehicle. The controller is also programmed to notify the occupant using the display based at least in part on the automation confidence level.
[0005] In another aspect of the present disclosure, the system further includes a plurality of vehicle sensors in electrical communication with the controller. To determine the automation confidence level of the target location, the controller is further programmed to perform a plurality of measurements of the vehicle surroundings using the plurality of vehicle sensors. To determine the automation confidence level of the target location, the controller is further programmed to determine the automation confidence level of the target location using the autopilot system based at least in part on the plurality of measurements.
[0006] In another aspect of the present disclosure, to determine the automation confidence level of a target location using an autonomous driving system, the controller is further programmed to determine the automation confidence level of the target location based at least in part on at least one of environmental conditions and road conditions.
[0007] In another aspect of the present disclosure, in order to notify the occupant using the display, the controller is further programmed to display a notification graphic superimposed on the target location using the display.
[0008] In another aspect of the present disclosure, in order to display a notification graphic superimposed on the target location using the display, the controller is also programmed to compare the automation confidence level of the target location with a first automation confidence level threshold. In order to display a notification graphic superimposed on the target location using the display, the controller is also programmed to generate a first graphic in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold. In order to display a notification graphic superimposed on the target location using the display, the controller is also programmed to generate a second graphic in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold. In order to display a notification graphic superimposed on the target location using the display, the controller is also programmed to display at least one of the first graphic and the second graphic using the display.
[0009] In another aspect of the present disclosure, in order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to compare the automation confidence level of the target location with a first automation confidence level threshold. In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to compare the automation confidence level of the target location with a second automation confidence level threshold. In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to generate a first graphic in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold. In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to generate a second graphic in response to determining that the automation confidence level of the target location is less than the second automation confidence level threshold. In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to generate a third graphic in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold and greater than or equal to the second automation confidence level threshold. In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to display at least one of the first graphic, the second graphic, and the third graphic using the display.
[0010] In another aspect of the invention, the first graphic includes a first distinguishing feature, the second graphic includes a second distinguishing feature, and the third graphic includes a third distinguishing feature. The first, second, and third distinguishing features include at least one of graphic color, graphic size, graphic shape, and graphic texture. The first, second, and third distinguishing features are different from each other.
[0011] In another aspect of the present disclosure, the display is an augmented reality head-up display (AR-HUD) system in electrical communication with the controller. The AR-HUD system includes an occupant position tracking device and an AR-HUD projector. In order to display at least one of the first, second, and third graphics, the controller is also programmed to determine the position of the vehicle occupant using the occupant position tracking device. In order to display at least one of the first, second, and third graphics, the controller is also programmed to calculate the size, shape, and position of at least one of the first, second, and third graphics based on the position of the occupant and the target position. In order to display at least one of the first, second, and third graphics, the controller is also programmed to display at least one of the first, second, and third graphics on the windshield of the vehicle using the augmented reality head-up display system based on the size, shape, and position.
[0012] In another aspect of the present disclosure, the display also includes a transparent windshield display (TWD) system in electrical communication with the controller. The transparent windshield display system includes a transparent phosphor and a TWD projector embedded in the vehicle windshield. In order to display at least one of the first, second, and third graphics, the controller is also programmed to calculate the size, shape, and position of at least one of the first, second, and third graphics based on the target position. In order to display at least one of the first, second, and third graphics, the controller is also programmed to display at least one of the first, second, and third graphics on the windshield of the vehicle using the TWD system based on size, shape, and position.
[0013] In another aspect of the present disclosure, the autonomous driving system is configured to be used in at least one of a manual operation mode, a partially automated operation mode, and a fully automated operation mode.
[0014] According to several aspects, a method for providing information to an occupant of a vehicle is provided. The method includes: determining, using an automated driving system, an automation confidence level of a target location. The target location is a location in an environment surrounding the vehicle. The method also includes: notifying the occupant using a display based at least in part on the automation confidence level.
[0015] In another aspect of the present disclosure, determining the automation confidence level of the target location may further include: utilizing multiple vehicle sensors to perform multiple measurements of the vehicle surroundings. Determining the automation confidence level of the target location may further include: utilizing the autonomous driving system to determine the automation confidence level of the target location based at least in part on the multiple measurements.
[0016] In another aspect of the present disclosure, determining the automation confidence level of the target location may also include: determining the automation confidence level of the target location based at least in part on at least one of traffic conditions in the vehicle's surroundings, pedestrians present in the vehicle's surroundings, and road conditions in the vehicle's surroundings.
[0017] In another aspect of the present disclosure, notifying the occupant using the display may further include: displaying a notification graphic superimposed on the target location using the display.
[0018] In another aspect of the present disclosure, using a display to display a notification graphic superimposed on a target location may further include: comparing the automation confidence level of the target location with a first automation confidence level threshold. Using a display to display a notification graphic superimposed on a target location may further include: generating a first graphic in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold. Using a display to display a notification graphic superimposed on a target location may further include: generating a second graphic in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold. Using a display to display a notification graphic superimposed on a target location may further include: displaying at least one of the first graphic and the second graphic using a display.
[0019] In another aspect of the present disclosure, displaying a notification graphic superimposed on a target location using a display may further include: comparing the automation confidence level of the target location with a first automation confidence level threshold. Displaying a notification graphic superimposed on a target location using a display may further include: comparing the automation confidence level of the target location with a second automation confidence level threshold. Displaying a notification graphic superimposed on a target location using a display may further include: generating a first graphic in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold. Displaying a notification graphic superimposed on a target location using a display may further include: generating a second graphic in response to determining that the automation confidence level of the target location is less than the second automation confidence level threshold. Displaying a notification graphic superimposed on a target location using a display may further include: generating a third graphic in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold and greater than or equal to the second automation confidence level threshold. Displaying a notification graphic superimposed on a target location using a display may further include: displaying at least one of the first graphic, the second graphic, and the third graphic using a display.
[0020] In another aspect of the present disclosure, displaying at least one of the first, second, and third graphics may also include: determining the position of the driver of the vehicle using a driver position tracking device. Displaying at least one of the first, second, and third graphics may also include: calculating the size, shape, and position of at least one of the first, second, and third graphics based on the driver's position and the target position. Displaying at least one of the first, second, and third graphics may also include: displaying at least one of the first, second, and third graphics on the windshield of the vehicle using an augmented reality head-up display projector based on the size, shape, and position. Displaying at least one of the first, second, and third graphics may also include: displaying at least one of the first, second, and third graphics on the windshield of the vehicle using a transparent windshield display (TWD) system based on the size, shape, and position.
[0021] According to several aspects, a system for providing information to an occupant of a vehicle is provided. The system includes an autonomous driving system, an augmented reality head-up display (AR-HUD), a plurality of vehicle sensors, and a controller in electrical communication with the autonomous driving system, the AR-HUD, and the plurality of vehicle sensors. The controller is programmed to take multiple measurements of the environment around the vehicle using the plurality of vehicle sensors. The multiple measurements include at least one of traffic conditions in the environment around the vehicle, pedestrians present in the environment around the vehicle, and road conditions in the environment around the vehicle. The controller is also programmed to determine an automation confidence level of a target location using the autonomous driving system based at least in part on the multiple measurements. The controller is also programmed to notify the occupant using the AR-HUD based at least in part on the automation confidence level.
[0022] In another aspect of the present disclosure, in order to notify the occupant using the AR-HUD, the controller is also programmed to compare the automation confidence level of the target location with the first automation confidence level threshold. In order to notify the occupant using the AR-HUD, the controller is also programmed to compare the automation confidence level of the target location with the second automation confidence level threshold. In order to notify the occupant using the AR-HUD, the controller is also programmed to generate a first graphic in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold. In order to notify the occupant using the AR-HUD, the controller is also programmed to generate a second graphic in response to determining that the automation confidence level of the target location is less than the second automation confidence level threshold. In order to notify the occupant using the AR-HUD, the controller is also programmed to generate a third graphic in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold and greater than or equal to the second automation confidence level threshold. In order to notify the occupant using the AR-HUD, the controller is also programmed to display at least one of the first graphic, the second graphic, and the third graphic using the display.
[0023] In another aspect of the invention, the first graphic includes a first distinguishing feature, the second graphic includes a second distinguishing feature, and the third graphic includes a third distinguishing feature. The first, second, and third distinguishing features include at least one of graphic color, graphic size, graphic shape, and graphic texture. The first, second, and third distinguishing features are different from each other.
[0024] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.
[0026] Figure 1 is a schematic diagram of a system for providing information to an occupant of a vehicle according to an exemplary embodiment;
[0027] Figure 2 is a schematic diagram of an AR-HUD system for use by an exemplary occupant according to an exemplary embodiment;
[0028] Figure 3 is a schematic front view of a dual focal plane augmented reality display according to an exemplary embodiment, highlighting a second image plane of the dual focal plane augmented reality display;
[0029] Figure 4 is a schematic diagram of a second image plane of a dual focal plane augmented reality display according to an exemplary embodiment;
[0030] Figure 5 is a first exemplary driving scenario according to an exemplary embodiment;
[0031] Figure 6 is a flow chart of a method for providing information to an occupant of a vehicle according to an exemplary embodiment; and
[0032] Figure 7 is a second exemplary driving scenario according to an exemplary embodiment. DETAILED DESCRIPTION
[0033] The following description is merely exemplary in nature and is not intended to limit the present invention, application, or uses.
[0034] Automated driving systems with various levels of capability may be employed in a vehicle. In one non-limiting example, an automated driving system may be configured to provide visual, auditory, and / or tactile feedback to a vehicle occupant to assist in the driving task. In one non-limiting example, an automated driving system may be configured to control portions of a vehicle system or all of a vehicle system in certain circumstances to provide a partially automated driving experience. In yet another non-limiting example, an automated driving system may be configured to take full control of the vehicle. An automated driving system may provide a confidence value indicating the ability of the automated driving system to control the vehicle at any given position. The present disclosure provides a new and improved system and method for visually providing information about the confidence of an automated driving system to an occupant, thereby enhancing the occupant's perception.
[0035] refer to Figure 1 , a system for providing information to an occupant of a vehicle is shown, generally indicated by the reference numeral 10. The system 10 is shown with an exemplary vehicle 12. Although a passenger vehicle is shown, it should be appreciated that the vehicle 12 may be any type of vehicle without departing from the scope of the present disclosure. The system 10 generally includes a controller 14, a plurality of vehicle sensors 16, an augmented reality head-up display (AR-HUD) system 18, a transparent windshield display (TWD) system 20, a human-machine interface (HMI) 22, and an autonomous driving system 24.
[0036] As described below, the controller 14 is used to implement a method 100 for providing information to an occupant of a vehicle. The controller 14 includes at least one processor 26 and a non-transitory computer-readable storage device or medium 28. The processor 26 can be a custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), a spare processor among several processors associated with the controller 14, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, or a combination thereof, or generally a device for executing instructions. For example, the computer-readable storage device or medium 28 can include volatile and non-volatile storage in a read-only memory (ROM), a random access memory (RAM), and a keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the processor 26 is powered off. The computer readable storage device or medium 28 may be implemented using a variety of known storage devices, such as PROM (Programmable Read Only Memory), EPROM (Electrically Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory, or other electrical, magnetic, optical, or combination storage devices capable of storing data (some of which represents executable instructions used by the controller 14 to control various systems of the vehicle 12). The controller 14 may also include multiple controllers that are in electrical communication with each other. The controller 14 may be interconnected with additional systems and / or controllers of the vehicle 12, thereby allowing the controller 14 to access data (such as the speed, acceleration, braking, and steering angle of the vehicle 12).
[0037] The controller 14 is in electrical communication with a plurality of vehicle sensors 16, an AR-HUD system 18, a TWD system 20, an HMI 22, and an autonomous driving system 24. In an exemplary embodiment, electrical communication is established using, for example, a CAN network, a FLEXRAY network, a local area network (e.g., WiFi, Ethernet, etc.), a serial peripheral interface (SPI) network, etc. It should be understood that various additional wired and wireless technologies and communication protocols for communicating with the controller 14 are within the scope of the present disclosure.
[0038] The plurality of vehicle sensors 16 are used to gather information about an environment 30 surrounding the vehicle 12 . In an exemplary embodiment, the plurality of vehicle sensors 16 include at least a camera system 32 , a vehicle communication system 34 , and a global navigation satellite system (GNSS) 36 .
[0039] In another exemplary embodiment, the plurality of vehicle sensors 16 further include sensors that determine performance data regarding the vehicle 12. In an exemplary embodiment, the plurality of vehicle sensors 16 further include at least one of a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor.
[0040] In another exemplary embodiment, the plurality of vehicle sensors 16 also include sensors that determine information about the interior environment of the vehicle 12 (eg, seat occupancy sensors, cabin air temperature sensors, cabin motion detection sensors, cabin cameras and / or cabin microphones, etc.).
[0041] In another exemplary embodiment, the plurality of vehicle sensors 16 also includes sensors that determine information about the environment surrounding the vehicle 12 (eg, an ambient air temperature sensor, an atmospheric pressure sensor, and / or a photo and / or video camera positioned to view the environment in front of the vehicle 12 ).
[0042] In another exemplary embodiment, at least one of the plurality of vehicle sensors 16 is capable of measuring distances in the environment surrounding the vehicle 12. In one non-limiting example, the plurality of vehicle sensors 16 include a stereo camera with distance measurement capabilities. In one example, at least one of the plurality of vehicle sensors 16 is fixed to the interior of the vehicle 12 (e.g., fixed to the roof lining of the vehicle 12) to obtain a view through the windshield of the vehicle 12. In another example, at least one of the plurality of vehicle sensors 16 is fixed to the exterior of the vehicle 12 (e.g., fixed to the roof of the vehicle 12) to obtain a view of the environment surrounding the vehicle 12. It should be understood that various other types of vehicle sensors (such as, for example, lidar sensors, ultrasonic ranging sensors, radar sensors, and / or time-of-flight sensors) are within the scope of the present disclosure. As described above, the plurality of vehicle sensors 16 are in electrical communication with the controller 14.
[0043] The camera system 32 is used to capture images and / or video of the environment 30 surrounding the vehicle 12. In one exemplary embodiment, the camera system 32 includes a photo and / or video camera positioned to view the environment 30 surrounding the vehicle 12. In one non-limiting example, the camera system 32 includes a camera mounted inside the vehicle 12 (e.g., mounted in the headliner of the vehicle 12) to obtain a view through the windshield 38. In another non-limiting example, the camera system 32 includes a camera mounted outside the vehicle 12 (e.g., mounted on the roof of the vehicle 12) to obtain a view of the environment 30 in front of the vehicle 12.
[0044] In another exemplary embodiment, the camera system 32 is a surround view camera system that includes multiple cameras (also referred to as satellite cameras) that are arranged to provide views of the environment 30 adjacent to all sides of the vehicle 12. In one non-limiting example, the camera system 32 includes a front camera (e.g., mounted in the front grille of the vehicle 12), a rear camera (e.g., mounted on the tailgate of the vehicle 12), and two side cameras (e.g., mounted below each of the two side mirrors of the vehicle 12). In another non-limiting example, the camera system 32 also includes an additional rear view camera mounted near the center high mounted brake light of the vehicle 12.
[0045] It should be understood that camera systems with additional cameras and / or additional mounting locations are within the scope of the present disclosure. It should also be understood that cameras with various sensor types (e.g., including charge coupled device (CCD) sensors, complementary metal oxide semiconductor (CMOS) sensors, and / or high dynamic range (HDR) sensors) are within the scope of the present disclosure. In addition, cameras with various lens types (e.g., including wide angle lenses and / or narrow angle lenses) are also within the scope of the present disclosure.
[0046] The controller 14 utilizes the vehicle communication system 34 to communicate with other systems external to the vehicle 12. For example, the vehicle communication system 34 includes the ability to communicate with vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems located at remote call centers (e.g., General Motors' ON-STAR), and / or personal devices. In general, the term vehicle-to-everything communication ("V2X" communication) refers to communication between the vehicle 12 and any remote system (e.g., vehicle, infrastructure, and / or remote system). In some embodiments, the vehicle communication system 34 is a wireless communication system that is configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communication (e.g., using GSMA standards (such as, for example, SGP.02, SGP.22, SGP.32, etc.)). Therefore, the vehicle communication system 34 may also include an embedded universal integrated circuit card (eUICC) that is configured to store at least one cellular connection configuration profile (e.g., an embedded subscriber identity module (eSIM) profile). The vehicle communication system 34 is also configured to communicate via a personal area network (e.g., Bluetooth) and / or near field communication (NFC). However, additional or alternative communication methods (such as dedicated short-range communication (DSRC) channels and / or mobile telecommunication protocols based on the 3rd Generation Partnership Project (3GPP) standard) are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range wireless communication channel specifically designed for automobiles and a corresponding set of protocols and standards. 3GPP refers to a partnership between several standard organizations that develop protocols and standards for mobile telecommunications. The 3GPP standards are structured as "versions." Therefore, communication methods based on 3GPP versions 14, 15, 16 and / or future 3GPP versions are considered to be within the scope of the present disclosure. Therefore, the vehicle communication system 34 may include one or more antennas and / or communication transceivers for receiving and / or sending signals, such as collaborative sensing messages (CSM). The vehicle communication system 34 is configured to perform wireless information communication between the vehicle 12 and another vehicle. In addition, the vehicle communication system 34 is configured to perform wireless information communication between the vehicle 12 and infrastructure or other vehicles. It should be appreciated that the vehicle communication system 34 may be integrated with the controller 14 (eg, integrated on the same circuit board as the controller 14 or be a part of the controller 14) without departing from the scope of the present disclosure.
[0047] GNSS 36 is used to determine the geographic location of vehicle 12. In an exemplary embodiment, GNSS 36 is a global positioning system (GPS). In a non-limiting example, GPS includes a GPS receiver antenna (not shown) and a GPS controller (not shown) in electrical communication with the GPS receiver antenna. The GPS receiver antenna receives signals from a plurality of satellites, and the GPS controller calculates the geographic location of vehicle 12 based on the signals received by the GPS receiver antenna. In an exemplary embodiment, GNSS 36 also includes a map. The map includes information about infrastructure such as municipal boundaries, roads, railways, sidewalks, buildings, etc. Therefore, map information is used to put the geographic location of vehicle 12 in context. In a non-limiting example, a wireless connection is used to retrieve the map from a remote source. In another non-limiting example, the map is stored in a database of GNSS 36. It should be understood that various other types of satellite-based radio navigation systems (such as, for example, the Global Positioning System (GPS), the Galileo Satellite Navigation System (Galileo), the GLONASS Satellite Navigation System (GLONASS) and the Beidou Navigation Satellite System (BDS)) are within the scope of the present disclosure. It should be appreciated that the GNSS 36 may be integrated with the controller 14 (eg, integrated on the same circuit board as the controller 14 or be a part of the controller 14) without departing from the scope of the present disclosure.
[0048] refer to Figure 2 , a system diagram of an AR-HUD system 18 for use with an exemplary occupant 40 is shown. Within the scope of the present disclosure, in one non-limiting example, the occupant 40 includes a driver, a passenger, and / or any other person in the vehicle 12. The AR-HUD system 18 is used to display AR-HUD graphics 42 (i.e., notification symbols that provide visual information to the occupant 40) on the windshield 38 of the vehicle 12. The AR-HUD system 18 includes an AR-HUD projector 44 and an occupant position tracking device 46. As described above, the AR-HUD system 18 is in electrical communication with the controller 14.
[0049] The AR-HUD projector 44 is used to project the AR-HUD graphics 42 onto the windshield 38 of the vehicle 12. It should be understood that various devices designed to project images (including, for example, optical collimators, laser projectors, digital light projectors (DLP) and the like) are within the scope of the present disclosure.
[0050] The occupant position tracking device 46 is used to determine the position of the occupant 40 within the vehicle 12. For example, the occupant position tracking device 46 can track the position of the head 40a or the eyes 40b of the occupant 40. The position of the occupant 40 within the vehicle 12 from the occupant position tracking device 46 is used to position the AR-HUD graphics 42 on the windshield 38 of the vehicle 12. In an exemplary embodiment, the occupant position tracking device 46 is one or more cameras disposed within the vehicle 12.
[0051] To operate the AR-HUD system 18, the controller 14 includes a plurality of software modules (including a system manager 48). During operation of the system 10, the system manager 48 receives at least a first input 50, a second input 52, and a third input 54. The first input 50 indicates the position of the vehicle 12 in space (i.e., the geographic location of the vehicle 12), the second input 52 indicates the position of the vehicle occupant 40 within the vehicle 12 (e.g., the position of the eyes and / or head of the occupant 40 within the vehicle 12), and the third input 54 is data related to the automation confidence level in the environment 30 surrounding the vehicle 12, which will be described in more detail below. The first input 50 may include data such as GNSS data (e.g., GPS data), vehicle speed, road curvature, and vehicle steering, and the data is collected from a plurality of vehicle sensors 16. The second input 52 is received from the occupant position tracking device 46. The third input 54 is data related to the automation confidence level in the environment 30 surrounding the vehicle 12. The system manager 48 is configured to determine (e.g., calculate) the type, size, shape, and color of the AR-HUD graphics 42 to be displayed using the AR-HUD projector 44 based on the first input 50 (i.e., the vehicle position in the environment 30), the second input 52 (e.g., the position of the eyes 40b and / or the head 40a of the occupant 40 in the vehicle 12), and the third input 54 (i.e., the automation confidence level in the environment 30 surrounding the vehicle 12). The system manager 48 commands the image engine 56, which is a software module or integrated circuit of the AR-HUD projector 44 or the controller 14, to display the AR-HUD graphics 42 using the AR-HUD projector 44. Based on the type, size, shape, and color of the AR-HUD graphics 42 determined by the system manager 48, the image engine 56 displays the AR-HUD graphics 42 on the windshield 38 of the vehicle 12 using the AR-HUD projector 44. The AR-HUD graphics 42 are projected onto the windshield 38 by the AR-HUD projector 44 to display the AR-HUD graphics 42 along the road surface 58 .
[0052] In an exemplary embodiment of the present disclosure, the AR-HUD system 18 is a dual focal plane AR-HUD system. Figure 3 and Figure 4 , while continuing to refer to Figure 2, the AR-HUD system 18 has a first image plane 60 and a second image plane 62. The first image plane 60 displays a view of the outside world, while the second image plane 62 is reserved for displaying the AR-HUD graphics 42. The second image plane 62 spans multiple lanes, and the AR-HUD graphics 42 appear at a farther position on the road surface 58 relative to the first image plane 60. For example, Figure 3 and Figure 4 As shown, the second image plane 62 superimposes the left lane 64, the center lane 66, and the right lane 68. As a non-limiting example, in the center lane 66, the second image plane 62 starts at a first predetermined distance D1 (e.g., 25 meters) from the vehicle 12 and ends at a second predetermined distance D2 (e.g., 90 meters) from the vehicle 12. Regardless of the specific distance, the second predetermined distance D2 is greater than the first predetermined distance D1 to help the occupant 40 see the AR-HUD graphics 42 displayed by the AR-HUD projector 44. In the left lane 64 and the right lane 68, the second image plane 62 is defined by an inclined boundary that starts at the first predetermined distance D1 from the vehicle 12 and ends at a third predetermined distance D3 (e.g., 50 meters) from the vehicle 12. The third predetermined distance D3 is greater than the first predetermined distance D1 and less than the second predetermined distance D2 to help the occupant 40 see the AR-HUD graphics 42 displayed by the AR-HUD projector 44. As used herein, the term "dual focal plane AR-HUD" refers to an AR-HUD system that presents images in a first image plane and a second image plane, wherein the first image plane and the second image plane are located at different positions. It is necessary to construct the AR-HUD system 18 as a dual focal plane AR-HUD to facilitate the manipulation of the AR-HUD graphics 42 on the scene of the outside world. For example, by utilizing a dual focal plane AR-HUD, the size, position, and features of the AR-HUD graphics 42 can be changed based on, for example, the position of the eye 40b of the occupant 40.
[0053] The TWD system 20 is used to display images on the windshield 38 of the vehicle 12. In an exemplary embodiment, the AR-HUD system 18 can display the AR-HUD graphics 42 in a predetermined area of the windshield 38 (e.g., in the first image plane 60 and the second image plane 62). The TWD system 20 can display TWD graphics (not shown) in any area of the windshield 38. Therefore, by jointly operating the AR-HUD system 18 and the TWD system 20, the controller 14 can display graphics in any area of the windshield 38. In an exemplary embodiment, the TWD system 20 includes a transparent phosphor (not shown) embedded in the windshield 38 and a TWD projector 70 ( Figure 1 As described above, the TWD system 20 is in electrical communication with the controller 14 .
[0054] The transparent phosphor is a luminescent particle that emits fluorescence in response to being excited by the TWD projector 70. In an exemplary embodiment, the transparent phosphor is a red, green and blue (RGB) phosphor, thereby allowing full-color operation of the TWD system 20. The use of monochromatic and / or bichromatic phosphors is also within the scope of the present disclosure. When the excitation light is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light can be, for example, violet light (ranging between about 380 nanometers and 450 nanometers) and / or ultraviolet light in the visible spectrum.
[0055] The TWD projector 70 is used to excite the transparent phosphors in a predetermined pattern to produce a TWD graphic on the windshield 38. In one exemplary embodiment, the TWD projector 70 is a violet / ultraviolet laser projector disposed near the headliner of the vehicle 12. The TWD projector 70 includes three lasers, each of which is configured to excite one of the red, green, or blue transparent phosphors.
[0056] In one exemplary embodiment, in addition to the AR-HUD system 18 and the TWD system 20, the HMI 22 is used to display information to the occupants 40 of the vehicle 12. In another exemplary embodiment, as a substitute for the AR-HUD system 18 and / or the TWD system 20, the HMI 22 is used to display information to the occupants 40 of the vehicle 12. In the above exemplary embodiments, the HMI 22 is a display system located within the field of view of the occupants 40 and is capable of displaying text, graphics and / or images. It should be understood that HMI display systems (including LCD displays, LED displays, etc.) are within the scope of the present disclosure. Other exemplary embodiments in which the HMI 22 is disposed in the rearview mirror are also within the scope of the present disclosure. As described above, the HMI 22 is in electrical communication with the controller 14.
[0057] The automated driving system 24 is used to provide assistance to the occupants 40 to enhance the occupants' perception and / or control of the behavior of the vehicle 12. Within the scope of the present disclosure, the automated driving system 24 includes systems that provide any level of assistance to the occupants 40 (e.g., blind spot warnings, lane departure warnings, etc.) and systems that are capable of autonomously driving the vehicle 12 under some or all conditions. It should be understood that all levels of driving automation defined by, for example, SAE J3016 (i.e., SAE LEVEL 0, SAE LEVEL 1, SAE LEVEL 2, SAE LEVEL 3, SAE LEVEL 4, and SAE LEVEL 5) are within the scope of the present disclosure.
[0058] In an exemplary embodiment, the autopilot system 24 is configured to detect and / or receive information about the environment 30 surrounding the vehicle 12 and process the information to provide assistance to the occupants 40. In some embodiments, the autopilot system 24 is a software module executed on the controller 14. In other embodiments, the autopilot system 24 includes an independent autopilot system controller (similar to the controller 14) that is capable of processing information about the environment 30 surrounding the vehicle 12. In an exemplary embodiment, the autopilot system 24 can operate in a manual operation mode, a partially automated operation mode, and a fully automated operation mode.
[0059] Within the scope of the present disclosure, manual operation mode refers to the autonomous driving system 24 providing warnings or notifications to the occupants 40, but not directly intervening or controlling the vehicle 12. In one non-limiting example, the autonomous driving system 24 receives information from a plurality of vehicle sensors 16. Using techniques such as, for example, computer vision, the autonomous driving system 24 understands the environment 30 around the vehicle 12 and provides assistance to the occupants 40. For example, if the autonomous driving system 24 identifies that the vehicle 12 may collide with a distant vehicle based on data from the plurality of vehicle sensors 16, the autonomous driving system 24 may utilize the HMI 22, the AR-HUD system 18, and / or the TWD system 20 to issue a warning to the occupants 40.
[0060] Within the scope of the present disclosure, a partially automated mode of operation refers to the automated driving system 24 providing warnings or notifications to the occupants 40 and may, in certain circumstances, directly intervene or control the vehicle 12. In one non-limiting example, the automated driving system 24 also electrically communicates with components of the vehicle 12 (e.g., the braking system, propulsion system, and / or steering system of the vehicle 12) so that the automated driving system 24 can control the behavior of the vehicle 12. In one non-limiting example, the automated driving system 24 may control the behavior of the vehicle 12 by applying brakes to the vehicle 12 to avoid an imminent collision.
[0061] Within the scope of the present disclosure, a fully automated mode of operation refers to the automated driving system 24 utilizing data from the plurality of vehicle sensors 16 to understand the environment 30 and control the vehicle 12 to drive the vehicle 12 to a predetermined destination without requiring control or intervention by the occupant 40 .
[0062] For a given location in the environment 30 surrounding the vehicle 12 (also referred to as a target location), the automated driving system 24 determines an automation confidence level. Within the scope of the present disclosure, the automation confidence level quantifies the confidence that the automated driving system 24 can inform the occupant 40 and / or control the vehicle 12 at a given location (i.e., the target location). For example, if the automation confidence level for the target location is high (e.g., close to 1), the automated driving system 24 determines that it is likely that the automated driving system 24 will be able to control the vehicle 12 at the target location. If the automation confidence level for the target location is low (e.g., close to 0), the automated driving system 24 determines that it is highly unlikely that the automated driving system 24 will be able to control the vehicle 12 at the target location, and intervention by the occupant 40 is required. The automated driving system 24 can use the automation confidence level to determine whether the occupant 40 must control the vehicle 12. The automation confidence level will be discussed in more detail below.
[0063] In one exemplary embodiment, the autonomous driving system 24 operates using a machine learning algorithm that is trained to output control signals for the vehicle 12 based on input data collected from the plurality of vehicle sensors 16. In another exemplary embodiment, the autonomous driving system 24 operates using a deterministic algorithm that is programmed to output control signals for the vehicle 12 based on data collected from the plurality of vehicle sensors 16. It should be understood that the autonomous driving system 24 may be any software and / or hardware module configured to operate in a manual operating mode, a partially automated operating mode, or a fully automated operating mode as described above.
[0064] refer to Figure 5 , a first exemplary driving scene 80 is shown, which shows the system 10 and method 100 in use. The first exemplary driving scene 80 includes an intersection 82 in the path of the vehicle 12. The first exemplary driving scene 80 also includes a distant vehicle 84 crossing the intersection 82. The system 10 and method 100 are used to generate a first graphic 86a, a second graphic 86b, and a third graphic 86c. Based on the automation confidence level, the first graphic 86a, the second graphic 86b, and the third graphic 86b are superimposed on multiple locations (i.e., target locations) of the environment 30. The generation and display of the first graphic 86a, the second graphic 86b, and the third graphic 86c will be discussed in more detail below. In an exemplary embodiment, the first graphic 86a includes a first distinguishing feature, the second graphic 86b includes a second distinguishing feature, and the third graphic 86c includes a third distinguishing feature. In a non-limiting example, the first, second, and third distinguishing features are at least one of a graphic color, a graphic size, a graphic shape, and / or a graphic texture. The first, second, and third distinguishing features are used to allow the occupant 40 to distinguish between the first graphic 86a, the second graphic 86b, and the third graphic 86c. Figure 5In the exemplary embodiment shown, the first, second and third distinguishing features are graphic colors. Thus, the first graphic 86a is green, the second graphic 86b is red, and the third graphic 86c is yellow. It should be understood that the use of alternative colors (such as, for example, blue and / or vermilion, etc.) is also within the scope of the present invention.
[0065] refer to Figure 6 , a flow chart of a method 100 for providing information to an occupant 40 of a vehicle 12 is provided. The method 100 starts at step 102 and proceeds to step 104. In step 104, the controller 14 uses multiple vehicle sensors 16 to make multiple measurements of the environment 30 around the vehicle 12. In an exemplary embodiment, the multiple measurements include measurements of environmental conditions (e.g., weather, time, etc.). In an exemplary embodiment, the multiple measurements include measurements of road conditions (e.g., traffic conditions, presence of pedestrians, lane marking quality, unexpected driving behavior, traffic signal status, etc.). In another exemplary embodiment, the multiple measurements also include information received from the vehicle communication system 34 (e.g., Internet data, V2V data and / or V2X data, etc.). After step 104, the method 100 proceeds to step 106.
[0066] In step 106, the controller 14 utilizes the automated driving system 24 to determine an automation confidence level for the target location. As described above, the target location is a location in the environment 30 surrounding the vehicle 12 (e.g., in the path of the vehicle 12 (e.g., twenty meters in front of the vehicle 12)). As described above, the automation confidence level quantifies the confidence that the automated driving system 24 can operate (i.e., notify the occupant 40 and / or control the vehicle 12) at the target location. In one exemplary embodiment, the controller 14 queries the automated driving system 24 for the automation confidence level for the target location. In another exemplary embodiment, the automated driving system 24 continuously transmits the automation confidence level for one or more target locations to the controller 14. As described above, the automation confidence level can be a normalized value between 0 and 1, where 0 represents no confidence and 1 represents complete confidence.
[0067] In one non-limiting example, the automation confidence level is determined based at least in part on the plurality of measurements performed in step 104. For example, in poor weather or lighting conditions, the data provided by the plurality of vehicle sensors 16 may not be sufficient for the automated driving system 24 to effectively control the vehicle 12, resulting in a reduced automation confidence level. In another example, unexpected actions of a remote vehicle may result in a reduced automation confidence level. After step 106, the method 100 proceeds to step 108.
[0068] In step 108, the controller 14 compares the automation confidence level determined in step 106 to a first automation confidence level threshold. In an exemplary embodiment, the first automation confidence level threshold is a high confidence threshold (e.g., 0.75). If the automation confidence level determined in step 106 is greater than or equal to the first automation confidence level threshold, the method 100 proceeds to step 110. If the automation confidence level determined in step 106 is less than the first automation confidence level threshold, the method 100 proceeds to step 112.
[0069] refer to Figure 5 , while continuing to refer to Figure 6 In step 110, the controller 14 generates a first graphic 86a. In an exemplary embodiment, the first graphic 86a is a green arrow, which indicates that the automation confidence level of the target position is relatively high. It should be understood that Figure 5 The illustrated first graphic 86a is merely exemplary in nature, and variations in the color, size, shape, and / or texture of the first graphic 86a are within the scope of the present invention. After step 110, the method 100 proceeds to step 114, which will be described in more detail below.
[0070] In step 112, the controller 14 compares the automation confidence level determined in step 106 to a second automation confidence level threshold. In an exemplary embodiment, the second automation confidence level threshold is a low confidence threshold (e.g., 0.25). If the automation confidence level determined in step 106 is less than the second automation confidence level threshold, the method 100 proceeds to step 116. If the automation confidence level determined in step 106 is greater than or equal to the second automation confidence level threshold, the method 100 proceeds to step 118.
[0071] refer to Figure 5 , while continuing to refer to Figure 6 In step 116, the controller 14 generates a second graphic 86b. In an exemplary embodiment, the second graphic 86b is a red arrow, which indicates that the automation confidence level of the target position is relatively low. It should be understood that Figure 5 The illustrated second graphic 86b is merely exemplary in nature, and variations in the color, size, shape, and / or texture of the second graphic 86b are within the scope of the present disclosure. After step 116, the method 100 proceeds to step 114, which will be described in more detail below.
[0072] refer to Figure 5 , while continuing to refer to Figure 6In step 118, the controller 14 generates a third graphic 86c. In an exemplary embodiment, the third graphic 86c is a yellow arrow, which indicates that the automation confidence level of the target position is relatively low. It should be understood that Figure 5 The third graphic 86c shown is merely exemplary in nature, and variations in the color, size, shape, and / or texture of the third graphic 86c are within the scope of the present invention. After step 118, the method 100 proceeds to step 114.
[0073] In step 114 , the controller 14 utilizes the occupant position tracking device 46 to determine the position of the occupant 40 , as described above. After step 114 , the method 100 proceeds to step 120 .
[0074] In step 120, as described above, the controller 14 calculates the size, shape, and position of the AR-HUD graphic 42 (i.e., one of the first graphic 86a, the second graphic 86b, and the third graphic 86c). The position of the AR-HUD graphic 42 is selected so that from the perspective of the occupant 40, the AR-HUD graphic 42 appears to be superimposed on the target location in the environment 30 surrounding the vehicle 12. After step 120, the method 100 proceeds to step 122.
[0075] In step 122, the controller 14 displays the AR-HUD graphic 42 (i.e., one of the first graphic 86a, the second graphic 86b, and the third graphic 86c) on the windshield 38 of the vehicle 12 using at least one of the AR-HUD system 18 and the TWD system 20, as described above. In an exemplary embodiment, the controller 14 also uses the HMI 22 to provide an augmented view of the vehicle 12 surrounding environment 30 including the AR-HUD graphic 42. In an exemplary embodiment, the controller 14 may employ alternative methods (e.g., auditory feedback and / or tactile feedback, etc.) to inform the occupant 40 of the automation confidence level without departing from the scope of the present disclosure. After step 122, the method 100 enters a standby state in step 124.
[0076] In an exemplary embodiment, the controller 14 repeatedly exits the standby state 124 and restarts the method 100 at step 102. In a non-limiting example, the controller 14 exits the standby state 124 and restarts the method 100 on a timer (e.g., every three hundred milliseconds). By repeatedly executing the method 100, the displayed graphics are updated to account for changing road conditions, and other graphics are displayed at other target locations in the environment 30 surrounding the vehicle 12.
[0077] refer to Figure 7, a second exemplary driving scenario 90 is shown, which shows the system 10 and method 100 in use. The second exemplary driving scenario 90 includes an intersection 82 in the path of the vehicle 12. The second exemplary driving scenario 90 also includes a distant vehicle 84 crossing the intersection 82. The system 10 and method 100 are used to generate a first graphic 86a and a second graphic 86b. As described above, based on the automation confidence level, the first graphic 86a, the second graphic 86b, and the third graphic 86b are superimposed on multiple locations (i.e., target locations) of the environment 30. The second exemplary driving scenario 90 is an exemplary time course of the first exemplary driving scenario 80. The distant vehicle 84 has further entered the intersection 82 and is approaching the path of the vehicle 12. Therefore, due to the repeated execution of the method 100, the graphics displayed by the AR-HUD system 18, the TWD system 20, and / or the HMI 22 have been updated to indicate that the occupant 40 must control the vehicle 12, so the red arrow (i.e., the second graphic 86b) is displayed.
[0078] The system 10 and method 100 of the present disclosure have several advantages. By providing the occupant 40 with information about the automation confidence level in a visual manner (i.e., through the display of the AR-HUD system 18, the TWD system 20, and / or the HMI 22), the occupant 40 is informed of the status of the automated driving system 24. In the manual and partially automated operating modes of the automated driving system 24, the occupant 40 is informed when manual intervention or control of the vehicle 12 is required, thereby improving occupant perception. In the fully automated operating mode, receiving real-time visual information about the status of the automated driving system 24 can improve occupant comfort.
[0079] The description of the present invention is merely exemplary in nature, and changes that do not depart from the gist of the present invention are included within the scope of the present invention. These changes should not be regarded as departing from the spirit and scope of the present invention.
Claims
1. A system for providing information to an occupant of a vehicle, the system comprising: Autonomous driving systems; a display configured to provide information to the occupant; as well as a controller in electrical communication with the autopilot system and the display, wherein the controller is programmed to: determining, using the automated driving system, an automation confidence level of a target location, wherein the target location is a location in an environment surrounding the vehicle; and The occupant is notified using the display based at least in part on the automation confidence level.
2. The system of claim 1 further comprising a plurality of vehicle sensors in electrical communication with the controller, wherein: To determine the automated confidence level of the target position, the controller is also programmed to: utilizing the plurality of vehicle sensors to perform a plurality of measurements of an environment surrounding the vehicle; and An automation confidence level of the target position is determined using the autonomous driving system based at least in part on the plurality of measurements.
3. The system according to claim 2, wherein: To determine an automation confidence level of the target position using the autopilot system, the controller is further programmed to: An automation confidence level for the target location is determined based at least in part on at least one of environmental conditions and road conditions.
4. The system according to claim 1, wherein: In order to inform the occupant using the display, the controller is also programmed to: The display is used to display a notification graphic superimposed on the target location.
5. The system according to claim 4, wherein: In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to: comparing the automation confidence level of the target location to a first automation confidence level threshold; generating a first graph in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold; generating a second graph in response to determining that the automation confidence level for the target location is less than the first automation confidence level threshold; as well as At least one of the first graphic and the second graphic is displayed using the display.
6. The system according to claim 4, wherein: In order to display the notification graphic superimposed on the target location using the display, the controller is further programmed to: comparing the automation confidence level of the target location to a first automation confidence level threshold; comparing the automation confidence level of the target location to a second automation confidence level threshold; generating a first graph in response to determining that the automation confidence level of the target location is greater than or equal to the first automation confidence level threshold; generating a second graph in response to determining that the automation confidence level for the target location is less than the second automation confidence level threshold; generating a third graph in response to determining that the automation confidence level of the target location is less than the first automation confidence level threshold and greater than or equal to the second automation confidence level threshold; as well as At least one of the first graphic, the second graphic, and the third graphic is displayed using the display.
7. The system according to claim 6, wherein: The first graphic includes a first distinguishing feature, the second graphic includes a second distinguishing feature, and the third graphic includes a third distinguishing feature, wherein the first, second and third distinguishing features include at least one of graphic color, graphic size, graphic shape and graphic texture, and wherein the first, second and third distinguishing features are different from each other.
8. The system according to claim 6, wherein: The display is an augmented reality head-up display (AR-HUD) system in electrical communication with the controller, wherein the AR-HUD system includes an occupant position tracking device and an AR-HUD projector, and wherein to display at least one of the first graphic, the second graphic, and the third graphic, the controller is further programmed to: Determining the position of the vehicle occupant using the occupant position tracking device; calculating a size, shape, and position of at least one of the first graphic, the second graphic, and the third graphic based on the position of the occupant and the target position; and Based on the size, shape, and position, at least one of the first graphic, the second graphic, and the third graphic is displayed on a windshield of the vehicle using the augmented reality head-up display system.
9. The system according to claim 8, wherein: The display further comprises a transparent windshield display (TWD) system in electrical communication with the controller, wherein the transparent windshield display system comprises a transparent phosphor and a TWD projector embedded in the vehicle windshield, and wherein to display at least one of the first graphic, the second graphic, and the third graphic, the controller is further programmed to: calculating a size, a shape, and a position of at least one of the first graphic, the second graphic, and the third graphic based on the target position; and Based on the size, shape, and position, at least one of the first graphic, the second graphic, and the third graphic is displayed on a windshield of the vehicle using the TWD system.
10. The system according to claim 1, wherein: The automated driving system is configured to be used in at least one of a manual operating mode, a partially automated operating mode, and a fully automated operating mode.
