Identifying relevant persons through augmented reality head-up display

Abstract

A system for identifying a relevant person for an occupant within a vehicle using a head-up system includes a controller in communication with an external scene camera adapted to: capture an image of an external environment in the vicinity of the vehicle; identifying at least one relevant person located in the external environment in the vicinity of the vehicle; describing at least one related person; determining a suitable augmented graph related to the identified at least one related person to be displayed to the occupant; and displaying, with the head-up display system, the enhanced graphic onto the interior surface of the windshield of the vehicle.

Description

Technical Field

[0001] The present disclosure relates to a system and method for displaying appropriate enhanced graphics in a vehicle to alert vehicle occupants to the presence of persons of interest near the vehicle. Background Art

[0002] In order to improve the cognitive ability and convenience of the occupants, the vehicle may be equipped with a display system configured to provide the occupants with information about the vehicle's surroundings. The display system may use a human-machine interface device (e.g., a touch screen) to provide information to the occupants. The display system may also use a head-up display (HUD) to provide information to the occupants. In order to control and / or illuminate the HUD, a graphic projector may be used to project graphics onto the surface of the vehicle (e.g., a windshield). However, current graphic projectors may utilize high-intensity, concentrated light sources, such as lasers. High-intensity, concentrated light sources may cause discomfort to the occupants when reflected by the surface of the vehicle. In addition, high-intensity, concentrated light sources may increase the weight and resource usage of the vehicle.

[0003] Therefore, while current graphic projectors achieve their intended purpose, a need exists for a new and improved system for graphic projection in a vehicle that is adapted to display appropriate enhanced graphics to alert occupants to the presence and nature of persons of interest in the vicinity of the vehicle. Summary of the invention

[0004] According to several aspects of the present disclosure, a method for identifying relevant persons for occupants in a vehicle using a head-up display system includes: using a controller that communicates with an external scene camera that is suitable for capturing images of the external environment near the vehicle, identifying at least one relevant person located in the external environment near the vehicle; using the controller, describing at least one relevant person; using the controller, determining appropriate enhanced graphics related to the at least one identified relevant person to be displayed to the occupants; and using a head-up display system, displaying the enhanced graphics on the inner surface of the windshield of the vehicle.

[0005] According to another aspect, the method further comprises sending information related to the description of at least one relevant person to a vehicle control module of the vehicle via a communication network within the vehicle; and initiating a vehicle control action based on the description of at least one relevant person identified using the vehicle control module.

[0006] According to another aspect, identifying at least one relevant person located in the external environment near the vehicle using a controller that communicates with an external scene camera suitable for capturing images of the external environment near the vehicle also includes: scanning the external environment near the vehicle using the external scene camera; and identifying people located in the external environment near the vehicle through a computer vision algorithm within the controller.

[0007] According to another aspect, identifying persons in an external environment near a vehicle using a computer vision algorithm within a controller also includes: collecting data related to the identified persons in the external environment near the vehicle from a remote source that communicates with the controller; and identifying, using the controller, at least one related person among the identified persons based on the data collected from the remote source.

[0008] According to another aspect, collecting data related to identified persons in the external environment near the vehicle from a remote source in communication with the controller also includes: collecting data related to identified persons in the external environment near the vehicle from at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records.

[0009] According to another aspect, using a controller, based on data collected from a remote source, identifying at least one related person among the identified persons also includes identifying at least one related person among the identified persons as at least one of the following: a follower of the occupant on social media, a person followed by the occupant on social media, a friend of the occupant, a family member of the occupant, a celebrity, a colleague of the occupant, a person with similar interests to the occupant, and a peer of the occupant.

[0010] According to another aspect, using the controller, describing at least one relevant person further includes: using a cloud computing algorithm in communication with the controller, describing the at least one relevant person as friendly or unfriendly based on data related to the identified person collected from a remote source.

[0011] According to another aspect, displaying appropriate enhanced graphics on the inner surface of a windshield of a vehicle using a head-up display system further includes: determining the position of an occupant's head and eyes using an occupant monitoring system; calculating the position of the appropriate enhanced graphics on the inner surface of the windshield based on the position of the occupant's head and eyes, wherein the enhanced graphics appear near at least one identified person of interest when the occupant views the enhanced graphics; and displaying the appropriate enhanced graphics on the inner surface of the windshield by: projecting excitation light using a light source of a digital light projector; focusing the excitation light from the light source using a condenser lens; separating the focused excitation light into red light, green light, and blue light using a color filter; focusing the excitation light transmitted through the color filter using a shaping lens; and displaying the appropriate enhanced graphics on the inner surface of the windshield by: projecting excitation light using a light source of a digital light projector; focusing the excitation light from the light source using a condenser lens; separating the focused excitation light into red light, green light, and blue light using a color filter; focusing the excitation light transmitted through the color filter using a shaping lens; and device (DMD); receiving the excitation light from the digital micromirror device using a projection lens; projecting the excitation light onto a location on an inner surface of a windshield using the projection lens based on the position of the occupant's head and eyes, wherein luminescent particles within the windshield emit visible light in response to absorbing the excitation light, and when viewed by the occupant, a suitable enhancement graphic appears in the vicinity of at least one identified person of interest.

[0012] According to another aspect, the method further includes: determining, using a controller, whether the windshield is in a privacy mode, wherein in the privacy mode, the windshield is opaque, the occupants cannot see the external environment through the windshield, and people in the external environment cannot see the occupants; and when the windshield is in the privacy mode, deactivating the privacy mode before displaying appropriate enhanced graphics on the inner surface of the windshield.

[0013] According to several aspects of the present disclosure, a system for identifying persons of interest for occupants in a vehicle utilizing a heads-up display system includes: a controller in communication with an external scene camera, adapted to: capture images of an external environment near the vehicle; identify at least one person of interest located in the external environment near the vehicle; describe the at least one person of interest; determine appropriate enhanced graphics to be displayed to the occupants related to the at least one identified person of interest; and display the enhanced graphics on an inner surface of a windshield of the vehicle utilizing a heads-up display system.

[0014] According to another aspect, the system further comprises an intra-vehicle communication network adapted to transmit information relating to the description of at least one associated person to a vehicle control module of the vehicle, the vehicle control module adapted to initiate a vehicle control action based on the identified description of at least one associated person.

[0015] According to another aspect, when identifying at least one relevant person in the external environment near the vehicle, the controller is also suitable for: scanning the external environment near the vehicle using an external scene camera; and identifying the person in the external environment near the vehicle through a computer vision algorithm within the controller.

[0016] According to another aspect, when identifying at least one relevant person in the external environment near the vehicle, the controller is also suitable for: collecting data related to the identified person in the external environment near the vehicle from a remote source communicating with the controller; and identifying at least one relevant person among the identified persons based on the data collected from the remote source.

[0017] According to another aspect, the controller is adapted to collect data related to identified persons located in the external environment near the vehicle from at least one of social media applications, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records.

[0018] According to another aspect, the controller is adapted to identify at least one relevant person as at least one of the following: a follower of the occupant on social media, a person the occupant follows on social media, a friend of the occupant, a family member of the occupant, a celebrity, a colleague of the occupant, a person with similar interests to the occupant, and a peer of the occupant.

[0019] According to another aspect, the controller is adapted to describe at least one associated person as friendly or unfriendly based on data associated with the identified person collected from a remote source via a cloud computing algorithm in communication with the controller.

[0020] According to another aspect, the digital light projector includes: a light source suitable for projecting excitation light; a focusing lens suitable for focusing the excitation light from the light source; a color filter suitable for separating the focused excitation light into red light, green light and blue light; a shaping lens suitable for focusing the excitation light passing through the color filter; a digital micromirror device (DMD) suitable for redirecting the excitation light; and a projection lens suitable for receiving the excitation light from the digital micromirror device and projecting the excitation light onto the inner surface of the windshield; and, the head-up system includes an occupant monitoring system suitable for receiving information related to the head and eye position of an occupant in the vehicle, wherein the digital light projector is suitable for displaying enhanced graphics at a certain position on the inner surface of the windshield based on the head and eye position of the occupant, so that when the occupant views the enhanced graphics, the enhanced graphics appear in the vicinity of at least one relevant person.

[0021] According to another aspect, the windshield operates in a privacy mode, wherein in the privacy mode, the windshield is opaque, the occupants cannot see the external environment through the windshield, and people in the external environment cannot see the occupants; the controller is suitable for determining whether the windshield is in the privacy mode; and when the windshield is in the privacy mode, the privacy mode is deactivated before displaying appropriate enhanced graphics on the inner surface of the windshield.

[0022] Further areas of applicability will become apparent from the description provided herein.It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way.

[0024] Figure 1 is a schematic diagram of a vehicle having a system according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 yes Figure 1 a schematic side view of the vehicle shown;

[0026] Figure 3 is a schematic diagram of a system according to an exemplary embodiment;

[0027] Figure 4 is a schematic side elevation view of a windshield of a vehicle as viewed from an occupant of the vehicle;

[0028] Figure 5 is a schematic diagram of a substrate for a windshield of a vehicle according to an exemplary embodiment;

[0029] Figure 6 yes Figure 5 The circle marked with "6" Figure 5 The enlarged part of

[0030] Fig. 7A is a schematic cross-sectional view of a windshield having a liquid crystal layer therein, wherein an electric current is passed through the liquid crystal layer and light is allowed to pass therethrough;

[0031] Figure 7B yes Fig. 7A A schematic cross-sectional view of the diagram shown, in which no current is present and light cannot pass; and

[0032] Figure 8 is a flow chart illustrating a method according to an exemplary embodiment of the present disclosure.

[0033] The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized, for example, to show details of a particular component. In some cases, well-known components, systems, materials, or methods are not described in detail to avoid obscuring the present disclosure. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching those skilled in the art to adopt the present disclosure in different ways. DETAILED DESCRIPTION

[0034] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application or use. In addition, it is not intended to be constrained by any express or implied theory presented in the aforementioned technical field, background technology, content of the invention or the following specific embodiments. It should be understood that throughout the drawings, the corresponding reference numerals represent similar or corresponding parts and features. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including but not limited to: application specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or grouped) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the described functions. Although the drawings shown herein depict examples with certain element arrangements, additional intermediate elements, devices, features or components may be present in actual embodiments. It should also be understood that the drawings are merely illustrative and may not be drawn to scale.

[0035] As used herein, the term "vehicle" is not limited to automobiles. Although this article mainly describes the present technology in conjunction with automobiles, the present technology is not limited to automobiles. These concepts can be used in a variety of applications, such as aircraft, ships, other vehicles, and consumer electronic components.

[0036] According to an exemplary embodiment, Figure 1 A vehicle 10 is shown with an associated head-up display system in the vehicle 10 according to various embodiments. Generally speaking, the head-up display system 11 works in conjunction with other systems in the vehicle 10 to display various information and infotainment content to passengers. The vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is arranged on the chassis 12 and generally surrounds the components of the vehicle 10. The body 14 and the chassis 12 can together form a frame. The front wheels 16 and the rear wheels 18 are each rotatably connected to the chassis 12 near the corresponding corners of the body 14.

[0037] In various embodiments, the vehicle 10 is an autonomous vehicle and the system 11 is incorporated into the autonomous vehicle 10. The autonomous vehicle 10 is, for example, a vehicle 10 that is automatically controlled to transport passengers from one location to another. The vehicle 10 is depicted as a passenger car in the illustrated embodiment, but it should be understood that any other vehicle may also be used, including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), etc. In an exemplary embodiment, the vehicle 10 is equipped with a so-called level 4 or level 5 automation system. The level 4 system represents "high automation" and refers to the driving mode-specific performance of the autonomous driving system in various aspects of dynamic driving tasks even if the human driver does not respond appropriately to the intervention request. The level 5 system represents "full automation" and refers to the full-time performance of the autonomous driving system in various aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage.

[0038] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, a vehicle controller 34, and a communication system 36. In an embodiment where the vehicle 10 is an electric vehicle, there may be no transmission system 22. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor, and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the front wheels 16 and rear wheels 18 of the vehicle according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission, or other suitable transmissions. The braking system 26 is configured to provide braking torque to the front wheels 16 and rear wheels 18 of the vehicle. In various embodiments, the braking system 26 may include a friction brake, a wire control brake, a regenerative braking system (e.g., an electric motor), and / or other suitable braking systems. The steering system 24 affects the position of the front wheels 16 and the rear wheels 18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present disclosure, the steering system 24 may not include a steering wheel.

[0039] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or internal environment of the autonomous vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The camera may include two or more digital cameras spaced at a selected distance from each other, wherein the two or more digital cameras are used to obtain stereo images of the surrounding environment in order to obtain a three-dimensional image. Multiple sensing devices 40a-40n are used to determine information about the environment around the vehicle 10. In an exemplary embodiment, the multiple sensing devices 40a-40n include a motor speed sensor, a motor torque sensor, an electric drive motor voltage and / or current sensor, an accelerator pedal position sensor, a coolant temperature sensor, a cooling fan speed sensor, and a transmission oil temperature sensor. In another exemplary embodiment, the multiple sensing devices 40a-40n also include sensors for determining information about the environment around the vehicle 10, such as an ambient air temperature sensor, an atmospheric pressure sensor, and / or a camera and / or a video camera, which is positioned to observe the environment in front of the vehicle 10. In another exemplary embodiment, at least one of the plurality of sensing devices 40a-40n is capable of measuring distances in the environment surrounding the vehicle 10. In a non-limiting example where the plurality of sensing devices 40a-40n include cameras, the plurality of sensing devices 40a-40n use an image processing algorithm configured to process images from the cameras and determine distances between objects to measure distances. In another non-limiting example, the plurality of vehicle sensors 40a-40n include a stereo camera with distance measurement capabilities.

[0040] In one example, at least one of the plurality of sensing devices 40a-40n is secured to the interior of the vehicle 10, e.g., secured in the roof lining of the vehicle 10, with a view through the windshield 56 of the vehicle 12. In another example, at least one of the plurality of sensing devices 40a-40n is secured to the exterior of the vehicle 10, e.g., secured to the roof of the vehicle 10, with a view of the environment surrounding the vehicle 10. It should be understood that various additional types of sensing devices, such as lidar sensors, ultrasonic ranging sensors, radar sensors, and / or time of flight sensors are within the scope of the present disclosure. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more features of the vehicle 10, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26.

[0041] The vehicle controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The at least one data processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary processor of multiple processors associated with the vehicle controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or any device generally used to execute instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage devices such as read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a persistent or non-volatile memory that can be used to store various operating variables when the at least one data processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROM (programmable read-only memory), EPROM (electrically programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data used by the controller 34 in controlling the vehicle 10, some of which represents executable instructions.

[0042] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by at least one processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 Only one controller 34 is shown, and embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication media and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control the functions of the autonomous vehicle 10.

[0043] In various embodiments, one or more instructions of the vehicle controller 34 are embodied in a trajectory planning system and, when executed by at least one data processor 44, generate trajectory outputs that address kinematic and dynamic constraints of the environment. For example, the instructions receive process sensor and map data as input. The instructions implement a graph-based approach and customized cost functions to handle different road scenarios of cities and highways.

[0044] The communication system 36 is configured to wirelessly communicate information with other remote entities 48, such as, but not limited to, other vehicles ("V2V" communication), infrastructure ("V2I" communication), remote systems, remote servers, cloud computers, and / or personal devices. In an exemplary embodiment, the communication system 36 is a wireless communication system configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. DSRC channels refer to unidirectional or bidirectional short-range to medium-range wireless communication channels designed specifically for automobiles, as well as a set of corresponding protocols and standards.

[0045] The vehicle controller 34 is a non-universal electronic control device having a pre-programmed digital computer or processor, a memory or non-transitory computer-readable medium for storing data such as control logic, software applications, instructions, computer code, data, look-up tables, etc., and a transceiver (or input / output port). Computer-readable media include any type of media that can be accessed by a computer, such as a read-only memory (ROM), a random access memory (RAM), a hard drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. "Non-transitory" computer-readable media do not include wired, wireless, optical or other communication links that transmit temporary electrical or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store and subsequently rewrite data, such as rewritable optical discs or erasable storage devices. Computer code includes any type of program code, including source code, object code, and executable code.

[0046] refer to Figure 2 , the controller 34 of the head-up system 11 communicates with the external scene camera 52. The various sensing devices 40a-40n collect information and provide the information to the controller 34. The sensing devices 40a-40n include an external scene camera 52 or an image capture device that takes periodic or sequential images representing a view from the vehicle 10. The controller 34 is adapted to capture images of the external environment 54 near the vehicle 10 using the external scene camera 52. The controller 34 is also adapted to identify at least one person of interest 56 located within the external environment 54 near the vehicle 10. When at least one person of interest 56 located within the external environment 54 near the vehicle 10 is identified, the controller 34 is further adapted to scan the external environment 54 near the vehicle 10 using the external scene camera 52, and identify the person located within the external environment 54 near the vehicle 10 using a computer vision algorithm 58 within the controller 34.

[0047] refer to Figure 3, the external scene camera 52 captures an image of a group of persons 60 standing near the vehicle 10. The controller 34 identifies the persons constituting the group 60 using the computer vision algorithm 58, and further collects data related to the identified persons within the external environment 54 located near the vehicle 10 from a remote source 48 in communication with the controller 34. In an exemplary embodiment, the controller 34 accesses the remote source 48 (including at least one of a social media application, criminal records, business networks, motor vehicle records, real estate records, voting records, and school records) to identify the individuals constituting the group of persons 60. Such remote sources 48 may include any publicly accessible database.

[0048] The controller 34 is further adapted to identify at least one related person 56 from the group of identified persons 60 based on the data collected from the remote source 48. By analyzing the data collected from the remote source 48, the controller 34 identifies individuals from the group of identified persons 60, i.e., related persons 56, for the occupant 80 in the vehicle 10. The related person 56 may be any one or more of: a person who follows the occupant 80 on social media, a person that the occupant 80 follows on social media, a friend of the occupant 80, a family member of the occupant 80, a celebrity, a co-worker of the occupant 80, a person who has similar interests to the occupant 80, and a travel companion of the occupant 80. Anyone who has some connection with the occupant 80, as determined by the controller 34 analyzing the data from the remote source 48, may be identified as a related person 56 for the occupant 80.

[0049] The controller 34 is also adapted to describe at least one person of interest 56. In an exemplary embodiment, the controller 34 is adapted to describe at least one person of interest 56 as friendly or unfriendly based on data collected from a remote source related to the set of identified persons 60 using a cloud computing algorithm 92 in communication with the controller 34. The controller 34 analyzes the data collected from the remote source 48 to determine whether each of the identified persons of interest 56 has positive or negative feelings toward the occupant 80. The controller 34 may identify the persons of interest 56 as unfriendly or even hostile based on communications or comments made by the persons of interest 56 and / or the occupant 80 on social media or other remote sources 48.

[0050] In an exemplary embodiment, after describing any relevant person 56, the controller 34 is adapted to send information related to the description of at least one relevant person 56 to the control module 104 of the vehicle 10 via the communication network 106 within the vehicle 10. The vehicle control module 104 is adapted to initiate a vehicle control action based on the description of the at least one relevant person 56 identified. For example, if the identified relevant person 56 is friendly, the vehicle control module 104 can stop the vehicle 10 to give the occupant 80 an opportunity to interact with the identified relevant person 56. In addition, the controller 34 and the vehicle control module 104 can prompt the vehicle infotainment system to provide an audible prompt to alert the occupant 80 of the presence of the friendly relevant person 56. In addition, the controller 34 can activate the internal and external speakers and microphones to allow the occupant to communicate with the identified relevant person 56. Alternatively, if the identified relevant person 56 is unfriendly, the controller 34 and the vehicle control module 104 can cause the vehicle 10 to continue to move past the unfriendly relevant person 56, or change the route of the vehicle 10 to avoid the unfriendly relevant person 56. Here, the controller 34 and the vehicle control module 104 may also prompt the vehicle infotainment system to provide an audible prompt to alert the occupant 80 of the presence of the unfriendly associate 56 .

[0051] The controller 34 is further adapted to determine appropriate augmented graphics 94 to be displayed to the occupant 80 in relation to the identified at least one or more persons of interest 56, and to display the appropriate augmented graphics 94 on the interior surface 96 of the windshield 98 of the vehicle 10 using the head-up display system 11. The appropriate augmented graphics 94 are adapted to inform the occupant 80 of the identity and category of the identified persons of interest 56. Figure 4 , showing the windshield 98 as viewed by the occupant 80. The occupant 80 can see the group of persons 60 standing near the vehicle 10. The system 11 has captured an image of the group of persons 60, accessed the remote source 48 to identify the individuals, and identified two persons of interest 56 for the occupant 80. The system 11 then displays an appropriate augmented graphic 94 for each of the two persons of interest 56.

[0052] For the first related person 56A, the individual identified by the system 11 is Sally 56A, who is a friend of the occupant 80 as indicated by the tags and conversations from the social media website. The appropriate enhanced graphic 94 for Sally 56A includes an outline 100A to distinguish Sally 56A as a person 56 related to the occupant 80. In addition, the appropriate enhanced graphic 94 for Sally 56A includes a text box 102A attached to the outline 100A that includes descriptive information about Sally 56A. Figure 4As shown, the text box 102A includes Sally's name "Sally" and the word "Friend", indicating that Sally 56A is a friend of the occupant 80. The appropriate augmented graphic 94 may be further customized based on the classification of Sally 56A. If the system 11 classifies Sally 56A as "friendly", the outline 100A may be displayed in a color indicative of the outline 100A, for example, the outline 100A may be green.

[0053] For the second person of interest 56B, the system 11 has identified the individual as Lolita 56B, who is a co-worker of occupant 80, as indicated by tags and conversations from social media sites (and possibly their employer's website). The appropriate enhanced graphic 94 for Lolita 56B includes an outline 100B to distinguish Lolita 56B as a person of interest 56 for occupant 80. In addition, the appropriate enhanced graphic 94 for Lolita 56B includes a text box 102B attached to the outline 100B that includes descriptive information about Lolita 56B. Figure 4 As shown, the text box 102B includes the name of the lolita "Lolita" and the word "colleague", indicating that the lolita 56B is a colleague of the occupant 80. The appropriate enhanced graphics 94 can be further customized based on the classification of the lolita 56B. If the system 11 classifies the lolita 56B as "unfriendly", the outline 100B can be displayed in a color indicating the outline 100B, for example, the outline 100B can be red.

[0054] It should be understood that the appropriate augmented graphics 94 may include any information that is considered relevant to the occupant 80. In an exemplary embodiment, the amount of information provided in the text boxes 102A, 102B associated with the identified person of interest 56 may include more or less detail, depending on the preferences of the occupant 80 and the nature of the information. If the system 11 identifies that the person of interest 56 is particularly hostile to the occupant 80, detailed information may be provided regardless of the preferences selected by the occupant 80. For example, if the occupant 80 is a celebrity or politician, and the system 11 identifies that the person of interest 56 in the vicinity of the vehicle 10 has made a threat to the occupant 80, the appropriate augmented graphics 94 may be adjusted to ensure that the occupant 80 is informed of such information. In this case, the system 11 may provide an appropriate augmented graphic 94 that includes an outline having a color (i.e., red) or alternating colors and / or includes dynamic features, such as pulsing or increased brightness / intensity, etc.

[0055] refer to Figure 3, the head-up display system 11 includes a digital light projector (DLP) 50 that is adapted to project suitable augmented graphics 94 onto the interior surface 96 of a windshield 98 of the vehicle 10. The DLP 50 projects the suitable augmented graphics 94 that contain various information, as described above. The DLP 50 is capable of projecting two-dimensional and three-dimensional images depicting objects such as pedestrians, buildings, and other vehicles in the external environment 54 surrounding the vehicle 10, which may be included in the suitable augmented graphics 94. In an exemplary embodiment, the DLP 50 includes: a light source 62, suitable for projecting excitation light 64; a focusing lens 66, suitable for focusing the excitation light 64 from the light source 62; a color filter 68 (color wheel), suitable for separating the focused excitation light 64 into red, green, and blue light; a shaping lens 70, suitable for focusing the excitation light 64 passing through the color filter 68; a digital micromirror device (DMD) 72, suitable for redirecting the excitation light 64; and a projection lens 74, suitable for receiving the excitation light 64 from the digital micromirror device DMD 72 and projecting the excitation light 64 onto the inner surface 96 of the windshield 98.

[0056] Reference again Figure 2 , the vehicle controller 34 includes a DLP engine 76. The DLP engine 76 includes a programmable processor that includes programming for monitoring various inputs and determining what information should be included in the appropriate enhanced graphics 94. The DLP engine 76 can communicate directly with the various systems and components, or the DLP engine 76 can alternatively or additionally communicate through a LAN / CAN system. The DLP engine 76 utilizes information about the operating environment of the vehicle 10 derived from a plurality of inputs.

[0057] The windshield 98 is equipped with features that can display suitable enhanced graphics 94 projected thereon while remaining transparent or substantially transparent, so that the occupants 80 of the vehicle 10 can clearly observe the exterior of the vehicle 10 through the windshield 98. It should be understood that, as shown, the windshield 98 is located at the front of the vehicle 10, and other surfaces within the vehicle 10 can also be used for projection, including the side windows and rear windows. In addition, the view on the front windshield 98 can be continued as a continuous image to the front vehicle "A-pillar" and on the side windows.

[0058] The DLP engine 76 includes display software or programming to translate requests to describe information from the DLP engine 76 in a graphical representation. The DLP engine 76 includes programming to compensate for the curved and tilted surfaces of the windshield 98 and any other surface upon which the appropriate enhancement graphics 94 are to be projected. The DLP engine 76 controls the light source 62, which includes a laser or projector device that produces the excitation light 64 to project the appropriate enhancement graphics 94.

[0059] The occupant monitoring system 78 includes sensors known in the art to generally assess the position of the occupant's 80 head and the orientation or gaze position of the occupant's 80 eyes. Based on the output of the occupant monitoring system 78 and the input data tracking position information about the vehicle 10 surrounding environment, the DLP engine 76 can accurately position the appropriate augmented graphics 94 so that the occupant 80 sees the appropriate augmented graphics 94 overlaid with the visual image (the identified person of interest 56) through the windshield 98. Figure 3 , the outline 100A and text box 102A of the first identified person of interest 56A and the outline 100B and text box 102B of the second identified person of interest 56B are displayed on the inner surface 96 of the windshield 98, at a position between the eyes of the occupant 80 and the first and second identified persons of interest 56A, 56B, with reference to Figure 4 , the outline 100A and text box 102A of the first related person 56A appear at a position that places the occupant 80 in proper relationship with the first related person 56A, and the outline 100B and text box 102B of the second related person 56B appear at a position that places the occupant 80 in proper relationship with the second related person 56B.

[0060] The system 11 includes eye sensing and head sensing devices that allow for estimation of eye position and for registration of an image on the windshield 98 so that the image corresponds to the view of the occupant 80. However, it should be understood that estimation of head and eye position can be achieved by a variety of methods. For example, in a process similar to adjusting a rearview mirror, an operator can use a calibration routine upon entering the vehicle to align the graphics with detected objects. In another embodiment, the longitudinal seat position in the vehicle 10 can be used to estimate the position of the head of the occupant 80. In another embodiment, manual adjustment of one or more rearview mirrors can be used to estimate the position of the eyes of the occupant 80. It should be understood that a combination of methods such as seat position and mirror adjustment angles can be utilized to estimate the head position of the occupant 80 with improved accuracy. Many methods for achieving accurate registration of the graphics on the windshield 98 are contemplated, and the present disclosure is not intended to be limited to the specific embodiments described herein.

[0061] The windshield 98 serves as a medium through which the relevant features can be viewed, and also as a display device upon which suitable enhancement graphics 94 can be displayed. The windshield 98 is transparent and is capable of displaying the image projected by the excitation light 64. Figure 5 and Figure 6, the occupant 80 is able to see any object (e.g., cube 82) through the substrate 84 located on the windshield 98. The substrate 84 can be transparent or substantially transparent. When the occupant 80 sees the arbitrary object 82 through the substrate 84, the occupant 80 can also see the image created at the substrate 84 (e.g., circle 86 and triangle 88). The substrate 84 can be a portion of the windshield 98, a glass substrate, a plastic substrate, a polymer substrate, or other transparent (or substantially transparent) medium that will be understood by those of ordinary skill in the art. Other substrates can supplement the substrate 84 to provide coloring, substrate protection, light filtering (e.g., filtering external ultraviolet light), and other functions.

[0062] Figure 5 The illumination of the transparent display is depicted by illumination with excitation light 64 (e.g., ultraviolet light or infrared light) from light source 62. Substrate 84 receives excitation light 64 from light source 62. The received excitation light 64 can be absorbed by luminescent material 90 of substrate 84. When the luminescent material receives the excitation light 64, the luminescent material 90 emits visible light. Thus, an image (e.g., circle 86 and triangle 88) can be generated on substrate 84 by selectively illuminating substrate 84 with excitation light 64.

[0063] In an exemplary embodiment, the luminescent material 90 includes a transparent phosphor embedded in the substrate 84. The transparent phosphor is a luminescent particle that emits fluorescence in response to excitation by the excitation light 64. In an exemplary embodiment, the transparent phosphor is a red, green and blue (RGB) phosphor, thereby allowing full color. The use of monochromatic and / or dual-color phosphors is also within the scope of the present disclosure. When the excitation light 64 is absorbed by the transparent phosphor, the transparent phosphor emits visible light. The excitation light 64 is provided by the light source 62. The scheme of displaying graphics using the substrate 84 and the luminescent material 90 is discussed in U.S. patent application No. 17 / 749,464, entitled “HYBRID AUGMENTED REALITY HEAD-UP DISPLAY FOR CREATING AN EDGE-TO-EDGE AUGMENTED REALITY VIEW” filed on May 20, 2022, the entire contents of which are incorporated herein by reference.

[0064] According to an embodiment of the present disclosure, the excitation light 64 may be ultraviolet light. If the excitation light 64 is ultraviolet light, when the luminescent material 90 emits visible light in response to the ultraviolet light, a down-conversion physical phenomenon occurs. Specifically, ultraviolet light has a shorter wavelength and higher energy than visible light. Accordingly, when the luminescent material 90 absorbs ultraviolet light and emits lower energy visible light, the ultraviolet light is down-converted into visible light because its energy level decreases when the ultraviolet light is converted into visible light.

[0065] It should be understood that other technologies may also be used, such as holographic grating (Bragg grating) based technologies using visible light projectors, where RGB LEDs are used to make vector graphics projectors and RGB-DLP projectors.

[0066] like Figure 2 and Figure 3 As shown, the excitation light 64 is output by the light source 62 of the DLP 50. In an exemplary embodiment, the light source 62 is a micro-mirror array (MMA) projector (e.g., a digital light processing (DLP) projector 50). In the DLP 50, a suitable enhancement pattern 94 is created by microscopically small mirrors that are arranged in a matrix on a semiconductor chip within the DMD 72. An MMA projector that outputs ultraviolet light can be similar to an MMA projector that outputs visible light, except that the color filter 68 has a filter customized for the ultraviolet spectrum. In other embodiments, the DLP 50 is a liquid crystal display (LCD) projector. In an embodiment, the DLP 50 can be a liquid crystal on silicon (LCOS) projector. In an embodiment, the DLP 50 can be an analog projector (e.g., a slide projector or a movie film projector). Those of ordinary skill in the art will understand other types of projectors that can be used to project ultraviolet light onto the substrate 84.

[0067] In an exemplary embodiment, the windshield 98 is operable in a privacy mode, wherein the windshield 98 is opaque, the exterior environment 54 is not visible to the occupant through the windshield 98, and the occupant 80 is not visible to persons within the exterior environment 54. The controller 34 is adapted to determine whether the windshield 98 is in the privacy mode and, when the windshield 98 is in the privacy mode, deactivate the privacy mode before displaying the appropriate enhanced graphics 94 on the interior surface 96 of the windshield 98.

[0068] In addition, after describing any identified persons of interest 56, the controller may activate or deactivate the privacy mode of the windshield 98 based on the description of the identified one or more persons of interest 56. For example, if the system 11 identifies and describes the person of interest 56 as friendly, and the windshield 98 is in the privacy mode, the controller will deactivate the privacy mode of the windshield 98, thereby allowing the occupant 80 to see the friendly persons of interest 56 and allowing the friendly persons of interest 56 to see the occupant 80. Alternatively, if the system 11 identifies and describes the person of interest 56 as unfriendly, and the windshield 98 is in the privacy mode, the controller will keep the windshield in the privacy mode. If the windshield 98 is not in the privacy mode, the controller will activate the privacy mode of the windshield 98, thereby preventing the occupant 80 from seeing the unfriendly persons of interest 56 and preventing the unfriendly persons of interest 56 from seeing the occupant 80.

[0069] In an exemplary embodiment, to implement the privacy mode, the liquid crystal layer 104 is located between two layers of glass 106 of the windshield 98. Fig. 7A , the liquid crystal 104 responds to the charge provided by the conductive rod 108 by aligning in parallel and allowing light to pass through, as shown by arrow 110. Figure 7B , when no charge is present, the liquid crystals 104 are randomly oriented, thereby blocking most light from passing through the windshield 98, as indicated by arrows 112, and rendering the windshield 98 opaque or translucent. Because of the liquid crystals 104, the glass is transparent or translucent, depending on the presence or absence of an electric current, and there is no intermediate setting.

[0070] refer to Figure 8 , a method 200 is shown for identifying a person 56 associated with an occupant 80 in a vehicle 10 using a heads-up system 11. Starting at box 202, the occupant 80 enters the vehicle 10. Moving to box 204, if the occupant 80 does not activate the system 11, then moving to box 206, no action is taken, and the method 200 returns to box 202.

[0071] If at box 204 , the occupant 80 does activate the system 11 , then moving to box 208 , the method 200 includes utilizing a controller 34 in communication with an external scene camera 52 adapted to capture images of the external environment 54 proximate the vehicle 10 , to identify at least one person of interest 56 located within the external environment 54 proximate the vehicle 10 .

[0072] Moving to box 210, if the system 11 does not identify any persons of interest 56, no action is taken and the system 11 returns to box 208. If at box 210, the system 11 does detect at least one person of interest 56, then moving to box 212, the method 200 includes describing the at least one person of interest 56 using the controller 34. Moving to box 214, the method 200 also includes determining, using the controller 34, suitable augmented graphics 94 to be displayed to the occupant 80 related to the identified at least one person of interest 56, and moving to box 216, displaying the suitable augmented graphics 94 on the inner surface 96 of the windshield 98 of the vehicle 10 using the heads-up display system 11. Moving to box 218, the system 11 checks whether there are still any identified persons of interest 56. If there are still no persons of interest 56, the method 200 returns to box 208 to identify any persons of interest 56 using the controller 34. If at least one associated person is still present, the method 200 returns to block 216 and continues to display the appropriate augmented graphics 94 onto the interior surface 96 of the windshield 98 of the vehicle 10 using the heads-up display system 11 .

[0073] In an exemplary embodiment, after describing at least one relevant person 56 using the controller 34 at box 212, the method also includes: moving to box 220, sending information related to the description of at least one relevant person 56 to the vehicle control module 104 of the vehicle 10 via the communication network 106 within the vehicle 10; and moving to box 222, using the vehicle control module 104, initiating a vehicle control action based on the description of at least one identified relevant person 56.

[0074] In another exemplary embodiment, at box 208, using a controller that communicates with an external scene camera suitable for capturing images of the external environment near the vehicle, identifying at least one relevant person located in the external environment near the vehicle further includes: moving to box 224, scanning the external environment 54 near the vehicle 10 using the external scene camera 52; and moving to box 226, identifying a person in the external environment 54 near the vehicle 10 through a computer vision algorithm 58 within the controller 34.

[0075] In another exemplary embodiment, identifying persons in the external environment 54 near the vehicle 10 using the computer vision algorithm 58 within the controller 34 at box 226 further includes: moving to box 228, collecting data related to the identified persons in the external environment 54 near the vehicle 10 from a remote source 48 in communication with the controller 34; and moving to box 230, identifying at least one related person 56 among the identified persons using the controller 34 based on the data collected from the remote source 48. At box 228, collecting data related to the identified persons in the external environment 54 near the vehicle 10 from the remote source 48 in communication with the controller 34 further includes: collecting data related to the identified persons in the external environment 54 near the vehicle 10 including at least one of a social media application, a criminal record, a business network, a motor vehicle record, a real estate record, a voting record, and a school record. In addition, at box 230, using the controller 34 based on the data collected from the remote source 48, identifying at least one related person 56 among the identified persons also includes identifying that the related person 56 among the identified persons is at least one of the following: followers of the occupant 80 on social media, people followed by the occupant 80 on social media, friends of the occupant 80, family members of the occupant 80, celebrities, colleagues of the occupant 80, people with similar interests as the occupant 80, and peers of the occupant 80.

[0076] In an exemplary embodiment, describing at least one relevant person 56 using the controller 34 at box 212 also includes: using a cloud computing algorithm 92 in communication with the controller 34 to describe at least one relevant person 56 as friendly or unfriendly based on data related to the identified person collected from the remote source 48.

[0077] In another exemplary embodiment, displaying the appropriate augmented graphics 94 on the inner surface 96 of the windshield 98 of the vehicle 10 using the head-up display system 11 at box 216 also includes: moving to box 232, determining the position of the head and eyes of the occupant 80 using the occupant monitoring system 78; moving to box 234, based on the position of the head and eyes of the occupant 80, calculating the position of the appropriate augmented graphics 94 on the inner surface 96 of the windshield 98, wherein when viewed by the occupant 80, the appropriate augmented graphics 94 appear near the at least one associated person 56 identified; and moving to box 236, projecting the excitation light 64 through the light source 62 of the digital light projector 50, focusing the excitation light 64 from the light source 62 using the focusing lens 66 , using a color filter 68 to separate the focused excitation light 64 into red light, green light and blue light, using a shaping lens to focus the excitation light 64 passing through the color filter 68, using a digital micromirror device (DMD) 72 to redirect the excitation light 64, using a projection lens 74 to receive the excitation light 64 from the digital micromirror device 72, and using the projection lens 74 to project the excitation light 64 onto the inner surface of the windshield based on the head and eye position of the occupant 80 to display a suitable enhanced graphic 94 on the inner surface 96 of the windshield 98, wherein the luminescent particles 90 in the windshield 98 emit visible light in response to absorbing the excitation light 64, and the suitable enhanced graphic 94 appears near at least one relevant person 56 being identified when viewed by the occupant 80.

[0078] In another exemplary embodiment, the method 200 includes moving to box 238 and determining, using the controller 34, whether the windshield 98 is in a privacy mode, wherein in the privacy mode, the windshield 98 is opaque, the occupant 80 cannot see the external environment 54 through the windshield 98, and the occupant 80 cannot be seen by people in the external environment 54. If at box 238, the windshield is not in the privacy mode, the method 200 moves directly to box 236, where the appropriate enhanced graphics 94 are displayed. If at box 238, the windshield is in the privacy mode, then moving to box 240, the method 200 includes deactivating the privacy mode before displaying the appropriate enhanced graphics 94 on the inner surface 96 of the windshield 98.

[0079] The systems and methods of the present disclosure provide the advantage of providing occupants within a vehicle with information regarding the presence and nature of persons of interest in the vicinity of the vehicle.

[0080] The description of the present disclosure is merely exemplary in nature, and variations that do not depart from the gist of the present disclosure are intended to fall within the scope of the present disclosure. These variations should not be regarded as departing from the spirit and scope of the present disclosure.

Claims

1. A method for identifying a person related to a passenger in a vehicle using a head-up system, comprising: identifying, using a controller in communication with an external scene camera adapted to capture images of the external environment in the vicinity of the vehicle, at least one person of interest located in the external environment in the vicinity of the vehicle; utilizing the controller, describing the at least one associated person; determining, with the controller, an appropriate augmented graphic to be displayed to the occupant associated with the identified at least one person of interest; as well as The enhanced graphics are displayed on an interior surface of a windshield of the vehicle using the heads-up display system.

2. The method according to claim 1, further comprising: sending information related to the description of the at least one associated person to a vehicle control module of the vehicle via a communication network within the vehicle; as well as A vehicular maneuver is initiated, using the vehicle control module, based on the identified description of the at least one associated person.

3. The method according to claim 1, wherein: Identifying, with a controller in communication with an external scene camera adapted to capture images of the external environment in the vicinity of the vehicle, at least one person of interest located in the external environment in the vicinity of the vehicle further comprises: Scanning the external environment near the vehicle using the external scene camera; and A computer vision algorithm within the controller is used to identify persons in the external environment near the vehicle.

4. The method according to claim 3, wherein: Using a computer vision algorithm within the controller, identifying a person in an external environment near the vehicle further comprises: collecting data related to identified persons located in an external environment near the vehicle from a remote source in communication with the controller; and At least one related person among the identified persons is identified, using the controller, based on the data collected from the remote source.

5. The method according to claim 4, wherein: Collecting data related to identified persons in an external environment near the vehicle from a remote source in communication with the controller further includes: Data related to the identified person in the external environment near the vehicle is collected from at least one of a social media application, a criminal record, a business network, a motor vehicle record, a real estate record, a voting record, and a school record.

6. The method according to claim 5, wherein: Identifying, with the controller, at least one related person among the identified persons based on the data collected from the remote source further comprises: Identify at least one related person among the identified persons as at least one of the following: followers of the occupant on social media, people followed by the occupant on social media, friends of the occupant, family members of the occupant, celebrities, colleagues of the occupant, people with similar interests to the occupant, and peers of the occupant.

7. The method according to claim 6, wherein, using the controller, describing the at least one related person further comprises: The at least one associated person is described as friendly or unfriendly based on data collected from the remote source related to the identified person using a cloud computing algorithm in communication with the controller.

8. The method according to claim 1, wherein: Utilizing the head-up display system, displaying appropriate augmented graphics on the inner surface of the windshield of the vehicle further comprises: determining, using an occupant monitoring system, the position of the occupant's head and eyes; calculating a location on the inner surface of the windshield to display an appropriate augmented graphic based on the position of the occupant's head and eyes, wherein the augmented graphic appears proximate to the identified at least one person of interest when viewed by the occupant; and Displaying a suitable enhanced graphic on the inner surface of said windshield by: Using the light source of the digital light projector to project the excitation light; focusing the excitation light from the light source using a condenser lens; Using color filters, the focused excitation light is separated into red, green, and blue light; focusing the excitation light transmitted through the color filter by using a shaping lens; Redirecting the excitation light using a digital micromirror device (DMD); receiving the excitation light from the digital micromirror device using a projection lens; and Using the projection lens, the excitation light is projected onto a position on the inner surface of the windshield corresponding to the position of the head and eyes of the occupant, wherein the luminescent particles in the windshield emit visible light in response to absorbing the excitation light, and when the occupant views the enhanced graphics, a suitable enhanced graphics appears near at least one identified relevant person.

9. The method according to claim 1, further comprising: Determining, using the controller, whether the windshield is in a privacy mode, wherein in the privacy mode, the windshield is opaque, the occupant cannot see the external environment through the windshield, and the occupant cannot be seen by people in the external environment; as well as When the windshield is in the privacy mode, the privacy mode is deactivated prior to displaying a suitable augmented graphic on an interior surface of the windshield.

10. A system for identifying relevant persons for passengers in a vehicle using a head-up system, comprising: Controller that communicates with external scene cameras, suitable for: capturing an image of the external environment near the vehicle; identifying at least one person of interest in an external environment located in the vicinity of the vehicle; describing the at least one relevant person; determining an appropriate augmented graphic to be displayed to the occupant related to the identified at least one person of interest; as well as The enhanced graphics are displayed on an interior surface of a windshield of the vehicle using the heads-up display system.

Images