Method and system for predicting driver intent and highlighting useful data with a camera

Abstract

In an example embodiment, a vehicle is provided that includes an interior camera, one or more exterior systems, a display system, and a processor. The interior camera is configured to obtain interior camera data related to a line of sight of a driver of the vehicle interior. The one or more exterior systems are configured to obtain exterior data regarding a region of interest exterior to the vehicle corresponding to the line of sight of the driver. The processor is coupled to the interior camera and the one or more exterior systems and is configured to facilitate, at least in part, providing feedback related to the region of interest to the driver of the vehicle via the display system based on the interior camera data and the exterior data.

Description

Technical Field

[0001] The technical field generally relates to vehicles, and more specifically to methods and systems for predicting driver intentions using cameras and highlighting useful data to the vehicle driver. Background Technology

[0002] Today, some vehicles are equipped with telematics units that provide information about points of interest near the vehicle via navigation systems. However, in some cases, such telematics units and navigation systems may not always provide optimal information.

[0003] Therefore, it may be desirable to provide improved methods and systems for providing drivers with information, for example, about nearby points of interest, based on driver intent. Furthermore, other desirable features and characteristics of the invention will become apparent from the accompanying drawings and this background information, as well as from the subsequent detailed description and appended claims. Summary of the Invention

[0004] According to an exemplary embodiment, a method is provided, the method comprising: obtaining interior camera data relating to the driver's line of sight inside a vehicle from one or more interior cameras disposed inside a vehicle; obtaining external data relating to a region of interest outside the vehicle corresponding to the driver's line of sight via one or more systems configured to generate external data relating to the environment outside the vehicle; and providing feedback relating to the region of interest via a processor based on the interior camera data and the external data.

[0005] Similarly, in an exemplary embodiment, external data about the region of interest outside the vehicle corresponding to the driver's line of sight is obtained via one or more external cameras facing the outside of the vehicle.

[0006] Similarly, in an exemplary embodiment, the detection of external data concerning the region of interest outside the vehicle is further enhanced by combining map data with one or more satellite-based positioning systems.

[0007] Also in an exemplary embodiment, the method further includes: identifying points of interest near the vehicle based on camera data and additional data; and obtaining relevant information about the points of interest from a computer database; wherein the step of providing feedback includes providing instructions to the vehicle driver via a processor to provide relevant information about the points of interest.

[0008] Similarly, in an exemplary embodiment, the step of obtaining relevant information includes obtaining advertising or promotional information related to the point of interest; and the step of providing feedback includes providing instructions to the vehicle's display via a processor to provide the vehicle driver with advertising or promotional information related to the point of interest.

[0009] Similarly, in an exemplary embodiment, the step of providing feedback includes providing instructions via a processor to the vehicle's visual display screen or visual projection system to provide the vehicle driver with relevant information related to the point of interest.

[0010] In another exemplary embodiment, a system is provided that includes an interior camera, one or more external systems, and a processor. The interior camera is configured to acquire interior camera data relating to the line of sight of a driver inside a vehicle. The one or more external systems are configured to acquire external data relating to a region of interest outside the vehicle corresponding to the driver's line of sight. The processor is coupled to the interior camera and the one or more external systems and is configured to at least contribute to providing feedback relating to the region of interest based on the interior camera data and the external data.

[0011] Similarly, in an exemplary embodiment, one or more external systems include one or more external cameras facing the exterior of the vehicle.

[0012] Also in an exemplary embodiment, one or more external systems further include one or more satellite-based positioning systems, wherein external data regarding the area of ​​interest outside the vehicle is further enhanced by combining map data via one or more satellite-based positioning systems.

[0013] Also in an exemplary embodiment, the processor is further configured to at least assist in: identifying points of interest near the vehicle based on camera data and external data; obtaining relevant information about the points of interest from a computer database; and providing instructions to the vehicle's display to provide the vehicle driver with relevant information about the points of interest.

[0014] Also in an exemplary embodiment, the processor is configured to at least assist in: obtaining advertising or promotional information related to a point of interest; and providing instructions to a display of the vehicle to provide the vehicle driver with advertising or promotional information related to a point of interest.

[0015] Also in an exemplary embodiment, the processor is configured to at least assist in: providing instructions to the vehicle's visual display screen and providing the vehicle driver with relevant information related to the point of interest.

[0016] Similarly, in an exemplary embodiment, the processor is configured to at least assist in: providing instructions to a visual projection system inside the vehicle and providing the vehicle driver with relevant information related to a point of interest.

[0017] In another exemplary embodiment, a vehicle is provided, including an interior camera, one or more external systems, a display system, and a processor. The interior camera is configured to acquire interior camera data relating to the line of sight of a driver inside the vehicle. The one or more external systems are configured to acquire external data relating to a region of interest outside the vehicle corresponding to the driver's line of sight. The processor is coupled to the interior camera and the one or more external systems and is configured to at least facilitate providing the driver of the vehicle with feedback relating to the region of interest via the display system based on the interior camera data and the external data.

[0018] Similarly, in an exemplary embodiment, one or more external systems include one or more external cameras facing the exterior of the vehicle.

[0019] Also in an exemplary embodiment, one or more external systems further include one or more satellite-based positioning systems, wherein external data regarding the area of ​​interest outside the vehicle is further enhanced by combining map data via one or more satellite-based positioning systems.

[0020] Also in an exemplary embodiment, the processor is further configured to at least assist in: identifying points of interest near the vehicle based on camera data and additional data; obtaining relevant information about the points of interest from a computer database; and providing instructions to a display system to provide the driver with relevant information about the points of interest.

[0021] Also in an exemplary embodiment, the processor is configured to at least assist in: obtaining advertising or promotional information related to a point of interest; and providing instructions to a display of the vehicle to provide the vehicle driver with advertising or promotional information related to a point of interest.

[0022] Similarly, in an exemplary embodiment, the display system includes a visual display screen.

[0023] Similarly, in an exemplary embodiment, the display system includes a visual projection system. Attached Figure Description

[0024] The present disclosure will be described below in conjunction with the accompanying drawings, wherein like reference numerals denote like elements, and wherein:

[0025] Figure 1 This is a functional block diagram of a communication system according to an exemplary embodiment, including a vehicle with a control system configured to determine the driver's intention and provide relevant information to the vehicle driver;

[0026] Figure 2 This is a description based on an exemplary embodiment. Figure 1 A flowchart of the functions of the control system; and

[0027] Figure 3 This is a flowchart of a process for determining a driver's intention and providing relevant information to the vehicle driver, according to an exemplary embodiment. This process can be combined with... Figure 1 The communication systems (including its vehicles), Figure 1 and 2 The control system and its other components are used together. Detailed Implementation

[0028] The following detailed description is exemplary in nature only and is not intended to limit this disclosure or its application and use. Furthermore, it is not intended to be bound by any theories presented in the preceding background or the following detailed description.

[0029] Figure 1 This is a functional block diagram of a communication system 10 according to an exemplary embodiment. As described in further detail below, the communication system 10 includes a vehicle 12, which includes a control system 11 configured to determine a driver's intention and provide relevant information to the driver 13 of the vehicle 12. As described below in conjunction with... Figure 1 as well as Figure 2 Further described, in various embodiments, the control system 11 includes a processor 38, a computer memory 40, a satellite-based positioning system component (e.g., GPS) 42, an internal camera 73, an external camera 74, and one or more displays 67.

[0030] like Figure 1 As shown, in some embodiments, user 13 also has device 15, such as a smartphone, computer and / or other electronic device 15, which can communicate with both user 13 and vehicle 12.

[0031] like Figure 1 As shown, the communication system 10 typically includes a vehicle 12 and one or more wireless carrier systems 14, one or more terrestrial networks 16, and one or more remote servers 18. It should be understood that the overall architecture, setup, and operation, as well as the various components of the illustrated system, are merely exemplary, and examples of the methods disclosed herein can be implemented using communication systems with different configurations. Therefore, the following paragraphs, which provide a brief overview of the illustrated communication system 10, are not intended to be limiting.

[0032] Vehicle 12 can be any type of mobile vehicle, such as a motorcycle, car, truck, recreational vehicle (RV), boat, airplane, agricultural equipment, etc., and is equipped with suitable hardware and software to enable it to communicate through communication system 10. Figure 1As shown, in various embodiments, vehicle hardware 20 is disposed within the body 19 of vehicle 12 and includes a telematics unit 24, a microphone 26, a speaker 28, and buttons and / or controls 30 connected to the telematics unit 24. A network connection or vehicle bus 32 is operatively coupled to the telematics unit 24. Examples of suitable network connections include Controller Area Network (CAN), System Transmission for Media (MOST), Local Interconnect Network (LIN), Ethernet, and other suitable connections, such as those conforming to known ISO (International Organization for Standardization), SAE (Society of Automotive Engineers), and / or IEEE (Institute of Electrical and Electronics Engineers) standards and specifications, to name just a few.

[0033] The telematics unit 24 is an in-vehicle device that provides various services through its communication with the remote server 18, and typically includes an electronic processing device (processor) 38, one or more types of electronic memory 40, a cellular chipset / component 34, a wireless modem 36, a dual-mode antenna 70, and a navigation unit containing a GPS chipset / component 42. In one example, the wireless modem 36 includes a set of computer programs and / or software routines suitable for execution within the electronic processing device 38.

[0034] In various embodiments, the telematics unit 24 may be embedded / installed within the vehicle 12 during manufacturing, or it may be an aftermarket unit installed after the vehicle 12 is manufactured. In various embodiments, the telematics unit 24 enables voice and / or data communication via one or more wireless networks (e.g., wireless carrier system 14) and / or via wireless networking, thereby allowing communication with a remote server 18 and / or other vehicles and / or systems.

[0035] In various embodiments, the telematics unit 24 can use radio transmissions to establish voice and / or data channels with the wireless carrier system 14, thereby enabling the transmission and reception of voice and data transmissions over these channels. Vehicle communication is enabled via a cellular chipset / component 34 for voice communication and a wireless modem 36 for data transmission. This example can use any suitable encoding or modulation technique, including digital transmission techniques such as TDMA (Time Division Multiple Access), CDMA (Code Division Multiple Access), W-CDMA (Wideband CDMA), FDMA (Frequency Division Multiple Access), OFDMA (Orthogonal Frequency Division Multiple Access), etc. In one embodiment, the dual-mode antenna 70 serves both the GPS chipset / component 42 and the cellular chipset / component 34. In various embodiments, the telematics unit 24 utilizes cellular communication according to industry standards such as LTE, 5G, etc. Additionally, in various embodiments, the telematics unit 24 performs wireless networking between the vehicle 12 and one or more other network devices, for example, using one or more wireless protocols, such as one or more IEEE 802.11 protocols, WiMAX, or Bluetooth.

[0036] The telematics unit 24 can provide various services to the user of vehicle 12, including recognizing the driver 13's intention to view nearby points of interest and providing the driver with information related to those points of interest. Additionally, in some embodiments, the telematics unit 24 can also provide supplementary services, such as connectivity to electronic device 15. In various embodiments, the electronic device may include, for example, various consumer electronics / mobile devices, such as smartphones, laptops, smart wearable devices, tablets, web computers, and / or one or more other electronic devices and / or combinations thereof.

[0037] In various embodiments, one or more Short-Range Wireless Connectivity (SRWC) protocols (e.g., Bluetooth / Bluetooth Low Energy or Wi-Fi) may be utilized. In various embodiments, once SRWC is established, electronic device 15 can be bound and / or identified as a network participant of telematics unit 24, for example, for current and future use. For example, in some embodiments, when electronic device 15 is subsequently within wireless range of telematics unit 24 after initial pairing, telematics unit 24 (and / or remote server 18) can confirm that electronic device 15 has been identified as paired or established as a network participant to communicate with and receive services from telematics unit 24.

[0038] In addition, in various embodiments, the telematics unit 24 may also provide other services, such as: segmented navigation and other navigation-related services provided with the GPS chipset / component 42; emergency assistance services; information requests from users of vehicle 12 (e.g., about points of interest along the route of vehicle 12); and / or infotainment-related services, such as music, internet web pages, movies, television programs, video games and / or other content downloaded by infotainment center 46, which may be part of the telematics unit 24 and / or operatively connected to the telematics unit 24 via vehicle bus 32 and audio bus 22; and various other types of possible services.

[0039] For other electronic components used in conjunction with the telematics unit 24, the microphone 26 provides the driver or other vehicle occupants with a means of inputting verbal or other auditory commands and may be equipped with an embedded voice processing unit utilizing human-machine interface (HMI) technologies known in the art. Conversely, the speaker 28 provides auditory output to the vehicle occupants and may be a separate speaker specifically used with the telematics unit 24, or it may be part of the vehicle audio component 64. In either case, the microphone 26 and speaker 28 enable the vehicle hardware 20 and the remote server 18 to communicate with the occupants via auditory voice. The vehicle hardware also includes one or more buttons and / or controls 30 to enable the vehicle occupants to activate or engage one or more vehicle hardware components 20. For example, one of the buttons and / or controls 30 may be an electronic button for initiating voice communication with the remote server 18 (whether by a person such as advisor 58 or an automated call response system). In another example, one of the buttons and / or controls 30 may be used to activate emergency services.

[0040] Audio component 64 is operatively connected to vehicle bus 32 and audio bus 22. Audio component 64 receives analog information via audio bus 22 and presents it as sound. Digital information is received via vehicle bus 32. Audio component 64 provides AM and FM radio, optical disc (CD), digital video disc (DVD), and multimedia functions independent of infotainment center 46. Audio component 64 may include a speaker system, or may utilize speaker 28 via arbitration on vehicle bus 32 and / or audio bus 22. In various embodiments, audio component 64 includes radio system 65 (which in some embodiments also includes antenna 70, as well as amplifiers, speakers, etc.).

[0041] Similarly, in various embodiments, display component 67 provides a visual display to driver 13 of vehicle 12. In various embodiments, display component 67 provides a visual display to driver 13 regarding relevant information related to a point of interest that driver 13 is currently viewing, for example, in conjunction with the following... Figure 2 and 3 A more detailed description follows. For example, in some embodiments, display component 67 may include one or more visual display systems, such as a visual display screen for a vehicle's navigation system, a head-up display (HUD), and / or other visual displays and / or visual projections, such as on the windshield of vehicle 12, etc. In some embodiments, such information may also be provided via an audio system, such as audio component 64.

[0042] Vehicle sensors 72, which are connected to various sensor interface modules 44, are operatively connected to vehicle bus 32.

[0043] In various embodiments, the vehicle sensor 72 includes an internal camera 73 and an external camera 74. In various embodiments, one or more internal cameras 73 (e.g., mounted on the vehicle roof, rearview mirror, dashboard, and / or other locations near the driver 13) measure and monitor the eye position and movement of the driver 13 to generate a vector from inside the vehicle 12 indicating the direction of the driver 13's gaze.

[0044] Similarly, in various embodiments, external cameras 74 are mounted at various locations on the exterior of vehicle 12 to determine the location from which driver 13 is looking outside vehicle 12, such as nearby points of interest. In some embodiments, at least four external cameras 74 are positioned around vehicle 12 to provide a 360-degree view of the exterior and surroundings of vehicle 12.

[0045] Additionally, in various embodiments, vehicle sensor 72 may also include any number of additional sensors, such as gyroscopes, accelerometers, magnetometers, emissions detection and / or control sensors, etc. Similarly, in various embodiments, exemplary sensor interface module 44 includes powertrain control, climate control, and body control, to name just a few.

[0046] In various embodiments, the wireless carrier system 14 can be any number of cellular phone systems, satellite-based wireless systems, and / or any other suitable wireless system, such as one that transmits signals between vehicle hardware 20 and terrestrial network 16 (and / or in some embodiments, communicates directly with vehicle 12 and / or remote server 18). According to some examples, the wireless carrier system 14 may include and / or be coupled to one or more cellular towers 48, satellites 49, base stations, and / or mobile switching centers (MSCs) 50, as well as any other networking components required to connect the wireless carrier system 14 to the terrestrial network 16. As those skilled in the art will understand, various cellular tower / base station / MSC arrangements are possible and can be used with the wireless carrier system 14.

[0047] Land network 16 can be a conventional land-based telecommunications network that connects to one or more land telephones and connects wireless carrier system 14 to remote server 18. For example, land network 16 may include a Public Switched Telephone Network (PSTN) and / or an Internet Protocol (IP) network, as those skilled in the art will understand. Of course, one or more segments of land network 16 may be implemented as a standard wired network, fiber optic or other optical network, cable network, other wireless networks such as wireless local area network (WLAN) or a network providing broadband wireless access (BWA), or any combination thereof.

[0048] The remote server 18 is designed to provide a number of different system backend functions to the vehicle hardware 20, and, according to the example shown here, typically includes one or more switches 52, a server 54, a database 56, an advisor 58, and various other telecommunications / computer equipment 60. These various call center components are appropriately coupled to each other via network connections or buses 62 (such as those previously described in conjunction with vehicle hardware 20). The switch 52 (which may be a dedicated small switch (PBX)) routes input signals to typically send voice transmissions to the advisor 58 or an automated response system, and to pass data transmissions to modems or other telecommunications / computer equipment 60 for demodulation and further signal processing.

[0049] Modem or other telecommunications / computer equipment 60 may include encoders, as previously described, and may be connected to various devices, such as server 54 and database 56. For example, database 56 may be designed to store subscriber profile records, subscriber behavior patterns, or any other relevant subscriber information. Although the illustrated example has been described as being used in conjunction with a manned remote server 18, it should be understood that remote server 18 may be any central or remote facility, manned or unmanned, mobile or fixed.

[0050] Figure 2 This is a description based on an exemplary embodiment. Figure 1 A flowchart illustrating the functions of the control system 11. (For example...) Figure 2 As shown, and similar to the discussion above, in various embodiments, the control system 11 includes Figure 1 The processor 38, computer memory 40, location (e.g., GPS) component 42, internal camera 73, external camera 74, and display system 75.

[0051] Similarly, Figure 2 As shown, in various embodiments, the interior camera 73 provides data (e.g., images and / or vector coordinates) of the driver 13's eyes from inside the vehicle 12, and the processor 38 uses this data to determine a gaze vector indicating the position the driver 13 is looking from inside the vehicle 12 (e.g., using algorithm 202 to detect the position the driver 13 is looking at).

[0052] Similarly, in various embodiments, processor 38 utilizes gaze vectors and map data / location (e.g., GPS) data 204 obtained from memory 40 and / or GPS system 42 to identify points of interest (POIs) that driver 13 is viewing (e.g., using algorithm 204 to identify POIs in the scene near vehicle 12). Additionally, in various embodiments, the processor also obtains POI data 208 (e.g., marketing and / or advertising data related to nearby POIs) and filters the POI data for the identified POIs that driver 13 is viewing (e.g., using algorithm 210 to filter data based on the identified POIs).

[0053] Additionally, in various embodiments, the processor 38 also provides filtered data 214 to the driver 13 on the display system 67 of the vehicle 12 (e.g., as part of a visual display screen and / or as a projection onto the windshield of the vehicle 12 via a visual projection system). In some embodiments, the display is also provided in conjunction with additional video 212 provided from one or more other vehicle systems, such as augmented reality video for the vehicle 12.

[0054] Figure 3 This is a flowchart of a process 300 for determining a driver's intention and providing relevant information to the vehicle driver, according to an exemplary embodiment. In various embodiments, process 300 may be... Figure 1 The communication system 10 (including its vehicle 12) Figure 1 and 2 The control system 11 and its other components are used together.

[0055] like Figure 3 As shown, in various embodiments, process 300 begins at step 302. In some embodiments, process 300 begins when one or more users of vehicle 12 (e.g., the driver) approach or enter vehicle 12, or start vehicle 12 and / or thereby ignite it (e.g., by turning a key, engaging a key card, or pressing a start button, etc.) and / or when starting or initiating vehicle drive. In some embodiments, the steps of process 300 are performed continuously during operation of vehicle 12.

[0056] In step 304, the driver's eyes are monitored. In various embodiments, Figure 1 One or more internal cameras 73 (inside the vehicle compartment of vehicle 12) monitor the driver 13’s eyes and their position and / or movement through internal camera data obtained by the internal cameras 73, including the direction of the gaze of one or both eyes of the driver 13.

[0057] During step 306, the driver's line of sight is analyzed. Specifically, in various embodiments, Figure 1The processor 38 utilizes the internal camera data from the internal camera 73 in step 304 to generate an eye gaze vector for the driver 13 using one or more extrinsic and intrinsic parameters. In various embodiments, examples of intrinsic parameters may include camera position (x, y, z) and camera orientation (pitch, roll, yaw), as well as other possible parameters. Also in various embodiments, examples of intrinsic parameters may include camera lens distortion coefficients and camera focus, as well as other possible parameters.

[0058] During step 308, one or more external devices are used to interpret the gaze vector from step 306. In some embodiments, processor 38 selects one or more external cameras 74 of vehicle 12 based on the gaze vector, such as external cameras 74 near the gaze vector based on external data about the external environment around vehicle 12 generated by one or more external systems. Additionally, in various embodiments, processor 38 also converts the vector to camera coordinates and maps the vector to certain objects or points within the field of view of the external camera 74 to further determine what the driver is specifically looking at.

[0059] Furthermore, in some other embodiments, for example, a satellite-based positioning system (such as GPS system 42) is used in conjunction with map data stored in computer memory 40 for location data corresponding to the eye line vector. In various embodiments, the external camera 74 and the satellite-based positioning system (such as GPS system 42) are combined to interpret the eye line vector and particularly help identify locations near vehicles that correspond to the eye line vector, including points of interest.

[0060] In various embodiments, the gaze vector is converted into a region of interest during step 310. Specifically, in various embodiments, processor 38 uses intrinsic and extrinsic parameters to convert the gaze vector in world coordinates into a region of interest. In some embodiments, processor 38 converts the gaze vector in world coordinates into a region of interest around vehicle 12 in one or more 2D images of an external camera identified in step 308. In some other embodiments, processor 38 converts the gaze vector in world coordinates into a point of interest around vehicle 12, which is mapped to a satellite-based positioning system (e.g., GPS system 42) identified in step 308.

[0061] Therefore, in various embodiments, external cameras and perception from those external cameras are used to identify what the driver is looking at. In some embodiments, GPS / map data can be used to further enhance the identification. Additionally, in some embodiments, the identification can be further enhanced by incorporating audio functionality. For example, in some embodiments, the driver can speak to a person and / or an automated advisor, and this can be combined with internal and external camera data to further enhance the identification of points of interest that the driver is looking at, as described in further detail below.

[0062] Similarly, in various embodiments, step 312 determines whether any points of interest have been set in the region of interest in step 310. In some embodiments, Figure 1 The processor 38 determines the location based on the mapping or correspondence between the line-of-sight vector from step 306 and the external camera and / or GPS data from steps 308 and 310. Figure 1 Whether any points of interest are set in the area of ​​interest viewed by driver 13. In various embodiments, a "point of interest" can refer to any of a restaurant, convenience store, grocery store, other types of store, gas station, service station, accommodation, tourist destination and / or many other different types of points of interest.

[0063] In some embodiments, during step 312, external camera images are used to identify points of interest. Additionally, in some embodiments, external camera data can be combined with GPS and map data to further enhance the identification of points of interest. Also in some embodiments, for example, if the driver is already using a navigation system and / or a personal and / or virtual assistant and / or advisor (e.g.,...),... Figure 1 Consultant 58, and / or through Figure 1 If a driver inputs a specific point of interest (POI) and / or POI type into an electronic device (such as an access advisor), the driver's input can also be used to identify the POI. For example, as described above, in some embodiments, the driver can speak to the person and / or the automated advisor, and this can be combined with internal and external camera data to further enhance the identification of the POI that the driver is viewing. For example, in one embodiment, the driver can say "I'm hungry" to the advisor, and the external camera can extract features from a nearby fast food restaurant (or other food establishment), then augment those features using GPS / map data, and further enhance the identification by providing marketing information related to that restaurant in the form of augmented reality text projected onto the windshield or video screen and / or other or more other types of displays for the driver inside the vehicle.

[0064] In various embodiments, if it is determined in step 312 that no corresponding point of interest has been identified, the process returns to step 304 as described above.

[0065] Conversely, if instead of determining that a corresponding point of interest has been identified in step 312, the process proceeds instead to step 314, as described below.

[0066] During step 314, data and information related to the points of interest identified in step 312 (collectively referred to herein as relevant information or information related to the points of interest) are retrieved. Specifically, in various embodiments, Figure 1 The processor 38 retrieves data related to the point of interest from the memory 40. For example, the data may relate to opening hours, pricing, sales, menus, and / or other information related to the point of interest. For instance, if the point of interest is a restaurant and / or fast food establishment, the information may relate to any sales of menu food and / or beverage items. In various embodiments, data / information may be obtained directly from the point of interest and / or indirectly through one or more third parties and / or intermediaries.

[0067] Additionally, in various embodiments, step 316 provides the vehicle's driver with relevant information related to the point of interest. In various embodiments, Figure 1 processor 38 Figure 1 The display system 67 provides instructions to display and / or project with Figure 1 The relevant advertising information and / or other data relating to the driver 13's points of interest. For example, in some embodiments, the data / information is provided to the driver 13 via a visual display for a vehicle's navigation system, a head-up display (HUD), and / or other visual displays (such as on the windshield of vehicle 12). In some other embodiments, the relevant information may also be provided via an audio system, such as the audio component 64 of vehicle 12.

[0068] Therefore, in various embodiments, during step 316, relevant information is provided to driver 13 based on the specific point of interest targeted by driver 13's gaze, such as based on internal camera 73 (inside vehicle 12, detecting driver 13's gaze and the vector it represents) and external camera 74, as well as a satellite-based positioning system such as GPS (outside vehicle 12, confirming real-world coordinates and identifying the specific point of interest corresponding to driver 13's gaze vector). In some embodiments, advertising and / or promotional information is provided to driver 13 only on the condition that driver 13 has previously agreed to receive such advertising.

[0069] In various embodiments, the process then terminates at step 318. In some embodiments, the process may resume at steps 302 and / or 304 when the travel of vehicle 12 has resumed through the point of interest.

[0070] Therefore, according to exemplary embodiments, methods and systems are provided for determining a driver's intention and providing relevant information to the vehicle driver. In various embodiments, one or more interior cameras of the vehicle measure data about the direction the driver is looking at relative to one or more of the driver's eyes. Also in various embodiments, a processor (and / or interior camera) generates vectors that include the direction the driver is looking at from inside the vehicle. Additionally, in various embodiments, an external camera and a satellite-based positioning system (e.g., a GPS system) capture data related to the point of interest being viewed by the driver based on vectors and data about the point of interest. Furthermore, in various embodiments, a processor obtains and filters relevant data about the point of interest being viewed by the driver and provides instructions to display and / or project information for the vehicle driver.

[0071] It should be understood that the system and method may differ from those depicted in the accompanying drawings and described herein. For example, in various embodiments, Figure 1 The communication system (including its vehicles and components) may differ from Figure 1 As depicted in and / or described herein. Similarly, it will be recognized that, Figure 1 and 2 The control system and / or its various components and / or their functions may differ from those of the control system and / or its various components and / or functions. Figure 1 and 2 The descriptions herein and the accompanying descriptions. It should also be understood that the processes (and / or subprocesses) disclosed herein may differ from those described herein and / or... Figure 3 The process shown, and / or may be performed simultaneously and / or as described herein and / or Figure 3 The different orders in which the steps are performed, as well as other possible variations, are described in the text.

[0072] Although at least one exemplary embodiment has been presented in the foregoing detailed description, it should be understood that numerous variations exist. It should also be understood that the one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of this disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient roadmap for implementing one or more exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the scope of this disclosure as set forth in the appended claims and their legal equivalents.

Claims

1. A method comprising: Internal camera data relating to the driver's line of sight inside the vehicle is obtained from one or more internal cameras located inside the vehicle. This internal camera data includes multiple intrinsic parameters regarding the position and orientation of the one or more internal cameras. The driver's eye gaze vector is generated by the processor based on internal camera data, including multiple intrinsic parameters regarding the position and orientation of one or more internal cameras; The system comprises one or more systems configured to generate external data relating to the environment outside the vehicle, including obtaining external data about a region of interest outside the vehicle corresponding to the driver’s eye line vector by using an external camera of the vehicle, and determining whether any point of interest is set in the region of interest; Based on what the driver says to the consultant, combined with eye gaze vectors and external data, the processor identifies points of interest. as well as Based on what the driver says to the advisor, and combined with data from both internal and external cameras, the identification of points of interest that the driver is looking at can be further enhanced.

2. The method according to claim 1, wherein, By combining map data with one or more satellite-based positioning systems, the detection of external data regarding areas of interest outside the vehicle is further enhanced.

3. The method according to claim 1, further comprising: Obtain relevant information about points of interest from computer databases; The feedback process includes providing instructions via the processor to the vehicle's display to provide the driver with relevant information about the point of interest obtained from a computer database.

4. The method according to claim 3, wherein: The steps for obtaining relevant information include obtaining advertising or promotional information related to points of interest from the computer database; and The feedback process includes instructions sent via the processor to the vehicle's display to provide the driver with advertising or promotional information related to points of interest obtained from a computer database.

5. The method according to claim 3, wherein, The feedback process includes providing instructions via a processor to the vehicle’s visual display or visual projection system to provide the driver with relevant information related to the point of interest.

6. A vehicle comprising: An interior camera configured to acquire interior camera data relating to the driver's line of sight inside the vehicle, the interior camera data including multiple intrinsic parameters regarding the position and orientation of one or more interior cameras; A processor, coupled to an internal camera, is configured to at least contribute to generating a driver’s eye gaze vector based on internal camera data, including multiple intrinsic parameters regarding the position and orientation of one or more internal cameras; One or more external systems configured to acquire external data about a region of interest outside the vehicle corresponding to the driver's eye gaze vector, including via external cameras of the vehicle, and to determine whether any points of interest are set within the region of interest; and Display system; The processor is also configured to at least contribute to: Based on what the driver said to the consultant, combined with eye gaze vectors and external data, points of interest were identified; and Based on what the driver says to the advisor, and combined with data from both internal and external cameras, the identification of points of interest that the driver is looking at can be further enhanced.

7. The vehicle according to claim 6, wherein, The processor is also configured to at least contribute to: Obtain relevant information about points of interest from computer databases; and Instructions are provided to the display system to provide the driver with relevant information about the point of interest obtained from the computer database.

8. The vehicle according to claim 7, wherein, The processor is configured to at least facilitate: Obtain advertising or promotional information related to points of interest; and Instructions are given to the vehicle's displays to provide the driver with advertising or promotional information related to points of interest.

Images