Vehicle occupant gaze detection system and method of use

Abstract

The present disclosure relates to a vehicle occupant gaze detection system and a method of use. The vehicle occupant gaze detection system includes a non-transitory computer readable medium configured to store commands and a processor coupled to the non-transitory computer readable medium. The processor is configured to execute the commands to receive gaze data from a vehicle. The gaze data includes a direction observed by an occupant of the vehicle, vehicle information, and timestamp information. The processor is configured to execute the commands to generate a grid map including a grid array, each grid of the grid array corresponding to a location. The processor is configured to execute the commands to generate a histogram including information related to at least one location being seen by the occupant of the vehicle, determine an object's validity based on the histogram, and send a suggestion based on the object's validity. Thus, the effectiveness of an advertisement is estimated based on actual gaze results of a vehicle occupant.

Description

Technical Field

[0001] This disclosure relates to a vehicle occupant gaze detection system and its usage. Background Technology

[0002] In many places, advertisements are displayed on signs around the road. The price of different advertising locations varies based on the assumed amount of time the advertisement is perceived. The advertiser's (the business or individual placing the advertisement) goal is to get drivers, who are vehicle occupants, to see the advertisement and purchase the advertised product or service.

[0003] In some instances, electronic signs change periodically, displaying different advertisements at different times. As a result, it is possible to display multiple advertisements in the same location using electronic signs. Summary of the Invention

[0004] The technical solution disclosed herein relates to a vehicle occupant gaze detection system, comprising a non-transient computer-readable medium configured to store commands, and a processor connected to the non-transient computer-readable medium. The processor is configured to execute commands to receive gaze data from a vehicle, including information related to the direction observed by the occupant of the vehicle, vehicle information, and timestamp information; generate a grid map based on map data and the received gaze data, the grid map having a grid array in which each grid point corresponds to one of a plurality of locations; generate a histogram based on the grid map, the histogram containing information related to the fact that at least one of the plurality of locations is seen by the vehicle occupant based on timestamp information; determine the validity of an object located at at least one location based on the histogram; and send a recommendation based on the validity of the object to a customer. Attached Figure Description

[0005] Figure 1 This is a schematic diagram of a vehicle occupant gaze detection system according to some implementation methods.

[0006] Figure 2 It is a diagram of the data structure of the gaze results detected according to some implementation methods.

[0007] Figure 3 It is a diagram of the data structure of the gaze results detected according to some implementation methods.

[0008] Figure 4 It is a diagram of the data structure of the gaze results detected according to some implementation methods.

[0009] Figure 5 It is a diagram of the data structure of gaze data according to some implementation methods.

[0010] Figure 6This is a flowchart of a method for using a vehicle occupant gaze detection system according to some implementation methods.

[0011] Figure 7 It is a graph of the data structure of a raster map according to some implementation methods.

[0012] Figure 8 It is a diagram of the data structure of gridded data according to some implementation methods.

[0013] Figure 9 It is a diagram of the data structure of log entries according to some implementation methods.

[0014] Figure 10 It is a graph of a histogram data structure based on some implementation methods.

[0015] Figure 11 This is a diagram of a system for implementing a vehicle occupant gaze detection system according to some embodiments. Detailed Implementation

[0016] The technical solutions of this disclosure can be best understood by reading the following detailed description in conjunction with the accompanying drawings. It should be noted that, according to standard practice in the art, the various features are not depicted to a fixed scale. In fact, the dimensions of the various features may be arbitrarily increased or decreased for clarity of description.

[0017] The following disclosure provides numerous different implementations or examples for achieving various features of the provided subject matter. For the sake of brevity, specific examples of constituent elements, values, actions, materials, configurations, etc., are described below. Of course, these are merely examples and are not intended to be limiting. Other constituent elements, values, actions, materials, configurations, etc., may also be considered. For example, the formation of the first feature on the second feature in the following description can include implementations where the first and second features are in direct contact, and can also include implementations where an additional feature is formed between the first and second features so that the first and second features may not be in direct contact. Furthermore, the present disclosure may repeat reference numerals and / or text in various examples. This repetition is for the purpose of brevity and clarity and does not itself determine the relationship between the various implementations and / or structures studied.

[0018] Furthermore, terms indicating relative spatial positions, such as "below," "below," "lower," "above," and "upper," may also be used in this text, as illustrated in the figures, to facilitate the description of the relationship between an element or feature and other elements or features. The terms indicating relative spatial positions are intended to include not only the orientation shown in the figures but also the different orientations of the device in use or in operation. The device can be in other orientations (rotated 90 degrees or other orientations), and the descriptors indicating relative spatial positions used here can also be interpreted accordingly.

[0019] Currently, advertisers cannot accurately determine what types of people will look at roadside signs and at what frequency. Consequently, advertisers cannot determine how effective signs are at attracting customers, nor whether the price paid for advertising is worthwhile in terms of creating the desired customer base. While it can be assumed that signs along high-traffic roads are more valuable than those along low-traffic roads, there is still no reliable benchmark regarding the number of potential customers looking at the signs. Similarly, while it can be assumed that slower-moving areas are more valuable than faster-moving areas, there is still no accurate method to measure whether signs actually attract the attention of a higher number of potential customers.

[0020] To provide reliable information, along with demographic information about potential customers, for determining whether a sign has attracted their attention, a vehicle occupant gaze detection system is used in combination with map data to determine whether vehicle occupants are looking at the sign. While the following description pertains to roadside signs, those skilled in the art will understand that the vehicle occupant gaze detection system can also be applied to different types of advertising materials, such as movable signs that can be moved by aircraft or other vehicles, people carrying signs, displays in shop windows, or other similar advertisements.

[0021] Vehicle occupant gaze detection systems capture images of more than one eye of vehicle occupants, such as the driver, front seat occupants, or rear seat occupants. If there are signs, the system uses the captured images, along with map data and / or images from outward-facing detectors installed in the vehicle, to determine which sign the occupant is looking at and for how long. In some implementations, the system also collects occupant data such as predicted age, predicted height, predicted weight, hair length, hair color, and clothing to infer the occupant's demographics. Based on this information, the system can generate historical data about the number of people who might be looking at signs or interested in locations where advertisers may potentially place ads in the future, even if there are currently no signs. This information can be used by advertisers to determine whether their advertising has the desired impact. It can also be used by signage rental agencies to help determine appropriate pricing for different signage locations.

[0022] Figure 1 This is a schematic diagram of a vehicle occupant gaze detection system 100 according to some embodiments. The vehicle occupant gaze detection system 100 includes a vehicle system 1 configured to collect data and generate gaze data 30. The vehicle occupant gaze detection system 100 also includes a server 2 configured to receive gaze data 30 and parse the gaze data 30 to generate advertising suggestions. The vehicle occupant gaze detection system 100 also includes a customer 3 configured to receive the generated advertising suggestions.

[0023] Vehicle system 1 includes an electronic control unit (ECU) 8, which is configured to receive signals from a driver monitoring camera 4, a forward-facing camera 5, a global positioning system (GPS) 6, and a ground-based camera. Figure 7 The ECU8 receives data. It includes a gaze detector 10 configured to receive data from the driver monitoring camera 4 and detect gaze direction and / or gaze depth based on the received data. The ECU8 also includes an object detector 11 configured to receive data from the forward-facing camera 5 and determine the position of any detected object based on the received data. The ECU8 further includes a positioning unit 12 configured to receive data from a GPS 6, a ground-based camera 4, and a GPS 7, a ground-based camera 5, and a ground-based camera 6. Figure 7 Object detector 11 and road recognition device ( Figure 1(Not shown in the figure) Receives data and determines the vehicle's position, as well as its posture and state relative to detected objects and / or known objects and / or the road. In some embodiments, the vehicle's position also means the vehicle's position vector. The vehicle's posture and state mean the vehicle's speed and path. In some embodiments, the vehicle's posture and state also mean the vehicle's velocity vector, acceleration vector, and jerk (jump) vector. In some embodiments, the position vector, velocity vector, acceleration vector, and jerk (jump) vector may also include angle vectors. In some embodiments, the vehicle's state also means whether the vehicle's engine or motor is operating. The ECU8 also includes a gaze data generator 20, configured to receive information from the gaze detector 10, the object detector 11, and the positioning unit 12, and generate gaze data 30.

[0024] A driver monitoring camera 4 is configured to capture images of the driver of the vehicle. The driver monitoring camera 4 is connected to the vehicle. In some embodiments, the driver monitoring camera 4 includes a visible light camera. In some embodiments, the driver monitoring camera 4 includes an infrared (IR) camera or other suitable sensor. In some embodiments, the driver monitoring camera 4 is movable relative to the vehicle to capture images of at least one eye of the driver of varying sizes. While capturing images of both eyes is preferred, there are instances where the driver has only one eye, and in some cases where the driver's head is not facing the driver monitoring camera 4, only one eye can be captured. In some embodiments, the driver monitoring camera 4 is automatically adjusted. In some embodiments, the driver monitoring camera 4 can be manually adjusted. In some embodiments, the captured images include at least one eye of the driver. In some embodiments, the captured images include additional information about the driver, such as approximate height, approximate weight, hair length, hair color, clothing, or other suitable information. In some embodiments, the driver monitoring camera 4 includes multiple imaging devices to capture images of different parts of the driver. In some embodiments, the driver monitoring camera is positioned in various locations within the vehicle. For example, in some embodiments, a first driver monitoring camera 4 is positioned close to a rearview mirror in the central area of ​​the vehicle, and a second driver monitoring camera 4 is positioned close to the driver's side door. Those skilled in the art will recognize that other locations for the driver monitoring camera 4 that do not impede the movement of the vehicle are also within the scope of this disclosure. In some embodiments, data from the driver monitoring camera 4 includes timestamps or other metadata to aid synchronization with other data.

[0025] Those skilled in the art will understand that in some embodiments, vehicle system 1 includes additional cameras for monitoring other occupants. Each additional camera is similar to the driver monitoring camera 4 described above. For example, in some embodiments, more than one monitoring camera is installed in the vehicle to capture images of at least one eye of a front seat occupant. In some embodiments, more than one monitoring camera is installed in the vehicle to capture images of at least one eye of a rear seat occupant. In some embodiments, the additional cameras are activated only in response to the vehicle detecting the corresponding front seat occupant or rear seat occupant. In some embodiments, the vehicle operator can also selectively deactivate the additional cameras. In embodiments including additional cameras, the captured images are still sent to gaze detector 10, which is capable of generating gaze results for each monitored occupant of the vehicle.

[0026] The forward-facing camera 5 is configured to capture images of the vehicle's surrounding environment. In some embodiments, the forward-facing camera 5 includes a visible light camera and an infrared camera. In some embodiments, the forward-facing camera 5 may be replaced with a LiDAR (Light Detection and Ranging) sensor, a RADAR (Radio Detection and Ranging) sensor, a SONAR (Sound Navigation and Ranging) sensor, or other suitable sensors, or may be further supplemented with such sensors. In some embodiments, the forward-facing camera 5 includes additional cameras located at other locations within the vehicle. For example, in some embodiments, additional cameras are located on the sides of the vehicle to detect a wider portion of the environment to the left and right of the vehicle. Vehicle occupants can see outside through the side windows of the vehicle, so using additional cameras to detect a wider portion of the environment surrounding the vehicle helps improve the accuracy of determining the objects observed by the vehicle occupants. For example, in some embodiments, additional cameras are located on the rear side of the vehicle to detect a wider portion of the environment to the rear of the vehicle. This information helps to capture additional objects that vehicle occupants other than the driver can observe from the rear window. The forward-facing camera 5 can also capture images to determine whether there are obstacles, such as buildings, between the known location of objects like signs and the occupants of the vehicle. In some implementations, the data from the forward-facing camera 5 includes timestamps or other metadata to help synchronize the data from the forward-facing camera 5 with the data from the driver monitoring camera 4.

[0027] GPS 6 is used to determine a vehicle's location. Knowing the vehicle's location helps associate objects and directions that attract the attention of occupants with the location. Figure 7The data includes objects and areas with known locations. Knowing the vehicle's route helps predict which direction the driver is looking, aiding in the generation of gaze data. Knowing the vehicle's speed helps determine how long it takes for vehicle occupants to see the sign. For example, it can be determined that occupants in a low-speed vehicle who are judged to have looked at the sign for two seconds have less interest in the advertised product or service compared to occupants in a high-speed vehicle who are judged to have looked at the sign for two seconds.

[0028] land Figure 7 It contains information related to roads and known objects along the roads. In some implementations, the land... Figure 7 It can be used in conjunction with GPS6 to determine the vehicle's location and route. In some implementations, it receives data from an external device such as server 2. Figure 7 In some implementations, the location is updated periodically based on information from the forward-facing camera 5 and / or GPS 6. Figure 7 In some implementations, the location is updated periodically based on information received from external devices. Figure 7 In some implementations, Simultaneous Localization and Mapping (SLAM: Simultaneous Self-Location Inference and Environment Mapping) algorithms are used to generate maps based on sensor data. Figure 7 .

[0029] For the sake of brevity, the following description focuses primarily on the analysis of information related to the driver. Those skilled in the art should understand that the description is also applicable to other occupants, such as those in the front or rear seats of the vehicle.

[0030] The gaze detector 10 is configured to receive data from the driver monitoring camera 4 and generate a detected gaze result 80. The detected gaze result 80 includes the direction in which the driver's eyes are looking. In some embodiments, the direction includes azimuth and elevation. Including azimuth and elevation allows for the determination of both the direction the driver is looking parallel to the horizontal direction and the direction perpendicular to the horizontal direction. In some embodiments, the detected gaze result 80 also includes depth information. Depth information is an estimated distance from the driver at the convergence of the driver's visual axis. Including depth information allows for the determination of the distance between the driver and the object being gazed at. Combining depth information with azimuth and elevation improves the accuracy of the detected gaze result 80. In some embodiments where the captured image only includes one of the driver's eyes, it is difficult to determine the depth information, therefore only the azimuth and elevation are determined by the gaze detector 10. In some embodiments, the gaze detector 10 is also configured to receive data from the forward-facing camera 5 and correlate the detected gaze result 80 with the pixel positions of the image from the forward-facing camera 5 based on the azimuth and elevation.

[0031] The object detector 11 is configured to receive data from any additional sensors and the forward-facing camera 5 to detect the environment around the vehicle. Based on the received data, the object detector 11 identifies objects in the surrounding environment. In some embodiments, the object detector is also configured to receive data from GPS 6 and / or ground... Figure 7 Receive data to help determine the vehicle's location based on GPS6 and the location from the ground. Figure 7 The known location of an object determines the object itself. This is achieved using GPS6 and geostationary orbit. Figure 7 The information helps reduce the processing load of the object detector 11. In some embodiments, the object detector 11 is also configured to identify the type of object, such as other vehicles, pedestrians, road signs, billboards, billboards on vehicles, buildings, etc. Those skilled in the art will understand that if there is a movable advertisement, this disclosure can also be applied to determine which movable advertisement, such as one on a truck or bus, is of interest to the occupants of a vehicle. To determine the presence of a movable advertisement, in some embodiments, camera images are parsed. In some embodiments, to determine whether an occupant is viewing a movable advertisement, location information for the movable advertisement is combined with data from the vehicle. In some embodiments, a second vehicle with a billboard periodically forwards its location information to server 2, where the location information is mapped to gaze data 30. In some embodiments, the object detector 11 outputs received data from the forward-facing camera 5 and / or additional sensors to the gaze data generator 20 and / or the positioning unit 12.

[0032] The positioning unit 12 is configured to receive signals from the object detector 11, GPS 6, and ground... Figure 7 Receive information and determine the vehicle's position in the world coordinate system, or the vehicle's position relative to the ground. Figure 7 The positioning unit 12 determines the location of objects on the road and the position of objects detected by the object detector 11. In some embodiments, the positioning unit 12 can be used to determine the vehicle's travel route and speed. The positioning unit 12 is also configured to determine state information for the vehicle. In some embodiments, the state information includes the vehicle's speed. In some embodiments, the state information includes the vehicle's velocity vector. In some embodiments, the state information includes the vehicle's travel route. In some embodiments, the state information includes the vehicle's acceleration vector. In some embodiments, the state information includes the vehicle's jerk (jump) vector. In some embodiments, the state information includes whether the vehicle's engine or motor is operating. In some embodiments, the state information includes other vehicle-related state information such as the operation of the windshield wipers.

[0033] The gaze data generator 20 is configured to receive detected gaze results 80 from the gaze detector 10, object detection information from the object detector 11, and vehicle position and status information from the positioning unit 12. The gaze data generator 20 detects lines along the direction the driver's gaze is directed and generates gaze data 30 based on these detected lines. Those skilled in the art will understand that the line representing the driver's gaze means a line or semi-straight line representing the direction the driver is looking from a point in the driver's eye. The gaze data 30 includes timestamp information, vehicle information of the vehicle system 1, the vehicle's position, and the driver's angle of gaze relative to the vehicle. In some embodiments, if there are signs, the gaze data generator 20 is configured to determine which sign the driver can see. In some embodiments, the gaze data generator 20 is also configured to receive from the ground... Figure 7 Identify any known objects obstructing the driver's view of an object. Identify these obstructed objects so that they can be excluded from consideration during gaze data 30 generation. In some embodiments, the gaze data generator 20 also associates the gaze data 30 with the position of the object from the object detector 11 and / or the positioning unit 12. In some embodiments, the gaze data generator 20 also excludes any gaze data 30 related to objects determined to be obstructed based on information from the object detector 11 and the positioning unit 12. In some embodiments, an obstruction threshold, such as 50% of the area of ​​an object being unclear, is used by the gaze data generator 20 to determine whether to exclude gaze data 30 related to obstructed objects.

[0034] In some embodiments, the gaze data generator 20 is further configured to include at least one image of the driver in the gaze data 30, enabling driver demographic analysis based on the captured image. In some embodiments, the gaze data generator 20 is configured to include driver identification data such as a driver identification number along with the gaze data 30, enabling driver demographic analysis. In some embodiments, driver identification data is received based on owner information or lease information stored in the ECU 8.

[0035] The gaze data generator 20 outputs gaze data 30 to be sent to the server 2. In some embodiments, the vehicle system 1 transmits the gaze data to the server 2 wirelessly. In some embodiments, the vehicle system 1 transmits the gaze data to the server 2 via a wired connection.

[0036] Server 2 includes map 101. The server also includes an attention region parser 40 configured to receive gaze data 30 from vehicle system 1. Attention region parser 40 is also configured to receive information from map 101. Attention region parser 40 is configured to generate a raster map 50 based on gaze data 30 and map 101. Server 2 also includes a storage unit 41 configured to store raster map 50 and histogram 53. Raster map 50 can be accessed by histogram generator 52 to associate raster map 50 with additional data such as date, time, demographic data, or other appropriate information. Histogram generator 52 is configured to generate histogram 53 based on the association with raster map 50 and store histogram 53 in storage unit 41. Server 2 also includes an advisor 60 configured to receive histogram 53 from storage unit 41. Advisor 60 is configured to parse histogram 53 to determine what type of people, when, and for how long observe the sign.

[0037] Map 101 contains information relating to roads and known objects along the roads. In some embodiments, map 101 is associated with... Figure 7 The same. In some implementations, map 101 is compared to the ground. Figure 7 More refined. In some embodiments, map 101 is received from an external device. In some embodiments, map 101 is periodically sent to vehicle system 1 for updates. Figure 7 In some implementations, map 101 is updated periodically based on the updated information received from server 2.

[0038] Note that the region parser 40 is configured to correlate the received gaze data 30 with map 101 to determine which position the driver is looking towards. This correlation is defined as a set of multiple grid points, such as grid point 51. Figure 7A grid map 50 is stored. Each grid point contains azimuth and elevation angles and together contains location information related to position coordinates, such as X, Y, and Z. Each grid point also contains information related to how long the driver's gaze lingers within an area defined by the location information. In some embodiments, the location information for the grid map 50 is determined based on known objects from map 101 and / or detected objects in the gaze data 30. In some embodiments, the grid map 50 contains only grid points with gaze durations exceeding a threshold. In some embodiments, the threshold is adjusted based on the state of vehicle information from positioning unit 12. For example, in some embodiments, as vehicle speed increases, the time a sign remains within the driver's potential field of vision decreases, thus reducing the threshold gaze duration. When the grid map 50 is generated, it is stored in storage unit 41. When additional gaze data 30 is received by server 2, the grid map 50 can be updated to include new grid points or to update information associated with existing grid points. In some embodiments, the attention area resolver 40 is configured to generate a raster map 50 for a single vehicle. In some embodiments, the attention area resolver 40 is configured to generate a raster map 50 for multiple vehicles. In some embodiments, the attention area resolver 40 is configured to generate a raster map 50 for a predetermined geographic area. In some embodiments, the attention area resolver 40 is configured to generate and manage multiple raster maps 50, such as a first raster map for a first geographic area, a second raster map for a second geographic area, and a third raster map for a specific group of vehicles.

[0039] Storage unit 41 is configured to store a raster map 50 and a histogram 53. In some embodiments, storage unit 41 is also configured to store additional information such as gaze data 30 and map 101. In some embodiments, storage unit 41 includes a solid-state memory device. In some embodiments, storage unit 41 includes dynamic random access memory (DRAM). In some embodiments, storage unit 41 includes a non-volatile memory device. In some embodiments, storage unit 41 includes cloud-based storage or other suitable storage structures.

[0040] The raster map 50 contains multiple grid points. Figure 7The diagram illustrates a non-limiting example of a grid map 50. In this example, the grid map 50 contains multiple grid data points 51. In some embodiments, the grid map 50 can be used individually for display based on requests from users such as advertiser 72 or advertising agency 74. In some embodiments, the grid points within the grid map 50 are color-coded based on gaze duration. In some embodiments, the grid map 50 can be customized to include only grid points associated with a specific user such as advertiser 72 or advertising agency 74. In some embodiments, the grid map 50 can be customized to include only grid points of selected users. In some embodiments, the selected users include competitors of advertiser 72.

[0041] Histogram generator 52 is configured to parse one or more raster maps 50 stored in storage unit 41 and identify historical data to be compiled into histogram 53. In some embodiments, histogram generator 52 generates histogram 53 based on time of day, weather conditions, sign location, angle of the sign relative to the vehicle's route, demographic information, or other suitable references. In some embodiments, demographic information is based on driver identification information associated with log entries in the grid of raster map 50. In some embodiments, demographic information is extracted based on characteristics of the driver from captured images associated with log entries in the grid of raster map 50. For example, in some instances, if the captured image of the driver indicates a person with facial hair, the driver is determined to be male. Other physical characteristics can be used to determine other demographic information. In some embodiments, a learned neural network can be used to extract demographic information based on captured images. In some embodiments, histogram generator 52 is configured to generate histogram 53 for a single vehicle. In some embodiments, the histogram generator 52 is configured to generate histograms 53 for multiple vehicles. In some embodiments, the histogram generator 52 is configured to generate histograms 53 for a predetermined geographical area. In some embodiments, the histogram generator 52 is configured to generate histograms 53 such as a first histogram for a first geographical area, a second histogram for a second geographical area, and a third histogram for a specific group of vehicles. In some embodiments, the histogram generator 52 continues to update the histograms 53 as new raster map 50 data becomes available. In some embodiments, the histogram generator 52 is configured to generate new histograms 53 based on requested parameters received from the customer 3.

[0042] Advertising consultant 60 is configured to receive histogram 53 from storage unit 41 and extract information to determine which sign or area is observed most frequently and / or for the longest time. Obtaining information related to areas of interest to vehicle occupants helps identify locations for future advertising. Advertising consultant 60 is configured to provide the extracted information to customer 3. In some embodiments, advertising consultant 60 is configured to receive a data request from customer 3 and extract information from histogram 53 based on the received data request. In some embodiments, the data request includes information about the type of product or service on a particular sign, and advertising consultant 60 extracts information to identify demographics of people interested in the corresponding product or service. In some embodiments, the product or service on the sign is extracted based on data captured by forward-facing camera 5.

[0043] In some implementations, the signage includes an electronic signage with periodically changing advertisements. In some implementations, advertising consultant 60, in order to determine which advertisement attracted the driver, associates timestamp information from histogram 53 with the advertisements displayed on the electronic signage during the corresponding timestamp period.

[0044] In some implementations, the signage includes a sign mounted on a moving second vehicle. In some implementations, advertising consultant 60, in order to determine which advertisement attracted the driver, correlated timestamp information and location information from histogram 53 with advertisements displayed on the second vehicle at the known location of the second vehicle during the corresponding timestamp period.

[0045] Advertising consultant 60 outputs histogram information to be sent to customer 3. In some embodiments, server 2 transmits the histogram information to customer 3 wirelessly. In some embodiments, server 2 transmits the histogram information to customer 3 via a wired connection. In some embodiments, server 2 instructs a printer to print the histogram information and provides the printed information to customer 3.

[0046] Customer 3 includes an advertising sponsor 72 who owns the advertised product or service. Customer 3 also includes an advertising agency that helps the advertising sponsor 72 sell its product or service. In some embodiments, the advertising agency 74 includes a company that leases signage space. In some embodiments, customer 3 includes other persons, companies, or agencies. For example, in some embodiments, customer 3 includes a government agency that can use information from advertising consultant 60 to authorize additional advertising placements or determine whether an advertising placement presents a danger that would overly distract drivers. In some embodiments, customer 3 provides server 2 with data requests for information related to a specific sign, advertising sponsor 72, or other desired data. In some embodiments, the data request is sent to server 2 wirelessly. In some embodiments, the data request is sent to server 2 via a wired connection.

[0047] By outputting histogram information to customer 3, the vehicle occupant gaze detection system 100 can use the actual detected data to determine the effectiveness of advertising in attracting the attention of vehicle occupants. This effectiveness information can be used to assist the advertising sponsor 72 in allocating resources more efficiently. The effectiveness information can also be used by the advertising agency 74 to set pricing options for various locations that match the actual attention obtained from vehicle occupants. Furthermore, by updating the histogram 53 periodically or when new gaze data 30 becomes available, the advertising sponsor 72 and the advertising agency 74 can identify consumer preferences even without directly conducting consumer surveys.

[0048] Figure 2 This is a diagram of the data structure 200 of the detected gaze result 80 according to some implementation methods. The detected gaze result 80 uses a gaze detector, such as gaze detector 10. Figure 1 The detected gaze result 80 includes a two-dimensional (2D) attention point 81. The attention point 81 is the direction the driver's gaze is directed. A 2D attention point 81 means that the attention point is in a two-dimensional plane. The 2D attention point 81 is based on data 82 that includes azimuth and elevation angles. Using the data 82, the gaze detector can determine the 2D attention point 81 for the detected gaze result 80. In some embodiments, if applicable, the detected gaze result 80 can be generated by a gaze data generator, such as gaze data generator 20. Figure 1 This is used to generate gaze data to determine which sign a driver is looking at, for example, gaze data 30 ( Figure 1 ).

[0049] Figure 3 This is a diagram of the data structure 300 of the detected gaze result 80 according to some implementation methods. The detected gaze result 80 uses a gaze detector, such as gaze detector 10. Figure 1 Generated by ) and data structure 200 ( Figure 2 Unlike 2D attention points, data structure 300 contains detected gaze results 80 with three-dimensional (3D) attention points 83. A 3D attention point 83 means that the attention point is defined by three parameters. The 3D attention point 83 is based on data 84 including azimuth, elevation, and depth. The 3D attention point 83 differs from the 2D attention point 81. Figure 2 Compared to providing higher accuracy, it generates data with a greater processing load. Using data 84, the gaze detector is able to determine the 3D attention point 83 for the detected gaze result 80. In some embodiments, if present, the detected gaze result 80 can be generated by a gaze data generator, such as gaze data generator 20. Figure 1This is used to generate gaze data to determine which sign a driver is looking at, for example, gaze data 30 ( Figure 1 ).

[0050] Figure 4 This is a diagram of the data structure 400 of the detected gaze result 80 according to some implementation methods. The detected gaze result 80 uses a gaze detector, such as gaze detector 10. Figure 1 Generated by ) and data structure 200 ( Figure 2 Similarly, data structure 400 contains detected gaze results 80 with two-dimensional (2D) attention points 85. Compared to data structure 200... Figure 2 Unlike other data structures, data structure 400 includes data from a forward-facing camera, such as forward-facing camera 5. Figure 1 The 2D attention point 85 is data 86 of the vertical and horizontal pixel positions in the image. In some embodiments, if present, the detected gaze result 80 can be generated by a gaze data generator, such as gaze data generator 20. Figure 1 This is used to generate gaze data to determine which sign a driver is looking at, for example, gaze data 30 ( Figure 1 ).

[0051] Figure 5 This is a diagram of the data structure 500 of gaze data 30 according to some implementations. Gaze data 30 uses a gaze data generator, such as gaze data generator 20 ( Figure 1 The gaze data 30 is generated by, for example, the driver's monitoring camera 4. The gaze data 30 can be used to determine under what conditions the driver observes the sign, and, if any, which sign the driver observes. The gaze data 30 includes a timestamp 31, which is generated by, for example, the driver's monitoring camera 4. Figure 1 The gaze data 30 is the time when the data used to generate the gaze data 30 was collected. The gaze data 30 also includes vehicle information 32, which contains identification information about the vehicle for which the driver gaze data was collected. The gaze data 30 also includes vehicle position 34, which is the position of the vehicle at the time the data used to generate the gaze data 30 was collected. In some embodiments, the vehicle position 34 is based on data from GPS, such as GPS6 (…). Figure 1 The gaze data 30 also includes a gaze angle 35 for the vehicle. In some embodiments, the gaze angle 35 includes azimuth and elevation angles. In some embodiments, the gaze data 30 also includes depth coordinates associated with the gaze angle 35. In some embodiments, the gaze data 30 includes the driver's detected gaze direction compared with information from a forward-facing camera, such as forward-facing camera 5. Figure 1The pixel locations associated with the pixels of the acquired image are used in place of the gaze angle 35. The gaze data 30 also includes the location 36 of the billboard. In some embodiments, the location 36 of the billboard is limited to billboards determined to attract the driver's attention. In some embodiments, the location 36 of the billboard includes all known billboards. In some embodiments, the location 36 of the billboard is based on a map, such as a local map. Figure 7 ( Figure 1 ) and / or, for example, from the forward-facing camera 5 ( Figure 1 The location 36 of the billboard is determined from the captured image of the gaze data 30. In some embodiments, the location 36 of the billboard is omitted from the gaze data 30. In some embodiments where the location 36 of the billboard is omitted from the gaze data 30, the gaze data 30 can be combined with other data, such as map 101 ( Figure 1 The location of the billboard that will attract the driver's attention is determined by comparison. In some implementations, if there is no billboard in the area, the location 36 for the billboard is empty.

[0052] Figure 6 This is a flowchart of a method 600 using a vehicle occupant gaze detection system according to some embodiments. In some embodiments, a vehicle occupant gaze detection system 100 ( Figure 1 ) Implementation method 600. In some embodiments, it is used in conjunction with the vehicle occupant gaze detection system 100 ( Figure 1 Different vehicle occupant gaze detection systems are implemented using method 600. Those skilled in the art should understand that while method 600 is described with respect to a driver, it can also be applied to any occupant of the vehicle. Figure 6 Includes and Figure 1 Similar reference numerals are used in the accompanying drawings, but those skilled in the art should recognize that these reference numerals are provided merely for clarity and ease of understanding, and are not intended to limit the scope of method 600 to only [specific areas]. Figure 1 The system described in the text.

[0053] Method 600 includes using a driver monitoring camera, such as driver monitoring camera 4 ( Figure 1 The action of capturing an image of the driver is 605. In some embodiments, a single image capturing device is used to capture an image of the driver. In some embodiments, multiple image capturing devices are used to capture an image of the driver. In some embodiments, the image capturing devices are positioned at different locations relative to the driver. In some embodiments, at least one of the image capturing devices can be selectively deactivated.

[0054] Method 600 further includes an action 610 for detecting the driver's gaze based on the captured image of the driver. In some embodiments, the driver's gaze is detected based on the azimuth and elevation angles of the driver's eyes detected from the captured image of the driver. In some embodiments, the driver's gaze is also detected based on depth coordinates determined according to the visual axis of the driver's eyes from the captured image of the driver. In some embodiments, the gaze is detected based on the image from the vehicle's forward-facing camera, such as forward-facing camera 5 (…). Figure 1 The system detects the driver's gaze using information from the camera and associates the gaze with the pixel location of the image captured by the camera. In some implementations, action 610 is performed based on the data collected in action 615. In some implementations, a gaze detector, such as gaze detector 10, is used. Figure 1 ) Detects the driver's gaze. In some implementations, the results of gaze detection have a data structure 200 ( Figure 2 In some implementations, the results of gaze detection have a data structure 300 (). Figure 3 In some implementations, the results of gaze detection have a data structure 400 (). Figure 4 ).

[0055] Following action 610, method 600 branches out. Along the first branch, method 600 returns to action 605, capturing an additional image of the driver. Along the second branch, method 600 proceeds to action 630.

[0056] Method 600 also includes action 615 of capturing images of the environment surrounding the vehicle. The images of the environment surrounding the vehicle can be captured using one or more sensors mounted on the vehicle. In some embodiments, the sensors include visible light cameras, infrared cameras, lidar sensors, radar sensors, sonar sensors, or other suitable sensors. In some embodiments, the sensors include a forward-facing camera 5 (…). Figure 1 In some implementations, the image captured by action 615 is used in action 610 to associate the driver's gaze with the image captured by action 615.

[0057] Method 600 also includes action 620 of detecting an object based on data collected in action 615. In some embodiments, the object is detected by parsing the data collected by action 615 using a learned neural network. In some embodiments, an object detector, such as object detector 11, is used. Figure 1Detecting objects. In some embodiments, the neural network of the object detector 11 uses a neural network architecture such as a single-click detector (SSD) or Faster R-CNN (a convolutional neural network based on fast regions for object detection). In some embodiments, in addition to detecting the position of the object, action 620 also includes detecting the angle of the object relative to the vehicle.

[0058] Following action 620, method 600 branches. Along branch 1, method 600 returns to action 615, capturing additional images of the environment surrounding the vehicle. Along branch 2, method 600 proceeds to action 630. Along branch 3, method 600 proceeds to action 625.

[0059] Method 600 also includes action 625, which determines the position of the vehicle in the world coordinate system, or the position of the vehicle relative to an object detected in action 620. In some embodiments, action 625 further determines state information for the vehicle. In some embodiments, the state information includes the vehicle's velocity. In some embodiments, the state information includes the vehicle's velocity vector. In some embodiments, the state information includes the vehicle's travel path. In some embodiments, the state information includes the vehicle's acceleration vector. In some embodiments, the state information includes the vehicle's jerk (jump) vector. In some embodiments, the state information includes whether the vehicle's engine or motor is operating. In some embodiments, the state information includes other vehicle-related state information such as changes in velocity (acceleration), windshield wiper operation, etc. In some embodiments, a positioning unit, such as positioning unit 12, is used. Figure 1 Action 625 is performed.

[0060] The method also includes action 630, which generates gaze data based on data obtained through actions 610, 620, and 625. A gaze data generator, such as gaze data generator 20, is used. Figure 1 ) Generate gaze data. In some implementations, gaze data 30 is generated via action 630. Figure 1In some embodiments, gaze data includes timestamp information, vehicle information from the vehicle system, the vehicle's location, and the driver's gaze angle toward the vehicle. In some embodiments, gaze data is also associated with the location of an object in either action 620 or action 625. In some embodiments, action 630, if applicable, determines which sign the driver can see. In some embodiments, action 630 identifies objects obstructing the driver's view of an obstructed object. In some embodiments, any gaze data relating to an object determined to be obstructed based on information in action 620 and / or action 625 is also excluded. In some embodiments, gaze data includes information about a second vehicle displaying the billboard (e.g., location, driver's license ID, etc.).

[0061] In some embodiments, the gaze data further includes at least one image of the driver that enables the analysis of the driver's demographics based on the captured images. In some embodiments, the gaze data includes driver identification data such as a driver identification number that enables the analysis of the driver's demographics. In some embodiments, the driver identification data is received from internal memory or based on owner or rental information from an external device.

[0062] Method 600 also includes sending the gaze data generated in action 630 to a server, such as server 2. Figure 1 Action 635. In some embodiments, gaze data is transmitted wirelessly. In some embodiments, gaze data is transmitted via a wired connection. Method 600 also includes a server, such as server 2 ( Figure 1 Action 640: Receiving gaze data.

[0063] Method 600 also includes an action 645 of updating a raster map. In the absence of a raster map, action 645 generates a raster map. The raster map contains grid points that associate locations with received gaze data. In some embodiments, the raster map includes raster map 50 (…). Figure 1 In some implementations, attention region resolvers are used, such as attention region resolver 40. Figure 1 Update the raster map. In some implementations, this is based on data from a map, such as map 101 (…). Figure 1 Known objects and / or detected objects identified in the gaze data determine the location information for the grid map. In some implementations, only grid points with gaze durations exceeding a threshold are updated in the grid map. In some implementations, the threshold is adjusted based on the state of vehicle information from the positioning unit 12. When the grid is generated, the grid map is stored in a storage unit, such as storage unit 41. Figure 1In some implementations, multiple raster maps are generated or updated based on the received gaze data.

[0064] Method 600 further includes action 650 of generating a histogram based on the grid map updated in action 645. If a histogram already exists, it is updated in action 650. In some implementations, the histogram is histogram 53 (…). Figure 1 In some implementations, a histogram generator, such as histogram generator 52, is used. Figure 1 Generate a histogram. The histogram correlates gaze data from the raster map with additional parameters such as time of day, weather conditions, sign location, the angle of the sign relative to the vehicle's path, demographic information, or other appropriate benchmarks.

[0065] Method 600 also includes action 655, which includes updating the proposed location for the advertisement. In the absence of any prior proposals, action 655 includes proposing a location for the advertisement. In some embodiments, the proposed location is determined based on received gaze data, specifically based on which area is viewed most frequently. In some embodiments, the proposed location is determined based on demographic information of the target customer and the gaze data of drivers consistent with the target customer. In some embodiments, in action 655, the type of a second vehicle (car model, color, size, etc.) for displaying the advertisement is proposed, along with the route of the second vehicle, to help maximize the visibility of the displayed advertisement to potential customers. In some embodiments, the updated proposed location is sent to the advertising sponsor, such as advertising sponsor 72, and / or an advertising agency, such as advertising agency 74. In some embodiments, the updated proposed location is transmitted wirelessly. In some embodiments, the updated proposed location is transmitted via a wired connection.

[0066] Method 600 also includes the action 660 of updating the proposed rate for the advertisement in a specific location. In the absence of any prior proposal, action 660 includes publishing the recommended rate for the advertisement. In some embodiments, the proposed rate is determined based on received gaze data, specifically based on which location is most frequently viewed. In some embodiments, the proposed rate is determined based on demographic information of the target customer and the gaze data of drivers consistent with the target customer. In some embodiments, the updated proposed rate is sent to the advertising sponsor, such as advertising sponsor 72, and / or advertising agency, such as advertising agency 74. In some embodiments, the updated proposed rate is sent wirelessly. In some embodiments, the updated proposed rate is sent via a wired connection.

[0067] In some implementations, method 600 includes additional actions. For example, in some implementations, method 600 includes generating driver demographic information based on captured images of the driver. In some implementations, the order of actions in method 600 may be changed. For example, in some implementations, the raster map is generated before being sent to the server. In some implementations, at least one action in method 600 may be omitted. For example, in some implementations, action 655 is omitted, and only a suggestion regarding rates is provided.

[0068] Figure 7 This is a diagram of the data structure 700 of a grid map 50 according to some implementations. The grid map 50 is based on gaze data received by the driver, such as gaze data 30 (…). Figure 1 ) and map data, such as map 101 ( Figure 1 The raster map 50 is generated or updated by the detected objects representing the location of the sign, and / or by the location of the sign. In some implementations, the raster map 50 is generated by the attention region parser 40. Figure 1 ) generated. In some implementations, the grid map 50 is generated in action 645 ( Figure 6 During the process, the grid map 50 is generated by the gaze data generator 20 in the vehicle system 1 and sent from the vehicle system 1 to the server 2 via a network. The grid map 50 contains an array of grid points 51. In some embodiments, the data structure 700 contains a two-dimensional array of grid points 51 with dimensions corresponding to the x-axis and y-axis parallel to the ground. In some embodiments, the data structure 700 contains a three-dimensional array of grid points 51 with dimensions corresponding to the x-axis, y-axis, and z-axis orthogonal to the ground. In some embodiments, the data structure 700 contains a four-dimensional array of grid points 51 with dimensions corresponding to the x-axis, y-axis, azimuth, and elevation. In some embodiments, the data structure 700 contains a five-dimensional array of grid points 51 with dimensions corresponding to the x-axis, y-axis, z-axis, azimuth, and elevation. In some implementations, data structure 700 includes a four-dimensional array of grid points 51, with latitude and longitude in the World Geodetic System (WGS84) and dimensions corresponding to latitude, longitude, azimuth, and elevation. Each grid point 51 contains azimuth and elevation relative to the vehicle and together they contain geographic location information. Figure 8 This includes non-limiting examples of data structures for grid 51 involved in some implementations.

[0069] Figure 8 This is a diagram of the data structure 800 for grid data 51 according to some implementations. In some implementations, the grid data 51 can be in a raster map 50 ( Figure 1 and 7The grid data 51 contains metadata 54. Metadata 54 contains information about the location range of the grid points, such as geographic coordinates, azimuth, and elevation. Metadata 54 contains "absolute" position and angle values. "Absolute" values ​​mean that the values ​​have been corrected for the vehicle's position and route of travel to determine the values ​​relative to a fixed point. In some implementations, metadata 54 contains position and angle values ​​relative to the Earth's latitude or longitude. Metadata 54 can be used to determine the real-world location corresponding to the grid data 51. Grid data 51 also contains a list 57 of objects known or detected to exist at the location of grid data 51. In some implementations, list 57 contains information based on a map, such as map 101 (…). Figure 1 The received information includes the objects. In some embodiments, list 57 contains objects based on object detection performed on images captured by the vehicle. In some embodiments, information related to object detection is used as gaze data, such as gaze data 30. Figure 1 ( ) is received by a part of the object detector, such as object detector 11 in the vehicle. Figure 1 ) generated. Grid data 51 also includes log 55. Log 55 contains data based on received gaze data, such as gaze data 30 ( Figure 1 The metadata 54 indicates when a location is seen. Log 55 contains more than one log entry 56. Each log entry 56 is associated with other detections of locations seen in relation to grid data 51. In some implementations, all log entries 56 are maintained within grid data 51. In some implementations, log entries 56 with timestamps exceeding a threshold are cleared or rewritten. In some implementations, data structure 700 contains a four-dimensional array of grid points 51 with dimensions corresponding to latitude, longitude, azimuth, and elevation. When a point corresponding to latitude and longitude is viewed in a gaze direction corresponding to azimuth and elevation, a log entry is appended to the grid points corresponding to latitude, longitude, azimuth, and elevation in the four-dimensional array. In some implementations, log entries 56 are generated only if the location associated with grid data 51 satisfies a threshold duration. In some implementations, the threshold duration is determined based on factors such as the vehicle speed at the time the data used to generate gaze data was collected. In some implementations, log entries 56 have data structure 900 (… Figure 9 ).

[0070] Figure 9 This is a diagram of data structure 900 for log entry 56 according to some implementations. In some implementations, data structure 900 can be used as a counterpart to data structure 800 ( Figure 8 Log entry 56 is used. Log entry 56 contains entries seen with grid points, such as grid point 51 ( Figure 7 The time 57 is associated with the location. In some implementations, time 57 is used as gaze data, such as gaze data 30. Figure 1 As part of the gaze data, the location 58 is received. Log entry 56 also contains the vehicle's location 58 as seen by the driver in relation to the grid points. In some implementations, location 58 is received as part of the gaze data, using GPS, such as GPS6 ( Figure 1 The log entry 56 also contains identification information 59 regarding the vehicle the driver sees in relation to the grid point. In some embodiments, the identification information 59 is received as part of gaze data and retrieved from memory in the vehicle. In some embodiments, the identification information 59 can be used to generate driver demographic information. In some embodiments, the log entry 56 also contains driver demographic information. In some embodiments, demographic information is generated based on the identification information 59, image parsing of the driver, or other suitable processes. In some embodiments, the demographic information includes gender, age, height, weight, or other suitable information.

[0071] Figure 10 It is a graph based on the data structure 1000 of histogram 53 according to some implementations. Based on a raster map, such as raster map 50 ( Figure 1 and Figure 7 Histogram 53 is generated by parsing the histogram generator, such as histogram generator 52. Figure 1 The histogram 53 contains information related to the count of how many times each grid point of the raster map has been observed. In some embodiments, the histogram 53 contains, for example, three-dimensional data including view count, x-axis coordinates, and y-axis coordinates. In some embodiments, the histogram 53 contains, for example, four-dimensional data including view count, x-axis coordinates, y-axis coordinates, and z-axis coordinates. In some embodiments, the histogram 53 contains, for example, five-dimensional data including view count, x-axis coordinates, y-axis coordinates, azimuth, and elevation. In some embodiments, the data in the histogram is analyzed based on external parameters to determine whether the view count is affected by external events and phenomena such as time of day, weather conditions, season of year, vehicle speed, location relative to the vehicle's location, driver demographics, or other appropriate parameters. In some embodiments, the parameters are predetermined. That is, the system analyzes the data based on commonly used, well-known parameters. In some embodiments, the parameters are set based on data requests received from advertisers or advertising sponsors.

[0072] In some implementations, a histogram generator, such as histogram generator 52, Figure 1Based on these parameters, a histogram 53 containing bins related to the view count is generated. In some embodiments, the histogram generator can configure the view count in the parameter bins based on timestamp information in the metadata. In some embodiments, the timestamp information is associated with external sources such as weather databases or calendar databases to determine the environmental conditions associated with such timestamp information, such as sunny day or daytime. In some embodiments, the histogram generator uses vehicle registration information in the metadata, used to determine the driver's home address, to determine the distance between the observed advertisement and the driver's home. Those skilled in the art will understand that additional parameters for parsing the view count can be considered based on information available in the metadata.

[0073] Figure 11 This is a diagram of a system 1100 for implementing a vehicle occupant gaze detection system according to one or more embodiments. System 1100 includes a hardware processor 1102 and a non-transitory computer-readable storage medium 1104, which is encoded by computer program code 1106, i.e., a set of executable commands, i.e., stores the computer program code 1106. The computer-readable storage medium 1104 is also encoded by commands 1107 for obtaining an interface with an external device. The processor 1102 is electrically coupled to the computer-readable storage medium 1104 via a bus 1108. The processor 1102 is also electrically coupled to an I / O interface 1110 via the bus 1108. A network interface 1112 is also electrically connected to the processor 1102 via the bus 1108. The network interface 1112 is connected to a network 1114 so that the processor 1102 and the computer-readable storage medium 1104 can connect to external elements via the network 1114. Processor 1102 is configured to execute computer program code 1106 encoded in computer-readable storage medium 1104 to enable system 1100 to perform vehicle occupant gaze detection system 100. Figure 1 Or in method 600 ( Figure 6 (The actions described in the text) are part or all of the actions described in the text.

[0074] In some implementations, processor 1102 is a central processing unit (CPU), a multiprocessor, a distributed processing system, an application-specific integrated circuit (ASIC), and / or a suitable processing unit.

[0075] In some embodiments, the computer-readable storage medium 1104 is an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system (or device or disk). For example, the computer-readable storage medium 1104 includes semiconductor memory or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, and / or optical disk. In some embodiments using optical disk, the computer-readable storage medium 1104 includes optical disc read-only memory (CD-ROM), rewritable optical disc (CD-R / W), and / or digital video disc (DVD).

[0076] In some embodiments, storage medium 1104 stores information configured to cause system 1100 to perform vehicle occupant gaze detection system 100. Figure 1 ) or method 600 ( Figure 6 Computer program code 1106 for all or part of the actions of the vehicle occupant gaze detection system 100. In some embodiments, storage medium 1104 also stores parameters such as gaze detection parameters 1116, object detection parameters 1118, map parameters 1120, raster map parameters 1122, histogram parameters 1124, and / or data for performing the vehicle occupant gaze detection system 100. Figure 1 ) or method 600 ( Figure 6 A set of executable commands, such as those for the vehicle occupant gaze detection system 100, which perform all or part of the actions of the system. Figure 1 ) or method 600 ( Figure 6 Information generated during all or part of the period of the vehicle occupant gaze detection system 100, and also stored for the purpose of performing the vehicle occupant gaze detection system 100. Figure 1 ) or method 600 ( Figure 6 (All or part of the information required.)

[0077] In some embodiments, storage medium 1104 stores commands 1107 for acquiring an interface with the manufacturing equipment. Command 1107 enables processor 1102 to generate commands that can be read by external devices to effectively implement the vehicle occupant gaze detection system 100. Figure 1 ) or method 600 ( Figure 6 All or part of ).

[0078] System 1100 includes an I / O interface 1110. The I / O interface 1110 is coupled to external circuitry. In some embodiments, the I / O interface 1110 includes a keyboard, keypad, mouse, trackball, touchpad, and / or cursor arrow keys for communicatively transmitting information and instructions to processor 1102.

[0079] System 1100 also includes a network interface 1112 integrated with processor 1102. Network interface 1112 enables system 1100 to communicate with a network 1114 connected to one or more other computer systems. Network interface 1112 includes wireless network interfaces such as Bluetooth, Wi-Fi, WiMAX, GPRS, or WCDMA, or wired network interfaces such as Ethernet, USB, or IEEE-1394. In some embodiments, vehicle occupant gaze detection system 100 ( Figure 1 ) or method 600 ( Figure 6 All or part of the information is implemented in two or more systems 1100, and information such as gaze data, map data, object detection data or captured images is exchanged between different systems 1100 via network 1114.

[0080] One technical solution described herein relates to a vehicle occupant gaze detection system. The vehicle occupant gaze detection system includes a non-transitory computer-readable medium configured to store commands. The system also includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute commands to receive gaze data from a vehicle, the gaze data including information related to the direction of observation by the vehicle occupant, vehicle information, and timestamp information. The processor is further configured to execute commands to generate a raster map based on the received gaze data and map data, the raster map comprising a grid array, each grid point corresponding to one of a plurality of locations. The processor is further configured to execute commands to generate a histogram based on the raster map, the histogram containing information related to the fact that at least one of the plurality of locations was seen by the vehicle occupant based on timestamp information. The processor is further configured to execute commands to determine the validity of an object located at at least one location based on the histogram. The processor is further configured to execute commands to send a recommendation based on the object's validity to a customer. In some embodiments, the processor is further configured to execute commands to generate demographic information for the vehicle occupant based on captured images of the occupant. In some embodiments, the processor is further configured to execute commands to generate a histogram based on demographic information. In some embodiments, the processor is further configured to execute commands to generate a pricing recommendation based on the histogram and send the pricing recommendation to the customer. In some embodiments, the processor is further configured to execute commands to receive a data request from the customer and generate a histogram based on the received data request. In some embodiments, the processor is further configured to execute commands to update the raster map in response to receiving second gaze data, and to update the histogram in response to the updated raster map. In some embodiments, the occupant of the vehicle is the driver of the vehicle.

[0081] One technical solution described herein relates to a method. The method includes receiving gaze data from a vehicle, the gaze data including information related to the direction observed by a vehicle occupant, vehicle information, and timestamp information. The method further includes generating a raster map based on map data and the received gaze data, the raster map comprising a grid array, each grid point corresponding to one of a plurality of locations. The method further includes generating a histogram based on the raster map, the histogram containing information related to the fact that at least one of the plurality of locations was seen by a vehicle occupant based on timestamp information. The method further includes determining the validity of an object located at at least one location based on the histogram. The method further includes sending a recommendation based on the object's validity to a customer. In some embodiments, the method further includes generating demographic information for the vehicle occupant based on captured images of the occupant. In some embodiments, generating the histogram includes generating the histogram based on the demographic information. In some embodiments, the method further includes generating a pricing recommendation based on the histogram and sending the pricing recommendation to a customer. In some embodiments, the method further includes receiving a data request from a customer and generating a histogram based on the received data request. In some embodiments, the method further includes updating the raster map in response to receiving second gaze data, and updating the histogram in response to the updated raster map. In some embodiments, the occupant of the vehicle is the driver of the vehicle.

[0082] One technical solution described herein relates to a vehicle occupant gaze detection system. The vehicle occupant gaze detection system includes sensors configured to capture data related to the environment surrounding the vehicle. The system also includes a camera configured to capture images of the vehicle occupants. Furthermore, the system includes a non-transitory computer-readable medium configured to store commands. The system further includes a processor connected to the non-transitory computer-readable medium. The processor is configured to execute commands to generate gaze detection results based on the captured images of the occupants, the gaze detection results indicating the direction of the occupants' gaze. The processor is also configured to execute commands to detect objects based on the data captured from the sensors. The processor is further configured to execute commands to generate vehicle state information for the vehicle based on the detected objects, the vehicle's position, and map data. The processor is also configured to execute commands to generate gaze data based on the gaze detection results, the detected objects, and the vehicle state information, the gaze data including timestamp information associated with the gaze detection results. Finally, the processor is configured to execute commands to send the gaze data to an external device. In some embodiments, the processor is further configured to execute commands to generate gaze detection results based on azimuth and elevation angles detected by parsing captured images of the occupant. In some embodiments, the processor is further configured to execute commands to generate gaze detection results including depth information based on the visual axes of the occupant's eyes in the captured images. In some embodiments, the processor is further configured to execute commands to generate gaze data based on the correlation between the gaze detection results and pixel positions in surrounding images captured by sensors. In some embodiments, the vehicle occupant gaze detection system further includes a Global Positioning System (GPS) and a map stored on a non-transitory computer-readable medium, and the processor is further configured to execute commands to generate vehicle status information based on information from GPS and map data from the map stored on the non-transitory computer-readable medium. In some embodiments, the occupant is the driver of the vehicle.

[0083] The foregoing has described some key features of the embodiments to enable those skilled in the art to better understand the technical solutions of this disclosure. Those skilled in the art should recognize that this disclosure can be readily used as a basis for designing or modifying other processes and structures for the embodiments described herein, in order to perform the same purpose and / or achieve the same advantages. Those skilled in the art should also recognize that such equivalent configurations do not depart from the spirit and scope of this disclosure, and various changes, substitutions, and modifications can be made herein without departing from the spirit and scope of this disclosure.

Claims

1. A vehicle occupant gaze detection system, comprising: It constitutes a non-transitory computer-readable medium for storing commands; and A processor connected to the non-transitory computer-readable medium, The processor is configured to execute the command to perform: Receive gaze data from the vehicle, which includes information related to the direction of observation of the vehicle's occupants, information related to the depth of vision of the occupants, vehicle information, and timestamp information; A raster map is generated based on map data and the received gaze data, the raster map having a grid array in which each grid point corresponds to one of a plurality of places; A histogram is generated based on the grid map, the histogram containing information related to the fact that at least one of the plurality of locations was seen by the occupants of the vehicle based on the timestamp information; Based on the histogram, the validity of the object located in the at least one location is determined; Recommendations based on the validity of the object will be sent to the customer.

2. The vehicle occupant gaze detection system according to claim 1, The processor is also configured to execute the command to generate demographic information of the occupants of the vehicle based on captured images of the occupants.

3. The vehicle occupant gaze detection system according to claim 2, The processor is further configured to execute the command to generate the histogram based on the demographic information.

4. The vehicle occupant gaze detection system according to any one of claims 1 to 3, The processor is further configured to execute the command to generate a pricing recommendation based on the histogram and send the pricing recommendation to the customer.

5. The vehicle occupant gaze detection system according to any one of claims 1 to 3, The processor is further configured to execute the command to receive a data request from the customer and generate the histogram based on the received data request.

6. The vehicle occupant gaze detection system according to any one of claims 1 to 3, The processor is further configured to execute the command to update the raster map in response to receiving the second gaze data, and to update the histogram in response to the updated raster map.

7. The vehicle occupant gaze detection system according to any one of claims 1 to 3, The occupant of the vehicle is the driver of the vehicle.

8. A method of use, comprising: Receive gaze data from the vehicle, which includes information related to the direction of observation of the vehicle's occupants, information related to the depth of vision of the occupants, vehicle information, and timestamp information; A raster map is generated based on the received gaze data and map data. The raster map has a grid array in which each grid point corresponds to one of a plurality of places. A histogram is generated based on the grid map, the histogram containing information related to the fact that at least one of the plurality of locations was seen by the occupants of the vehicle based on the timestamp information; Based on the histogram, the validity of the object located in the at least one location is determined; and Recommendations based on the validity of the object will be sent to the customer.

9. The method of use according to claim 8 further includes: Based on the captured images of the occupants, demographic information about the occupants of the vehicle is generated.

10. The method of use according to claim 9, Generating the histogram includes generating the histogram based on the demographic information.

11. The method of use according to any one of claims 8 to 10, further comprising: Based on the histogram, a pricing suggestion is generated; and The pricing proposal will be sent to the customer.

12. The method of use according to any one of claims 8 to 10, further comprising: Receive data requests from the customer; and The histogram is generated based on the received data request.

13. The method of use according to any one of claims 8 to 10, further comprising: The raster map is updated in response to the receipt of the second gaze data; and The histogram is updated in response to an update to the grid map.

14. The method of use according to any one of claims 8 to 10, The occupant of the vehicle is the driver of the vehicle.

15. A vehicle occupant gaze detection system, comprising: It is configured to capture data related to the vehicle's surrounding environment; A camera configured to capture images of the occupants of the vehicle; It constitutes a non-transitory computer-readable medium for storing commands; and A processor connected to the non-transitory computer-readable medium, The processor is configured to execute the command to perform: Based on the captured images of the occupant, a gaze detection result is generated that represents the direction observed by the occupant and includes the occupant's visual depth; Based on the data captured from the sensor, the object is detected; Based on the detected location of the object and the vehicle, and map data, vehicle status information about the vehicle is generated; Based on the gaze detection results, the detected object and the vehicle status information, gaze data containing timestamp information associated with the gaze detection results is generated; and The gaze data is sent to an external device.

16. The vehicle occupant gaze detection system according to claim 15, The processor is further configured to execute the command to generate the gaze detection result based on the azimuth and elevation angles detected by parsing the captured image of the occupant.

17. The vehicle occupant gaze detection system according to claim 16, The processor is further configured to execute the command to generate the gaze detection result containing depth information based on the visual axis of the occupant's eyes in the captured image.

18. The vehicle occupant gaze detection system according to any one of claims 15 to 17, The processor is further configured to execute the command to generate the gaze data based on the correlation between the gaze detection result and the position of pixels in the surrounding image captured by the sensor.

19. The vehicle occupant gaze detection system according to any one of claims 15 to 17, further comprising: GPS, or Global Positioning System; and Maps stored on the non-transitory computer-readable medium, The processor is further configured to execute the command to generate the vehicle status information based on information from the GPS and map data from the map stored in the non-transitory computer-readable medium.

20. The vehicle occupant gaze detection system according to any one of claims 15 to 17, The occupant is the driver of the vehicle.

Images