Brightness Control of In-Vehicle Infotainment Systems Using Gaze Estimation

Abstract

The present disclosure relates to an infotainment brightness control system 100 for a vehicle. The system 100 includes a processing device 102 configured with a pre-trained gaze detection module and a learning module. The processing device 100 is configured to determine yaw and pitch indicative of a user's gaze direction over real-time frames and previous time frames based on images of the user's eye region. The processing device 102 further estimates the user's subsequent gaze direction based on the determined gaze direction over the real-time frames and previous time frames. The processing device 102 adjusts the brightness of the display 104-1 of the infotainment system 104 to a predetermined level based on an overlap between the display 104-1 of the infotainment system 104 and the estimated subsequent gaze direction.

Description

[Technical Field]

[0001] The present disclosure relates to the field of infotainment units for vehicles. In particular, the present disclosure provides a system and method for controlling the brightness of an infotainment screen in a vehicle using gaze estimation. [Background technology]

[0002] Vehicles are equipped with infotainment units that provide a combination of entertainment and information to vehicle passengers. Typically, the infotainment screen or display automatically turns on when the vehicle's ignition key is turned on and remains on for the entire journey until the ignition key is turned off again. The infotainment system generally draws power from the vehicle's battery, and the vehicle itself draws power from the vehicle's power pack or from an external power source while the battery is charging.

[0003] As global electricity consumption increases, resulting in negative consequences for the environment and society, the world is focusing on having sustainable resources and conserving energy. Smartly controlling the brightness and lighting duration of infotainment devices can also help reduce energy consumption to some extent. For example, while driving, a vehicle driver may look at the screen to check directions and then look elsewhere for the rest of the time; however, the infotainment screen remains lit at the same brightness level even when the driver is looking elsewhere. Therefore, it would be advantageous from the perspective of energy conservation and sustainability to provide a simple, automated, and efficient solution for activating the infotainment screen only when the driver is looking at the screen.

[0004] Furthermore, during night driving, the light from the infotainment screen causes eye strain and distracts the driver. Therefore, providing a simple, automated, and efficient solution for controlling the brightness of the infotainment screen based on the driver's line of sight would be advantageous from the perspective of driver safety and comfort. However, it is difficult to accurately identify the gaze direction in which the vehicle driver is looking. Conventional systems predict the current gaze vector, but this gaze vector may not be stable at a frame-by-frame level because it is usually prone to a lot of flicker due to fine-grained eye movements.

[0005] Korean Patent Application Publication No. 101469978 discloses a brightness adjustment device for a vehicle display device. The device includes a gaze detection unit for detecting a driver's gaze based on the direction of the driver's face and eyes, and a pupil diameter detection unit for detecting the driver's pupil diameter. The brightness of the display device is changed based on the pupil diameter of the driver's eyes.

[0006] The methodologies in the above cited documents that are based on face orientation and eye orientation do not provide a very accurate estimate of the user's current gaze direction.

[0007] Therefore, there is a need in the art to overcome the above-mentioned drawbacks, limitations and shortcomings by helping to accurately identify the direction an occupant is looking and likely looking in while driving a vehicle and control the brightness of the vehicle's infotainment screen in order to provide a comfortable driving experience and save energy. Summary of the Invention [Problem to be solved by the invention]

[0008] [Purpose of this disclosure] A general objective of the present disclosure is to control the brightness of an infotainment system in a vehicle based on the driver's gaze direction.

[0009] One object of the present disclosure is to provide a system and method for controlling brightness of an infotainment system in a vehicle using gaze estimation.

[0010] Another object of the present disclosure is to accurately identify the direction a driver is looking and will look in while operating a vehicle to assist in controlling the brightness of the vehicle's infotainment screen.

[0011] It is yet another object of the present disclosure to provide a system and method for controlling brightness of an infotainment system that helps reduce driver eye strain and distraction at night.

[0012] It is yet another object of the present disclosure to provide a system and method for providing hands-free brightness control of an infotainment system in a vehicle.

[0013] Yet another object of the present disclosure is to efficiently control the brightness of infotainment system screens in vehicles to save energy and provide a comfortable driving experience. [Means for solving the problem]

[0014] Aspects of the present disclosure relate to the field of infotainment units for vehicles. In particular, the present disclosure provides a system and method for controlling brightness of infotainment systems in vehicles using gaze estimation.

[0015] One aspect of the present disclosure relates to an infotainment brightness control system for a vehicle. The system includes a processing device configured with a pre-trained gaze detection module and a learning module. The processing device includes a processor coupled to a memory, the memory storing one or more instructions executable by the processor to: use the pre-trained gaze detection module to determine yaw and pitch indicative of a user's gaze direction over a real-time frame and one or more previous time frames based on one or more images of one or more eye regions of the user; use the learning module to estimate a subsequent gaze direction of the user based on the determined gaze direction over the real-time frame and one or more previous time frames; and send a set of control signals to the infotainment system to adjust brightness of a display of the infotainment system to a predetermined level based on overlap of one or more regions of interest (ROI) associated with an interior of the vehicle with the estimated subsequent gaze direction.

[0016] The processing unit may be configured to project the estimated subsequent gaze direction into a 3D vehicle coordinate system and align the estimated subsequent gaze direction with one or more ROIs in the vehicle interior.

[0017] One or more ROIs may be associated with a display of a vehicle or infotainment system.

[0018] The processing device may be configured to derive a confidence region representing an ellipse within the vehicle interior, where the yaw and pitch of the estimated subsequent gaze direction correspond to the center point of the ellipse, where the variances of the yaw and pitch of the corresponding gaze direction (yaw variance and pitch variance) determined over the real-time frame and one or more previous time frames correspond to the radius of the ellipse, calculate an overlap of the ellipse with one or more ROIs, and adjust the brightness of the display accordingly, where the processing device triggers one or more ROIs of the infotainment system if the derived confidence region is within the one or more ROIs.

[0019] The pre-trained gaze detection module may include a first branch comprising a first neural network architecture for regressing corresponding eye images to a gaze map and a second neural network architecture for regressing the gaze map to a gaze direction represented by yaw and pitch, and a second branch configured as a classification network architecture fed by the gaze map to reduce entropy loss to enable the model to determine an actual gaze map based on training with synthetic data, wherein the gaze detection module has been pre-trained using synthetic data and / or real-time data including cropped images of the eye region.

[0020] The processing device may be configured to detect eye closure in at least one eye region from one or more captured images, and to correspondingly select and detect a corresponding gaze direction from images of unclosed eyes.

[0021] The processing unit may be configured to maintain the luminance level of the display at a predetermined level for a predetermined time after the estimated subsequent gaze direction leaves the ROI.

[0022] Another aspect of the present disclosure relates to a method for controlling brightness of an infotainment system of a vehicle, the method including: determining, by a processing device configured with a pre-trained gaze detection module, yaw and pitch indicative of a user's gaze direction over a real-time frame and one or more previous time frames based on one or more images of one or more eye regions of the user; estimating, using a learning module of the processing device, a subsequent gaze direction of the user based on the corresponding gaze directions determined over the real-time frame and the one or more previous time frames; and adjusting brightness of a display of the infotainment system to a predetermined level based on an overlap of one or more regions of interest (ROIs) associated with an interior of the vehicle with the estimated subsequent gaze direction.

[0023] The method may further include the steps of capturing one or more facial images of the user by an image acquisition unit equipped with a camera, cropping one or more eye regions from the captured one or more facial images by a processing device, and determining, by the processing device, yaw and pitch indicating the user's gaze direction based on the cropped images of the eye regions.

[0024] Various objects, features, aspects and advantages of the present subject matter will become more apparent from the following detailed description of preferred embodiments, taken in conjunction with the accompanying drawings in which like numerals represent like elements.

[0025] The accompanying drawings are included to provide a further understanding of the present disclosure, and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is an exemplary block diagram of a proposed system for controlling the brightness of a display of an infotainment system in a vehicle, according to an embodiment of the present invention. [Figure 2] 1 is an exemplary block diagram illustrating functional units of a processing device associated with the proposed system according to an embodiment of the present invention; [Figure 3] FIG. 1 is an exemplary diagram of a vehicle interior with a camera and infotainment system to detail the operation of the present invention, according to an embodiment of the present disclosure. [Figure 4A] FIG. 1 is a flow diagram illustrating steps of a proposed method for controlling the brightness of a display in a vehicle infotainment system, according to one embodiment of the present disclosure. [Figure 4B] 1 is an exemplary flowchart illustrating the operation of the proposed system and method for controlling the brightness of a display in a vehicle infotainment system, according to an embodiment of the present invention. [Figure 5A]FIG. 1 illustrates an exemplary architecture of a gaze detection module of the proposed system, according to an embodiment of the present disclosure. [Figure 5B] FIG. 1 illustrates an exemplary architecture of the gaze detection module of the proposed system using an LSTM model, according to an embodiment of the present disclosure. [Figure 6] FIG. 10 illustrates an exemplary representation of the final gaze estimation as a 2D ellipse predicted by the proposed system, according to an embodiment of the present disclosure. [Figure 7] 1 is an exemplary diagram illustrating the intersection of a 2D ellipse corresponding to a predicted gaze direction with an ROI of an infotainment system to detail the operation of the present invention, according to an embodiment of the present disclosure. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0027] The following is a detailed description of embodiments of the present disclosure, as illustrated in the accompanying drawings. The embodiments are in such detail as to clearly communicate the disclosure. However, the gist of the details provided is not intended to limit the possible variations of the embodiments, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the appended claims.

[0028] DETAILED DESCRIPTION OF THE INVENTION The embodiments described herein relate to systems and methods for controlling brightness of infotainment systems in vehicles using gaze estimation.

[0029] Referring to FIG. 1 , a proposed system 100 (also referred to herein as system 100) for controlling the brightness of a display of an infotainment system in a vehicle is disclosed. System 100 helps control the brightness of the infotainment screen of a vehicle by accurately identifying the direction in which a vehicle occupant / user / driver is currently looking while driving the vehicle and the direction in which the vehicle occupant / user / driver will look in a subsequent time frame, in order to provide a comfortable driving experience and save energy.

[0030] In one embodiment, system 100 may include a processing device 102 in communication with a vehicle's infotainment system 104 (also referred to herein as a head unit). In another embodiment, processing device 102 may be a server that may maintain communication with infotainment systems 104 associated with one or more vehicles. System 100 may further include an image capture unit 106 that includes one or more image sensors or cameras (collectively referred to herein as cameras or image sensors) installed within the vehicle to capture one or more images or videos of users, including a driver, passengers, and / or occupants, seated / moving in the vehicle. Image capture unit 106 may also capture images or videos of the road or exterior of the vehicle. Image capture unit 106 may communicate with processing device 102 and / or a vehicle control unit (VCU) 108 of the vehicle.

[0031] 3 , which shows an exemplary diagram of the interior of a vehicle 300, the infotainment system 104 is typically installed on a dashboard 302 inside the interior of the vehicle 300. Furthermore, an image acquisition unit 106 including a camera and / or an image sensor 106 may also be installed inside the vehicle interior to capture or monitor images or videos of the user / driver's face in real time. The camera 106 may be aimed at the front seat of the vehicle 300 where the user / driver may be seated. The camera or image sensor 106 may be positioned on the dashboard 302 or ceiling 304 of the vehicle 300 to fully cover the interior of the vehicle 300 and capture images of the user / driver's face. The camera 106 may be an infrared (IR) camera, such as a near-infrared camera, a mid-wave infrared camera, a long-wave infrared camera, etc.

[0032] In one embodiment, infotainment system 104 may include an input unit such as a keyboard and / or buttons, an output unit with a display interface 104-1 (also referred to herein as a display) and / or speakers for audio / visual output, and a communication unit to enable communication between infotainment system 104 and processing unit 102 and vehicle VCU 108. In another embodiment, infotainment system 104 may be connected to a power source of vehicle 300 via a wired medium or may include an internal power source.

[0033] In one embodiment, the processing device 102 may communicate with or be operatively coupled to the image acquisition unit 106, the infotainment system 104, and the vehicle's VCU 108 via a network. Furthermore, the network may be implemented as one of different types of networks, such as an intranet, a local area network (LAN), a wide area network (WAN), the Internet, etc., and may be a wireless network, a wired network, or a combination thereof. Furthermore, the network may be either a dedicated network or a shared network. A shared network may represent different types of network connections that may use various protocols, such as HyperText Transfer Protocol (HTTP), Transmission Control Protocol / Internet Protocol (TCP / IP), Wireless Application Protocol (WAP), etc.

[0034] In one embodiment, the processing unit 102 may be implemented using any or a combination of hardware and software components, such as a cloud, a server, a computing system, a computing device, a network device, etc. Furthermore, the processing unit 102 may interact with the image acquisition unit 106, the infotainment system 104, and the VCU 108 via a wired or wireless network.

[0035] The image acquisition unit 106 may be configured to capture images / videos of the user / driver's face. The processing device 102 may be configured with a pre-trained gaze detection module that may enable the processing device 102 to crop one or more eye regions from the captured image of the user / driver's face and correspondingly determine yaw and pitch indicative of the user / driver's gaze direction over a real-time frame and one or more previous time frames. The processing device 102 may further be configured with a learning module, such as, but not limited to, an LSTM model, that may enable the processing device 102 to estimate the user / driver's subsequent gaze direction based on the determined gaze direction over the real-time frame and one or more previous time frames. Accordingly, the processing device 102 may send a set of control signals to the infotainment system 104 or the VCU 108 to adjust the brightness of the display of the infotainment system 104 to a predetermined level based on the overlap of the infotainment system's display area with the user / driver's estimated subsequent gaze direction. While driving the vehicle, the system 100 accurately identifies the direction the driver is currently looking and the direction they will be looking in the subsequent time frame, which helps to efficiently and comfortably control the brightness of the vehicle's infotainment screen / display 104-1.

[0036] In one embodiment, the processing unit 102 may be configured to project the estimated subsequent gaze direction into a 3D coordinate system of the vehicle, as shown in Figures 3 and 6, and align the estimated subsequent gaze direction with one or more regions of interest (ROIs) of the display 104-1 within the vehicle interior, as shown in Figure 7. The processing unit 102 may be configured to derive a confidence region representing an ellipse within the vehicle interior, where the yaw and pitch of the estimated subsequent gaze direction may correspond to a center point of the ellipse, and the yaw and pitch variances of the corresponding gaze direction determined over the real-time frame and one or more previous time frames may correspond to the radii of the ellipse, as shown in Figure 6.

[0037] 7, the processing unit 102 can calculate the overlap of the ellipse with the ROI (display 104-1 of infotainment system 104) and can adjust the brightness of display 104-1 accordingly. The processing unit 102 can trigger the ROI of display 104-1 of infotainment system 104 if the derived confidence region (ellipse) is within the ROI of display 104-1 of infotainment system 104.

[0038] In one embodiment, the pre-trained gaze detection module can include a first branch with a first neural network architecture for regressing corresponding eye images to a gaze map. Additionally, the pre-trained gaze detection module can include a second neural network architecture for regressing the gaze map to a gaze direction represented by yaw and pitch. The pre-trained gaze detection module can include a second branch, which is fed by the gaze map and configured as a classification network architecture for reducing entropy loss to enable the model to determine an actual gaze map based on training with synthetic data.

[0039] In one exemplary embodiment, the gaze detection module can be pre-trained using synthetic and / or real-time data. The pre-trained gaze detection module can enable the processing device to detect eye closure in at least one eye region from captured images. The processing device 102 can then correspondingly select and detect a corresponding gaze direction from the open eye image. This allows the system 100 to identify the direction a user / driver wearing eyeglasses or sunglasses is looking and would look while operating a vehicle, which can aid in controlling the brightness of the vehicle's infotainment screen 104-1. In one implementation, the processing device 102 can be configured to maintain the brightness level of the display 104-1 at a predetermined level for a predetermined time after an estimated subsequent gaze direction leaves the ROI / display area of ​​the infotainment system 104.

[0040] Referring to FIG. 2 , a block diagram illustrates exemplary functional units of the processing device 102, which may include one or more processors 202, memory 204, interface 206, processing engine 208, and database 210. The one or more processors 202 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, logic circuits, and / or any device that manipulates data based on operational instructions. Among other capabilities, the one or more processors 202 may be configured to fetch and execute computer-readable instructions stored in the server's memory. The memory 204 may store one or more computer-readable instructions or routines that may be fetched and executed to create or share data units via the network service. The memory 204 may include any non-transitory storage device, including, for example, volatile memory such as RAM, or non-volatile memory such as EPROM, flash memory, etc.

[0041] In one embodiment, processing unit 102 may also include interface 206. Interface 206 may include various interfaces, such as interfaces for data input / output devices, referred to as I / O devices, storage devices, etc. Interface 206 may facilitate communication of processing unit 102 with various devices coupled to a server, such as an infotainment system, an image capture unit, a VCU, and a vehicle power source. Interface 206 may also provide a communication path for one or more components of processing unit 102. Examples of such components include, but are not limited to, processing engine 208 and database 210.

[0042] In one embodiment, processing engine 208 may be implemented as a combination of hardware and programming (e.g., programmable instructions) to implement one or more functions of processing engine 208. In the examples described herein, such a combination of hardware and programming may be implemented in a variety of different ways. For example, the programming of processing engine 208 may be processor-executable instructions stored on a non-transitory machine-readable storage medium, and the hardware of processing engine 208 may include processing resources (e.g., one or more processors) for executing such instructions. In this example, the machine-readable storage medium may store instructions that, when executed by the processing resources, implement processing engine 208. In such an example, processing device 102 may include a machine-readable storage medium that stores instructions and the processing resources for executing the instructions, or the machine-readable storage medium may be separate from and accessible to processing device 102 and the processing resources. In other examples, processing engine 208 may be implemented by electronic circuitry. Database 210 may include data stored or generated as a result of functions implemented by any component of processing engine 208.

[0043] In one embodiment, processing engine 208 may include gaze detection module 212, learning module 214, actuation and control unit 216, alert unit 218, and other units 220. Other units 220 may implement functionality complementary to applications or functions performed by processing device 102 or processing engine 208.

[0044] According to one embodiment, the gaze detection module 212 enables the processing device 102 to enable the image acquisition unit 106 to capture images / videos of the user / driver's face. The processing device 102 may be configured with a pre-trained gaze detection module as shown in Figures 5A and 5B. The gaze detection module may enable the processing device 102 to crop one or more eye regions from the captured image of the user / driver's face and correspondingly determine the yaw and pitch indicative of the user / driver's gaze direction not only across a real-time frame but also across one or more previous time frames.

[0045] In one embodiment, the processing unit 102 may further be configured with a learning module 214, such as an LSTM model as shown in Figure 5B, that enables the processing unit 102 to estimate a subsequent gaze direction of the user / driver based on gaze directions determined over a real-time frame and one or more previous time frames.

[0046] In one embodiment, the processor can cause the processing unit 102 to project the estimated subsequent gaze direction into the vehicle's 3D coordinate system and align the estimated subsequent gaze direction with a region of interest (ROI) of the display 104-1 within the vehicle's interior. Furthermore, the learning module 214 can cause the processing unit 102 to derive a confidence region representing an ellipse within the vehicle's interior, where the yaw and pitch of the estimated subsequent gaze direction can correspond to a center point of the ellipse, and the yaw and pitch variances of the corresponding gaze direction determined over the real-time frame and one or more previous time frames can correspond to the radius of the ellipse, as shown in FIG. 6 . Furthermore, the learning module 214 can cause the processing unit 102 to calculate the overlap of the ellipse with the ROI (the display 104-1 of the infotainment system 104) as shown in FIG. 7 and adjust the brightness of the display 104-1 accordingly. In one implementation, the processing unit 102 can be configured to maintain the brightness level of the display 104-1 at a predetermined level for a predetermined time after the estimated subsequent gaze direction leaves the ROI.

[0047] Cartesian Equation

number

[0048] The yaw variance (yawV) and pitch variance (pitchV) are calculated over the past N frames (i∈{t, t−1, . . . , tn−1}) using the following equations:

number

number

number

[0049] In one or more examples, an infotainment system ROI can be triggered if the 3D projection (ellipse) of the predicted yaw and pitch is calculated to be within the ROI. Furthermore, if no solution is determined, the ROI and the ellipse do not intersect, meaning the ROI is not triggered. Furthermore, if only one solution exists, the ellipse is only tangent to the edge of the ROI. Furthermore, if two or more solutions exist, the ROI can be triggered.

[0050] In one embodiment, the actuation and control unit 216 can cause the processing unit 102 to send a set of control signals to the infotainment system 104 or the VCU 108 to adjust the brightness of the display 104-1 of the infotainment system 104 to a predetermined level based on the overlap of the display's ROI with the user / driver's estimated subsequent gaze direction. Thus, the system 100 accurately identifies the direction the driver is currently looking and will look in a subsequent time frame while driving the vehicle. This helps control the brightness of the vehicle's infotainment screen. The processing unit 102 can trigger the ROI of the display 104-1 of the infotainment system 104 if the derived confidence region is within the ROI associated with the display 104-1 of the infotainment system 104.

[0051] In another embodiment, the ROI may be a road on which the vehicle is traveling. The alert unit 218 may cause the processing unit 102 to generate an alert if the estimated gaze direction of the user / driver is estimated to be off the road (ROI) for a first predetermined time period, indicating that the user / driver's attention is diverted. Furthermore, the alert unit 218 may also cause the processing unit 102 to generate an alert if the user / driver's eyes are found to be closed for a second predetermined time period, indicating that the user / driver is asleep or unconscious.

[0052] 5A and 5B, which illustrate an exemplary architecture of the gaze detection module 212 of the proposed system 100 using an LSTM model. In one embodiment, the pre-trained gaze detection module 212 can include a first branch with a first neural network architecture for regressing corresponding eye images to a gaze map. Furthermore, the pre-trained gaze detection module 212 can include a second neural network architecture for regressing the gaze map to a gaze direction represented by yaw and pitch. The pre-trained gaze detection module 212 can include a second branch, which is fed by the gaze map and configured as a classification network architecture for reducing entropy loss to enable the model to determine an actual gaze map based on training with synthetic data. The LSTM model can include one or more LSTM layers, a dropout layer, and a dense layer.

[0053] In one exemplary embodiment, the gaze detection module 212 can be pre-trained using synthetic and / or real-time data by reducing the M-Entropy loss. The pre-trained gaze detection module 212 can enable the processing unit 102 to detect eye closure in at least one eye region from a captured image. The processing unit 102 can accordingly select and detect a corresponding gaze direction from an image of an open eye. This enables the system 100 to identify the direction a user / driver wearing eyeglasses or sunglasses is and would look while operating a vehicle, which can be useful for controlling the brightness of the vehicle's infotainment screen.

[0054] 4A and 4B, a proposed method 400 for controlling brightness of a vehicle infotainment system includes an image acquisition unit and a processing device connected to the infotainment system. The method 400 includes step 402, in which an image of a user / driver's face is captured by the image acquisition unit with a camera. The method 400 further includes step 404, in which the processing device crops one or more eye regions from the face image captured in step 402. Step 404 may include a face detector that detects a face from the captured image based on facial landmarks present in the image. Furthermore, the eye regions are cropped from the face image, but head pose may also be detected for gaze detection. The left eye, right eye, or both eye regions can be cropped from the face image.

[0055] Method 400 further includes step 406 of determining, by a pre-trained gaze detection module associated with the processing device, yaw and pitch indicative of the user / driver's gaze direction over the real-time frame and one or more previous time frames based on the cropped images of the user / driver's one or more eye regions in step 404. Method 400 further includes step 408 of estimating, using a learning module (LSTM model) associated with the processing device, the user's subsequent gaze direction based on the corresponding gaze directions determined in step 406 over the real-time frame and one or more previous time frames.

[0056] Method 400 further includes step 410 of adjusting brightness of a display of the infotainment system to a predetermined level based on an overlap between one or more regions of interest (ROIs) associated with the vehicle interior and the estimated subsequent gaze direction estimated in step 408. The ROIs may be associated with a display of the infotainment system. When the ROIs are mapped to the display of the infotainment system in step 410, an overlap area between the ROIs and the estimated subsequent gaze direction may be used to adjust the brightness of the display.

[0057] In one embodiment, at block 410, the method 400 includes deriving, by a processing device, a confidence region representing an ellipse within the vehicle interior, as shown in FIG. 6 . As shown, the yaw and pitch of the estimated subsequent gaze direction can correspond to a center point of the ellipse. Furthermore, the yaw variance and pitch variance of the corresponding gaze direction determined over the real-time frame and one or more previous time frames can correspond to the radius of the ellipse. The method 400 further includes calculating, by the processing device, a convolution of the ellipse with a display ROI, as shown in FIG. 7 , followed by controlling the brightness of the display accordingly. Furthermore, if the derived confidence region is within the ROI, the ROI of the infotainment system can be triggered.

[0058] In one embodiment, if one of the user / driver's eyes is closed in the captured facial image, method 400 may include detecting eye closure in at least one eye region from one or more captured images, followed by detecting a corresponding gaze direction from an image of the unclosed eye. In another implementation, method 400 may include maintaining a brightness level of the display at a predetermined level for a predetermined time after the estimated subsequent gaze direction leaves the ROI.

[0059] Therefore, the present invention (system and method) helps to control the brightness of a vehicle's infotainment screen by accurately identifying the direction a user / driver is looking and will look in while driving a vehicle, which helps reduce eye strain and distraction, accurately identifies the direction a user / driver is looking and will look in while driving a vehicle, helps to control the brightness of a vehicle's infotainment screen at night, provides hands-free brightness control of the infotainment system, saves energy, and provides a comfortable driving experience.

[0060] While various embodiments of the present invention have been described above, other and further embodiments of the present invention may be devised without departing from the basic scope thereof. The scope of the invention is determined by the claims that follow. The invention is not limited to the described embodiments, variations, or examples. The described embodiments, variations, or examples are included to enable those skilled in the art to make and use the invention in combination with available information and knowledge.

[0061] Benefits of this disclosure The present disclosure controls the brightness of an infotainment system in a vehicle based on the driver's gaze direction.

[0062] The present disclosure provides a system and method for controlling brightness of an infotainment system in a vehicle using gaze estimation.

[0063] The present disclosure helps control the brightness of a vehicle's infotainment screen by accurately identifying the direction a driver is looking and will look while driving the vehicle.

[0064] The present disclosure provides a system and method for controlling brightness of an infotainment system that helps reduce driver eye strain and distraction at night.

[0065] The present disclosure provides a system and method for providing hands-free brightness control of an infotainment system in a vehicle.

[0066] The present disclosure efficiently controls the brightness of infotainment system screens in vehicles to save energy and provide a comfortable driving experience. [Prior art documents] [Patent documents]

[0067] [Patent Document 1] Korean Patent Application Publication No. 101469978

Claims

1. An infotainment brightness control system (100) for a vehicle, comprising: a processing device (102) configured with a pre-trained gaze detection module (212) and a learning module (214); The pre-trained gaze detection module (212) a first branch comprising a first neural network architecture for regressing corresponding eye images onto a gaze map, and a second neural network architecture for regressing the gaze map onto a gaze direction represented by yaw and pitch; a second branch fed by the attention map and configured as a classification network architecture to reduce entropy loss so that the model can determine the actual attention map based on training with synthetic data; The gaze detection module (212) is pre-trained using synthetic and / or real-time data including cropped images of eye regions, and the processing device (102) comprises a processor (202) coupled to a memory (204), the memory (204) comprising: using the pre-trained gaze detection module (212) to determine yaw and pitch indicative of the user's gaze direction over a real-time frame and one or more previous time frames based on one or more images of the user's one or more eye regions; using the learning module (214) to estimate a subsequent gaze direction of the user based on the gaze direction determined over the real-time frame and the one or more previous time frames; and storing one or more instructions executable by the processor for transmitting a set of control signals to an infotainment system (104) to adjust brightness of a display (104-1) of the infotainment system (104) to a predetermined level based on an overlap of one or more regions of interest (ROIs) associated with an interior of the vehicle with the estimated subsequent gaze direction. System (100).

2. an image capture unit (106) comprising a camera for capturing one or more facial images of the user; The gaze detection module (212) is configured to: cropping the one or more eye regions from the one or more captured facial images; determining the yaw and pitch indicative of the corresponding gaze direction of the user based on the cropped image of the eye region. The system (100) of claim 1.

3. the processing unit (102) is configured to project the estimated subsequent gaze direction into a 3D vehicle coordinate system and align the estimated subsequent gaze direction with the one or more ROIs within the interior of the vehicle. The system (100) of claim 1.

4. the one or more ROIs are associated with the display (104-1) of the vehicle or the infotainment system (104); The system (100) of claim 1.

5. The processing device (102) deriving a confidence region representing an ellipse within the interior of the vehicle, the yaw and the pitch of the estimated subsequent gaze direction corresponding to a center point of the ellipse, and a variance of the yaw and the pitch of the corresponding gaze direction determined over the real-time frame and the one or more previous time frames corresponding to a radius of the ellipse; configured to calculate the overlap of the ellipse with the one or more ROIs and adjust the brightness of the display (104-1) accordingly; the processing unit (102) triggers the one or more ROIs of the infotainment system (104) if the derived confidence region is within the one or more ROIs. The system (100) of claim 1.

6. the processing unit (102) is configured to detect eye closure in at least one of the eye regions from the captured one or more images, and to select and detect the corresponding gaze direction from images of eyes that are not closed accordingly. The system (100) of claim 1.

7. the processing unit (102) is configured to maintain the luminance level of the display (104-1) at the predetermined level for a predetermined time after the estimated subsequent gaze direction leaves the ROI. The system (100) of claim 1.

8. A method (400) for controlling brightness of an infotainment system (104) of a vehicle, comprising: determining (406) a yaw and pitch indicative of the user's gaze direction over a real-time frame and one or more previous time frames based on one or more images of the user's one or more eye regions, using a processing device (102) configured with a pre-trained gaze detection module; using a learning module of the processing device (102) to estimate (408) a subsequent gaze direction of the user based on the real-time frame and the corresponding gaze direction determined over the one or more previous time frames; and adjusting (410) a brightness of a display (104-1) of the infotainment system (104) to a predetermined level based on an overlap of one or more regions of interest (ROIs) associated with an interior of the vehicle with the estimated subsequent gaze direction. Method (400).

9. Capturing (402) an image of the one or more faces of the user by an image capture unit comprising a camera; cropping (404) the one or more eye regions from the one or more captured facial images by the processing device (102); and determining (406) the yaw and pitch indicative of the gaze direction of the user based on the cropped image of the eye region by the processing device (102).

9. The method (400) of claim 8.

Images